When human subjects are involved in an experiment, the cost often goes
up dramatically, while significance goes down. When are expensive
experiments justified? When the implications of the insights gained
outweigh the cost. Let us take an example. A significant segment of
the software industry has converted from C to C++ at a substantial
cost in retraining. One might well ask how solidly grounded the
decision to switch to C++ was. Other than case studies (which are
questionable because they don't generalize easily and may be under
pressure to demonstrate desired outcomes), I'm not aware of any solid
evidence showing that C++ is superior to C with respect to programmer
productivity or software quality. Nor am I aware of any independent
confirmation of such evidence. However, while training students in
improving their personal software processes, my research group has
recently observed that C++ programmers may make many more mistakes and
take much longer than C programmers of comparable training - both
during initial development and maintenance. Suppose this
observation is not a fluke.
Then running experiments to test the
fundamental tenets of object-oriented programming would be truly
valuable. These experiments might save resources far in excess of
their cost. The experiments might also have a lasting and positive
effect on the direction of programming language research. They may not
only save industry money, but also save research effort.
It is useful to check what scientists in other disciplines spend on experimentation. Everyone realizes that drug testing in medicine is extremely expensive, but only desperate patients accept poorly tested drugs and therapies. In aeronautics, we demand that airfoils be tested; expensive wind tunnels have been built for just this purpose. Numerical simulation has reduced the number of such tests, but not eliminated them. In many sciences, simulation has become an important form of experimentation, and computer science might also benefit from good simulation techniques. In biology, Wilson names the Forest Fragmentation Project in Brasilia as the most expensive biological experiment ever[16]. While clearing a large tract of the Amazon jungle, isolated patches of various sizes (1 to 1000 hectares) were left standing. The purpose was to test hypotheses regarding the relationship between habitat size and number of species remaining. And the list of experiments continues - in physics, chemistry, ecology, geology, climatology, and on and on. Any reader of Scientific American can find experiments in every issue. Computer scientists need not be afraid or ashamed of conducting large experiments when exploring important questions.