Walter F. Tichy
University of Karlsruhe, Germany
Nov. 1997
For the reader eager to learn about the role of experimentation in general, I suggest the following literature. Chalmers, ``What is this Thing Called Science'' [3] is enligthening about the philosophical underpinnings of the scientific process. It discusses, among others, inductivism, Popper's falsificationism, Kuhn's paradigms, objectivism, and the theory dependence of observation. Latour, ``Science in Action'' [11] describes the social processes of science-in-the-making as opposed to ready-made science. The fact-building and convincing power of laboratories is illustrated with fascinating examples. I devoured both of these books. I highly recommend reading at least one of them.
The following literature concerns experimentation in computer science. For an appreciation of how software researchers experiment, sample the material cited in the main article and the papers by Basili and Frankl [1, 8]. A good example of a repeated experiment in compiling is Brett [2]. Denning [6] argues that performance evaluation is an excellent form of experimentation in computer science. A landmark in making computer architecture research quantitative is Hennesey's and Patterson's classic book ``Computer Architecture'' [9]. Cohen, ``Empirical Methods in Artificial Intelligence'' [5] covers empirical methods in AI, but a large part applies to all of computer science. An excellent discussion of experimental designs as well as a wealth of material on experimentation with software appears in Fenton and Pfleeger, ``Software Metrics'' [7].
General experimental methods are presented by Christensen, ``Experimental Methodology'' [4] and Judd at al., ``Research Methods in Social Relations'' [10]. There are numerous titles regarding statistics; an excellent introductory text is Moore [12]. One of the best stastical packages available today is S-Plus. The book ``Modern Applied Statistics with S-PLUS'' by Venables and Ripley [13] is both a guide to using S-Plus and a course in modern statistical methods. However, keep in mind that sophisticated statistical analysis is no substitute for good experimental design.