Should Computer Scientists Experiment More?
16 Excuses to Avoid Experimentation

Walter F. Tichy
University of Karlsruhe, Germany

Nov. 1997

Abstract:

Computer scientists and practitioners defend the lack of experimentation with a wide range of arguments. Some arguments suggest that experimentation may be inappropriate, too difficult, useless, and even harmful. This article discusses several such arguments to illustrate the importance of experimentation for computer science.

This is a preprint of an article with the same title that appeared in IEEE Computer, 31(5), May 1998, 32-40.

Keywords: Empiricism, experiments, laboratory, scientific method.

• Is computer science an experimental science?
• Why should we experiment?
  • Traditional scientific method isn't applicable
  • The current level of experimentation is good enough
  • Experiments cost too much
    • Benchmarking.
    • Costly experiments.
  • Demonstrations will suffice
  • There is too much noise in the way
  • Progress will slow
  • Technology changes too fast
  • You'll never get it published
• Why substitutes won't work
  • Feature comparison is good enough
  • Trust your intuition
  • Trust the experts
• Problems do exist
  • Flawed experiments
  • Competing theories
  • Soft science
  • Misuse
• Conclusion
• References
• About this document ...