The second line of defense against experimentation goes like this: ``There are too many variables to control, and the results would be meaningless, because the effects I'm looking for are swamped by noise.''
True, experimentation is difficult - for researchers in all disciplines, not just computer science. I think researchers who are invoking this excuse are looking for an easy way out.
An effective simplification for repeated experiments is benchmarking. Fortunately, benchmarking can be used for many questions in computer science. The subjective and therefore weakest part in a benchmark test is the composition of the benchmark; everything else, if properly documented, can be checked by the skeptic. Hence, the composition of the benchmark is always hotly debated (is it representative enough?), and benchmarks must evolve over time to get them closer to what one wants to test.
Experiments with human subjects involve many additional challenges. Several fields have found techniques for dealing with human variability, notably medicine and psychology. We've all heard about control groups, random assignments, placebos, pre- and post-testing, balancing, blocking, blind and double-blind studies, and the battery of statistical tests. The fact that a drug influences different people in different ways doesn't stop medical researchers from testing. And when control is impossible, then case studies, observational studies and an assortment of other investigative techniques are used. Indeed, medicine offers many important lessons on experimental design, on how to control variables and how to minimize errors. Eschewing experimentation because of difficulties is not acceptable.