This concern comes up frequently in computer architecture. Trevor Mudge summarizes it: ``...the rate of change in computing is so great that by the time results are confirmed they may no longer be of any relevance.''[9] The same can be said about software. What good is an experiment when the duration of the experiment exceeds the useful life of the experimental subject, i.e., of a software product or tool?
If a question becomes irrelevant quickly, it is perhaps too narrow and not worth spending a lot of effort on it. But behind many questions with a short lifetime lurks a fundamental problem with a long lifetime. My first advice to scientists here is to probe the fundamental and not the ephemeral, and to learn to tell the difference. My second advice hinges on the observation that technological change often shifts or eliminates assumptions that were taken for granted. Therefore, scientists should anticipate changes in assumptions and pro-actively employ experiments to explore the consequences of such changes. Note that this type of work is much more demanding, and can have much higher long-term value, than merely comparing software products.