Abstract: Sometimes one wants to measure the speed of software, for instance, to measure whether a new way to solve a problem is faster than the old one. Making such time measurements and microbenchmarks requires considerable care, especially on managed platforms like the Java Virtual Machine and Microsoft’s Common Language Infrastructure (.NET), or else the results may be arbitrary and misleading. Here we give some advice on running microbenchmarks, in particular for managed platforms. Most examples are in Java but the advice applies to any language executed on a managed platform, including Scala, C # and F#. We consider primarily single-threaded non-concurrent applications. 1 The challenge of managed platforms Measuring the execution time of a piece of software is an experimental activity, involving the software and a computer system, itself consisting of much (systems) software and some hardware. Whereas biological experiments, such as measuring bacterial growth, are influenced by natural variation and many unknown circumstances, software performance measurements may seem straightforward: in principle, everything is man-made and under the experimenter’s control. In practice, software performance measurements are influenced by so many factors, and modern computer systems are growing so complex, that software experiments increasingly resemble biological experiments
