The Art of (Java) Benchmarking

About this Presentation

People write toy Java benchmarks all the time. Nearly always they “get it wrong” – wrong in the sense that the code they write doesn't measure what they think it does. Oh, it measures something all right – just not what they want. This session presents some common benchmarking pitfalls, demonstrating pieces of real, bad (and usually really bad) benchmarks. The session is for any programmer who has tried to benchmark anything. It provides specific advice on how to benchmark, stumbling blocks to look out for, and real-world examples of how well-known benchmarks fail to actually measure what they intended to measure.

People write toy Java benchmarks all the time. Nearly always they “get it wrong” – wrong in the sense that the code they write doesn't measure what they think it does. Oh, it measures something all right – just not what they want. This session presents some common benchmarking pitfalls, demonstrating pieces of real, bad (and usually really bad) benchmarks such as the following: SpecJVM98 209_db isn't a DB test; it's a bad string-sort test and indirectly a measure of the size of your TLBs and caches. SpecJAppServer2004 is a test of your DB and network speed, not your JVM. SpecJBB2000 isn't a middleware test; it's a perfect young-gen-only garbage collection test. The session goes through some of the steps any programmer would go through to make a canned program run fast – that is, it shows you how benchmarks get “spammed.”

The session is for any programmer who has tried to benchmark anything. It provides specific advice on how to benchmark, stumbling blocks to look out for, and real-world examples of how well-known benchmarks fail to actually measure what they intended to measure.