Evaluating the Evaluation: A Benchmarking Checklist


A co-worker introduced me to Craig Hanson and Pat Crain’s performance mantras, which neatly summarize much of what we do in performance analysis and tuning. They are:

Performance mantras

  1. Don’t do it
  2. Do it, but don’t do it again
  3. Do it less
  4. Do it later
  5. Do it when they’re not looking
  6. Do it concurrently
  7. Do it cheaper

These have inspired me to summarize another performance activity: evaluating benchmark accuracy. Good benchmarking rewards investment in engineering that actually improves performance, but, unfortunately, poor benchmarking is a lot more common. I have spent a lot of my career refuting bad benchmarks, and have developed such a knack for it that prior employers adopted a rule that no benchmark can be published unless approved by me. Because benchmarking is so important for our industry, I’d like to share with you how I do it.

Read more at Brendan Gregg’s Blog