The Measurement Curmudgeon — Part 1: Detection

Perhaps you’ve heard him in meetings — he is the one questioning your results. Perhaps you’ve seen him at his desk surrounded by tombs and tables in an effort to lower incremental sales calculations — he calls it reducing bias. Perhaps you’ve hoped he will not be assigned to your project — he delivers lower lift estimates than his peers. He is the measurement curmudgeon.

How do you detect if a measurement curmudgeon resides in your office? Listen for the following clues/questions:

  • Is that control group really comparable to the experimental group? Isn’t it biased toward less engaged customers and inflating your measured lift?
  • Wasn’t that concurrent with our fall promotion? Isn’t that event likely accounting for most of your positive results?
  • Haven’t sales been trending up? Did you incorporate that trend into your analysis?
Read more