Perhaps you’ve heard him in meetings — he is the one questioning your results. Perhaps you’ve seen him at his desk surrounded by tombs and tables in an effort to lower incremental sales calculations — he calls it reducing bias. Perhaps you’ve hoped he will not be assigned to your project — he delivers lower lift estimates than his peers. He is the measurement curmudgeon.
How do you detect if a measurement curmudgeon resides in your office? Listen for the following clues/questions:
Is that control group really comparable to the experimental group? Isn’t it biased toward less engaged customers and inflating your measured lift?
Wasn’t that concurrent with our fall promotion? Isn’t that event likely accounting for most of your positive results?
Haven’t sales been trending up? Did you incorporate that trend into your analysis?