The Measurement Curmudgeon — Part 1: Detection

Perhaps you’ve heard him in meetings — he is the one questioning your results. Perhaps you’ve seen him at his desk surrounded by tombs and tables in an effort to lower incremental sales calculations — he calls it reducing bias. Perhaps you’ve hoped he will not be assigned to your project — he delivers lower lift estimates than his peers. He is the measurement curmudgeon.

How do you detect if a measurement curmudgeon resides in your office? Listen for the following clues/questions:

  • Is that control group really comparable to the experimental group? Isn’t it biased toward less engaged customers and inflating your measured lift?
  • Wasn’t that concurrent with our fall promotion? Isn’t that event likely accounting for most of your positive results?
  • Haven’t sales been trending up? Did you incorporate that trend into your analysis?

Sometimes his questions take on a more academic tone, sprinkled with words that bring back vague memories of a long-forgotten analytics course:

  • If you take into account customer level heterogeneity do you get the same results? (Is the average misleading here?)
  • Isn’t that simply regression toward the mean? (Shouldn’t we naturally expect last year’s underperformers to partially rebound this year?)
  • Doesn’t multicollinearity call your recommendations into question? (Does correlation between your predictors make it hard to figure out which one we should address first?)

When explaining his results, as opposed to questioning yours, he may sound a bit academic. While not meaning to be defensive, it can be hard to collaborate with the curmudgeon when you hear:

  • Correlation simply shows co-occurrence and should not be thought of causally.
  • Those counterintuitive results are a simple example of Simpson’s paradox.
  • An adequate counterfactual cannot be created, so I cannot provide any insights on that topic.

Measurement curmudgeons are tremendous assets to Customer Insight teams, though a little behavioral modification is often necessary to increase their effectiveness. In later blogs we will further explore the marketing curmudgeon, his mindset, and how to increase his impact on the organization.

But first I would like to know your experience — Do you have a measurement curmudgeon in your office? Would you define the working relationship as helpful, frustratingly helpful, or simply frustrating?

Comments

Measurement

Nigel - these are entirely reasonable questions, I am unsure why the article tone is tending to negative. On the other side of the coin there are measurement nutters, who think corrrelation is the same as causation, think you can and should measure everything and it is possible to define a root cause of everything. I would love to see an article from Forrester that talks of the limitations of measurement or the cultural impact in terms of reduced creativity and distance between management and people who actually do the work.

By the way you forgot to mention non-linear relationships.

Yes to the non-linear!

I completely agree that the measurement curmudgeon would discuss non-linear relationships. I assume hierarchical effects and nested models are also part of his vernacular!

But does the measurement curmudgeon need to be a curmudgeon? That’s a question for a future blog, but several interviewees have suggested an association between the measurement skillset and the curmudgeon mindset.