Deconstruct Any Metric with a Few Simple ‘What’ Questions

“The development is 5x” [FULL STOP], it’s designed to make you assume, “That’s spectacular; it should be price my money and time.”

Nonetheless, a standalone assertion like that could be a crimson flag, and understanding find out how to interrogate imprecise metrics is a foundational talent for anybody who desires to separate actual worth from intelligent advertising.

That’s why I invite you to place in your information analyst goggles and uncover the context wanted to get insights you’ll be able to act on.

The popularisation of analytics adopted an identical path to psychology, transferring from specialist steering to self-help books and ultimately to weblog posts, YouTube movies, and inspirational Instagram quotes stamped on T-shirts. The entire concept was to pour our ache into phrases we may proudly put on on the very best piece of clothes on the market — the hoodie — and at last clarify to the world what an analyst (me!) does, considering it explains the analytics area, too.

In fact, for the oldsters who caught me on a laundry day, I had a easy clarification of analytics able to go, and shared that analytics is “What you see is never what you get,” whereas statistics, then again, is “The reality is on the market” (as acknowledged in The X-Files).

Then I added that I’m at all times obsessed with trying to find the “fact,” referencing my six years of Ivory Tower expertise, however I desire to swim in analytical waters now. 

That’s why I stand by the assertion:

Flashy dashboards backed with information storytelling typically goal to obfuscate the untrained eye by presenting cherry-picked insights. 

In the meantime, the educated eye is aware of one thing extra is hiding behind the metrics every time a press release like this exhibits up:

“The development is 5x” 

Adopted by the total cease/interval/full level/dot or no matter you like to name it. I don’t find out about you, nevertheless it hurts after I take a look at that sentence understanding somebody dared to name it a hit metric.

For the sake of the frequent sense I hope nonetheless exists within the present AI limbo, let’s rewrite the above assertion the best way a metric must be introduced by including…

#1: The size

I do know a couple of of my associates who’re nonetheless Simon Sinek followers would say, “Begin with why.” 🙂

Nope. I began with “what,” and requested, “The development of what?” 

I dared as a result of the very first thing that went by my thoughts after seeing the phrase “enchancment” hanging like that was: 

print(“The development is 5x.”)

And that’s no method to get insights anybody can act on, is it?

However think about for a second the assertion was formulated as “The mannequin accuracy enchancment is 5x.”  

If that had been the case, I might have pictured one thing completely different and envisioned a small desk with particular efficiency dimensions, resembling mannequin accuracy, and their precise logged measurements.

Nonetheless, to make sure the “5x” wasn’t pulled out of skinny air, the second dimension lacking from this imaginary desk of mine is the date/datetime. Which might make our assertion sound like:

“The month-to-month mannequin accuracy enchancment is 5x.”

Now, that’s higher, or not less than it feels this manner, as a result of we will declare we derived the advance by evaluating mannequin accuracy logs throughout month-to-month runs.

However, to know this “5x”, or any enchancment in anyway, one other necessary lacking piece of knowledge is…

#2: The baseline

Which is why I’ll proceed with my “what” questions by asking, “The development from what?” 

A “5x enchancment” actually does sound superb till we realise the baseline mannequin accuracy for predicting the precise proper consequence out of 100 prospects was 1% final month, and now it’s 5%. 

If we noticed the uncooked worth of 5%, we might comprehend it means the mannequin’s predictions are nonetheless unsuitable 95% of the time, and wouldn’t take into account it a type of enchancment that ought to drive our actions. That’s why we frequently received’t get introduced with the uncooked numbers, however merely a “5x” as a result of it seems nice on a dashboard.

Having this info, we will rewrite our assertion once more:

“The month-to-month mannequin accuracy enchancment is 5x, rising from a baseline of 1% to five%.”

That appears higher once more. Nonetheless, seeing a particular time interval hooked up brings me to the subsequent lacking piece…

#3: The comparability interval

Which leads me to my final “what” query: “The development in contrast to what interval?” 

Our assertion nonetheless doesn’t inform us what sort of comparability it’s making — an evolution over time, a fixed-cadence comparability (month-over-month, quarter-over-quarter, year-over-year), or an arbitrary interval comparability?

To reframe: Is it a direct comparability to the earlier month, or throughout a number of consecutive months? Possibly a year-over-year comparability, the place this month was in comparison with the identical month final yr? Or is it simply an arbitrary comparability between two hand-picked months?

Assuming we acquired a solution, we will tweak the earlier model as soon as extra:

“Evaluating Might 2026 to April 2026 outcomes, the month-to-month mannequin accuracy enchancment is 5x, rising from a baseline of 1% to five%.”

A lot better. The sentence lastly tells us what improved, from what baseline, and over which interval. And now, to the delight of a few of my associates, I’ll ask…

The “why” questions

To wrap up, the 2 “why” questions I might ask when served with a press release like “The development is 5x” are:

#1: Why ought to this metric concern me?
The metrics that drive the presenter’s selections aren’t at all times those that drive yours, and the identical goes for the extent of element. What’s sufficient for them to resolve typically isn’t sufficient so that you can act. So the subsequent time somebody presents a press release just like the one above, ask them who it’s for and which actions it’s speculated to drive.

#2: Why had been the “what” questions omitted?
Sure, beginning with “why” is related, particularly when you’re attempting to know a problem for a particular use case. However on the subject of understanding the metric in entrance of you, somebody introduced with its personal “why” in thoughts, it’s good to interrogate it with “what” questions so that you don’t get fooled by flashy dashboards and information storytelling.