The Trap in the Measurement

The same number tells a different story under a different definition

The claim I came across is: [conclusion + data]. Do one thing only: take apart how it was measured. 1) What are the numerator and the denominator, and does either include things that should not count; 2) How was the time window chosen, and does the conclusion hold under a different one; 3) Who is in the sample, and who was excluded; 4) Is it a mean, a median, or a percentage — what happens with another one; 5) Is there survivorship bias, or fake growth caused by a shrinking denominator. End with one line: how is this number most likely to be misread. Do not judge whether the conclusion is right.

How to use

Paste the raw data and the measurement definition together; otherwise it can only guess the definition.

Good for

A number looks too good to be true — 300% growth, a 40% efficiency gain — and it happens to support the decision you already lean toward.
Before you put a KPI into a team's targets or into public marketing — define the metric wrong once and everyone optimizes toward the wrong thing afterwards.

Not for

The number comes from your own raw data and you computed the numerator and denominator yourself — auditing it again is just ceremony.
The real question is a value judgment (should we build this at all) and the number is one line of support — the audit will look rigorous and leave your actual question unanswered.

Example

Input

Our retention went from 32% to 51% because we changed the signup flow.

What you get

You want it to ask whether the denominator changed first — counting only users who verified their email, while the verification rate fell from 70% to 45%, will flatter retention on its own. Then whether the conclusion holds under a different window (7-day vs 30-day), and finally the most likely misreading: “the signup flow got better” being heard as “the product got better.”

Common mistakes

Turning the audit into “prove this number is fake”: you end up alleging the data is fabricated when the real problem is that the definition is fine and the conclusion is over-read.
Auditing one metric and stopping: the numerator and denominator get dissected, but nobody asks whether this metric and the outcome you want are the same thing.

Try tweaking it like this

Swap “unpack the definition” for “assume the number is right — what is the strongest claim it supports” — set the ceiling first, then audit, so you don't throw out good data along with the bad.
Add “give me a conservative definition from the same source and say how far apart the two are” — the audit becomes two comparable calculation methods.