Does the Claim Have Data Behind It?
Turn 'studies show' into things you can actually check
Check this claim: [paste claim]. Please: 1) split it into specific propositions that data could falsify; 2) say what data would be needed — sample, time range, control group — and which of these the claim leaves out; 3) separate correlation from causation and give at least one alternative explanation; 4) rate the source (primary research, secondhand report, vendor material, no source); 5) give me a checklist of where to look up the underlying numbers.
How to use
Step 3 is what dismantles 'the data shows' — there's usually a third variable sitting between the two.
Good for
- You're about to cite a 'studies show' claim in a report or an article
- A number keeps getting forwarded (a 300% lift in conversion, industry-average attrition) and nobody can say how it was computed
Not for
- You need a rough direction for brainstorming and won't be held to any single figure
- The claim is a value judgment (should we work overtime), where more data doesn't settle it
Example
Input
'Teams using our tool ship 40% faster.'
What you get
It should restate the claim as the falsifiable proposition that delivery cycles shortened by 40% after adoption, then note the missing control group, the undefined metric (which cycle, which sample), and the vendor source. At least one alternative explanation is required, such as teams willing to adopt new tooling already having tighter process. The check is to ask for parallel data from non-adopting teams.
Common mistakes
- Attacking the source without splitting the claim, since a weak source alone doesn't make the claim false
- Reading correlation as cause and funding a decision on it while selection bias goes unexamined
Try tweaking it like this
- Cut to 'the three numbers I must obtain and the fastest path to each' when time is short
- Add 'if the data holds, where is it most likely to be over-extrapolated'