AI / Analysis & Decision Making
Noise or Signal?
Check the variance band before you call it a trend
I'm seeing [metric] move [change] and reading it as [trend]. Answer one question only: noise or signal? 1) What is this metric's normal historical range, and how many standard deviations is the current move; 2) How many periods of data do I have — enough to call a trend; 3) Is a one-off event, a definition change, or seasonality at work; 4) If it's noise, how long until it reverts on its own; 5) To confirm a signal, what is the minimum further evidence and when would it arrive. End with one sentence on what I can claim today.
How to use
Paste the last 12 raw periods. Told only that it “went up 20%,” it can't tell whether that's normal variance.
Good for
- Someone at the weekly brings a rising KPI curve and asks for more budget, and you want to know if the line is just jitter.
- Your metric has fallen for two weeks and the team is already changing strategy; you need to know whether that's an overreaction.
Not for
- The move is far outside the historical band and there's a clear mechanism — time for attribution, not more doubt about the data.
- You have one period and no baseline — any verdict is a guess; go collect data first.
Example
Input
Our day-1 retention fell from 31% to 28.4% and has stayed there for two weeks. We shipped a new onboarding flow in the same window; DAU is 42,000, so the sample isn't small.
What you get
Start with the band: over the past 12 weeks this product's day-1 retention has a standard deviation of about 0.8 points, so a 2.6-point drop is more than 3 sigma — noise doesn't explain it. To blame the onboarding flow, though, you still have to rule out the one-off: the store featuring slot ended in those same two weeks, and the change in acquisition mix may be the real cause. Today you can say the change is real; you cannot say it was caused by onboarding — one more week, splitting retention by new versus existing flows, would settle which.
Common mistakes
- Using it as a delaying tactic: every bad number is “let's see if it's noise,” and by the time it's confirmed you've lost two months of adjustment.
- Applying it only when numbers fall, and crediting your own change when they rise — the same standard has to run both ways.
Try tweaking it like this
- Swap “is it noise” for “if it's noise, when does it revert,” and commit to a specific observation date.
- Add “list three mechanisms that could explain this and one observation that separates them,” upgrading a verdict into attribution.