A significant result is not the same as a true one
💎CORE ARGUMENT
A significant result tells me one narrow thing: if there were no real effect, data this extreme would be unusual. It does not tell me the effect is real, how big it is, or whether I should care. The step from unusual under no effect to therefore true needs an ingredient the test does not contain: how likely the idea was before I ran it.
The arithmetic is short enough to do on a tram. Suppose I test a hundred product ideas and ten of them really work. With decent power I detect eight of the ten. Of the ninety that do nothing, the usual threshold lets four or five through by chance. So I hold twelve or thirteen wins, and about a third of them are noise. If only two in a hundred of my ideas are any good, most of my wins are noise. Nothing about the test changed. The prior did.
This reorders what I worry about. Long shots need more evidence than safe bets, not the same amount. A surprising result from a small test deserves less belief than a boring one, which is the opposite of how a meeting treats them. And running many comparisons and reporting the one that crossed the line is the same lottery with more tickets.
In May I presented the auto-triage test. It would have been easy to say significant, ship it. I gave the estimated effect with its range, and said how plausible I had found the idea beforehand.
The rule I keep: a p-value answers a question nobody in the meeting asked. They asked whether it is true, and that needs the prior, the size of the effect and a second look.
Where I might be wrong. I almost never know the prior. I am guessing it, and a guessed prior is a tidy way to dismiss any result I dislike while sounding rigorous.
I also stated the definition loosely in the first draft of the issue, and a reader who is a biostatistician put it right before sending. That is a fair measure of how well I hold this: well enough to use, not well enough to teach unsupervised.
🧱BUILT FROM
🌍WHY IT MATTERS
- I write down how likely I think an idea is before the test runs, in a sentence, where I cannot edit it afterwards.
- A surprising win gets a rerun before it gets a roadmap slot.
- I do not say significant in a readout without the effect size next to it.
☑️NEXT STEPS
- Add a prior-belief line to the experiment template at work
- Redo the hundred-ideas sum with our real hit rate, if Bao can find one
- Ask Hale how he would explain this without the word prior
A paper on why low prior odds and small studies make most positive findings unreliable, and a statisticians' statement on what a p-value does and does not mean.
The prior is a base rate, as in 💎 A number without its base rate is a story. What to report in place of a verdict is in 💎 An interval is more honest than a point.
A significance test controls how often a true null is rejected. The share of rejections that are real depends also on power and on the proportion of tested ideas that are true. When that proportion is low, false positives outnumber true ones even with a strict threshold.
A team has to decide on Thursday. A threshold, crude as it is, gives a rule that nobody can bend towards the answer they wanted, and a rule applied consistently may beat judgment applied hopefully.
An issue a reader corrected before it went out, and a readout given as a range and not as a win.