Spiegelhalter on uncertainty
📚SOURCE
💎PERMANENT NOTES
🧵SYNTHESIS
Paraphrase, not quotation: chapters 7 to 9 of 📕 The Art of Statistics, with a look back at chapter 13.
Any number computed from a sample is one of many numbers we could have got. Spiegelhalter's question is how to say how far off it might be, and he answers it twice. The modern way is the bootstrap: treat the sample as a stand-in for the population, draw from it again and again with replacement, recompute the statistic each time, and see how much it wobbles. The spread of those results is the uncertainty, with no formula required. The older way reaches the same place through probability theory and the bell curve.
Either route gives an interval. He is strict about what it means: the method captures the truth in 95 of 100 uses, which is not the same as a 95 percent chance for this one.
And an interval only covers the luck of sampling. A biased sample gives a confident wrong answer.
✨KEY IDEAS
Resample your own data with replacement many times; the variation between resamples estimates the variation between samples you never took.
A margin of error belongs to every published estimate, and for small changes it is often larger than the change.
A 95 percent interval describes how often the procedure is right over many uses, not the probability that this particular interval contains the truth.
Sampling error is the part we can calculate. Systematic error from who was asked and how is usually bigger and has no formula.
Uncertainty should be communicated plainly and confidently, as part of the finding and not as an apology for it.
📝READING NOTES
The week I wrote this
Exam on Tuesday. I finished the book last night, four days too late to be revision and exactly on time to understand what I had been doing in the exam. This morning I fitted the lamp in the office and wrote this at the new desk, which is the first note written there.
Three chapters in plain terms
Chapter 7: how sure can we be. Start with one survey sample. Pretend it is the whole population. Draw a new sample from it of the same size, putting each person back after drawing, so that some appear twice and some not at all. Compute the average. Repeat a thousand times. The thousand averages form a distribution, and the middle 95 percent of it is an interval. Nothing in that paragraph needed a Greek letter.
Chapter 8: probability. The language for what might have happened. He admits it is hard and unintuitive, which I found consoling after a spring of Tuesdays.
Chapter 9: putting them together. With probability theory, the same interval can be had from a formula: the estimate, plus or minus about two standard errors. The standard error shrinks with the square root of the sample size, so four times the data buys only half the width.
What an interval is
See 🔤 Confidence interval for my wording. The short version:
- Correct: if we repeated this, 95 of 100 intervals built this way would contain the true value.
- Wrong, and what I said for years: there is a 95 percent chance the true value is in here.
The difference sounded like pedantry until I had to explain a result to Hélène. The first sentence is a claim about my method. The second is a claim about the world that the method does not license.
At work, before I had the words
On 8 May I presented the triage test as a range and not as a single uplift. I did it because the point estimate looked too good and I did not trust it. The meeting was shorter and calmer than any result I have presented. Nobody asked is it significant. They asked what would narrow it.
He is persuasive that intervals are more honest. He is quieter on what happens when the audience wants a number to put in a plan. The range I showed was heard by one person as its top end and by another as its bottom. An interval tells the truth and still needs a sentence from me about what to do with it. I think the honest order is: range first, then the middle, then what I would decide.
What it produced
💎 An interval is more honest than a point.
Added 30 November: a month of 🧪 Report every metric with a range for a month. Every number in my weekly update has had a range. Two people have started asking for it when it is missing. Still running.
Related notes
- 🔤 Confidence interval
- 🧪 Report every metric with a range for a month
- 📖 Spiegelhalter on asking a question of data
Reading record
- Section read: Chapters 7 to 9
🗣️IN SHORT
- In : The Art of Statistics, chapters 7 to 9
- Spiegelhalter argues that : every estimate from data should travel with a statement of how uncertain it is, and that this can be found by resampling as well as by theory
- by stating that : bootstrap resampling shows how much a statistic would vary between samples, that a confidence interval is a claim about the procedure, and that calculated margins leave out systematic error