REFERENCE
Incident and rollback playbook
๐ TL;DR / Quick Summary <div style="background: rgba(0,0,0,0.3); border: 1px dashed rgba(255,255,255,0.2); padding: 15px; border-radius: 8px; color: rgba(255,255,255,0.7); margin-bottom: 30px;"> What the team agreed at the โ Launch retrospective on 19 March, a week after the night it was needed. Priya Raman drafted it; I kept the customer's side in. Who decides, who writes, and when to roll back without a meeting. At 02:00 on 12 March nobody knew the first of those, and it took forty minutes to find out it was me. </div>
๐ ๏ธ Instructions / Core Material
- Every launch plan names one rollback decider. By name, not by role, with a telephone number and a deputy. For ๐ Exceptions Inbox general availability it was me and it was written nowhere.
- The decider decides alone. Others advise. If the decider cannot be reached in ten minutes, the deputy has the full authority and does not need to apologise for using it.
- Roll back without a meeting when all three are true: customers are seeing wrong data, not merely slow data; the cause is inside what was launched; a rollback path exists and was tested that week.
- Roll back the smallest thing. One feature, not the release. See ๐ Roll back auto-triage on launch night: triage off, everything else left alone.
- One writer. From the first minute somebody who is not fixing anything keeps the timeline: time, what was seen, what was done. No names in the sentences, only in the roster.
- Customers hear from us first. Customer Success and Support get a plain message before the first dispatcher starts at 06:00, even if it says only what is switched off.
- Write-up within five working days, blameless, in the form Priya uses: what happened, what we saw, what we changed.
- The feature comes back on a calm afternoon, announced the day before, with the decider at a desk.
โ ๏ธ Key Warnings / Constraints
- A rollback path that has not been run this week is a diagram.
- Patch forward at three in the morning is nearly always the wrong answer, and it is always proposed by the most tired person on the call.
- Carried over unchanged when the teams merged in August. Lucรญa Ortega owns it now, and the decider on the one roadmap is named per item.
- The playbook is for the night. The reason the night happened was a load test that was too gentle, and no playbook covers that.
๐ Related Entities & Resources
- โ Launch retrospective: the meeting on 19 March where this was agreed
- ๐ Roll back auto-triage on launch night: the decision the playbook is modelled on
- ๐ Exceptions Inbox general availability: the launch it was first needed for
- Launch readiness checklist: the list that now names the rollback decider as line fifteen