Incidents Updated
Blameless incident postmortem template
A postmortem (or incident review) records what happened, why, and what will change to make a recurrence less likely or less harmful. Copy the Markdown template below into your wiki or repository, then read the guidance underneath.
On this page
# Postmortem: [short title describing the impact]
Status: [draft | in review | final]
Incident date: [YYYY-MM-DD] Severity: [SEV1-SEV4]
Authors: [names] Incident commander: [name]
Reviewed by: [names, date]
## Summary
[Two or three sentences: what happened, who was affected, how it was fixed.]
## Impact
- User impact: [what users saw, how many, which regions or plans]
- Duration: [start - end, UTC; time to detect; time to mitigate]
- SLO / error budget: [SLO affected, % of budget consumed]
- Business impact: [revenue, support tickets, SLA credits, data loss]
## Timeline (UTC)
| Time | Event |
|-------|-------|
| 14:02 | [Deploy of version X begins] |
| 14:09 | [First alert fires: ...] |
| 14:15 | [Incident declared, SEV2, IC: ...] |
| 14:31 | [Mitigation: rollback started] |
| 14:40 | [Error rate back to normal; monitoring] |
| 15:10 | [Incident resolved] |
## Detection
[How was the problem detected? Could it have been detected sooner? Did alerts fire as intended?]
## Response and mitigation
[What did responders do? What helped and what slowed them down?]
## Root cause and contributing factors
[Describe the technical and organisational conditions that allowed the incident,
not who made a mistake. There is usually more than one contributing factor.]
## What went well
- [...]
## What went poorly
- [...]
## Where we got lucky
- [...]
## Action items
| Action | Type | Priority | Owner | Ticket |
|--------|------|----------|-------|--------|
| [Add canary analysis to deploy pipeline; done when bad canaries halt rollout] | prevent | P1 | [named person] | [link] |
| [Alert on queue depth > 10,000 for 5 min] | detect | P2 | [named person] | [link] |
| [Document and rehearse rollback for service Y] | mitigate | P2 | [named person] | [link] |
## Lessons learned
[What would you tell another team so they avoid the same incident?]
## Supporting information
[Links to dashboards, graphs, chat logs, related incidents.]
When to write one
Agree on triggers in advance so writing a postmortem is routine, not an accusation. Common triggers:
- User-visible downtime or degradation beyond a set threshold, or any SEV1 or SEV2.
- Data loss of any kind.
- On-call intervention such as a rollback or traffic reroute.
- Resolution time above a threshold.
- A monitoring failure, meaning a human or customer discovered the problem before alerts did.
- An incident that consumed a large share of the error budget, as defined in your error budget policy.
Anyone should be able to ask for a postmortem.
Keeping it blameless
A blameless postmortem assumes that everyone acted reasonably given the information, tools and pressure they had at the time. The goal is to find the conditions that made the failure possible. If people expect punishment, they hide information, and the organisation learns less.
- Write about systems and decisions, not people. “The deploy tool allowed a config with no replicas” rather than “Alex pushed a bad config”.
- Ask “how” and “what” rather than “why did you”: what did the dashboard show, what made that action look right?
- Avoid counterfactuals such as “should have” and “failed to”. They describe a world that did not exist.
- Accept that there is rarely a single root cause. List contributing factors.
- Name people only where it helps the reader, such as roles in the timeline.
Writing a useful timeline
- Use one time zone, normally UTC, and say so.
- Include detection, declaration, key decisions, mitigation attempts (including the ones that did not work) and resolution.
- Build it from the incident channel and the scribe’s log, not from memory.
- Compute time to detect and time to mitigate from it; these are worth tracking across incidents.
Action items that get done
- Every item has one owner and a ticket. Items without owners do not happen.
- Classify by type: prevent (stop recurrence), detect (find it faster) and mitigate (reduce impact or recovery time).
- Prefer fewer, concrete items over a long wishlist. “Improve monitoring” is not an action item; “alert when queue depth exceeds 10,000 for 5 minutes” is.
- If responders lacked instructions or found them wrong, add an item to write or fix the alert’s runbook.
- Review open items regularly, for example in a weekly operations review.
Reviewing and sharing
Have someone outside the incident review the draft for clarity and blame. Share the final version widely; other teams learn from your incidents. Many organisations hold a regular postmortem reading or discussion session.
Sources: Google, Site Reliability Engineering, ch. 15 “Postmortem Culture: Learning from Failure”; The Site Reliability Workbook, ch. 10 “Postmortem Culture”.