Incidents Updated
Runbook template
A runbook (Google calls it a playbook) tells the on-call engineer what an alert means, how to check its impact and which steps to try first. Copy the Markdown template, compare it with the filled example, then read the guidance below.
On this page
Template
# Runbook: [alert name]
Service: [service] Owning team: [team]
Alert rule: [link] Dashboard: [link]
Last reviewed: [YYYY-MM-DD] by [name]
## What this alert means
[One or two sentences: which SLO or symptom it watches and the condition that fires it.]
## User impact
[What users see while it fires. Usual severity: SEV[n].
Declare a higher severity if: (criteria).]
## Triage (first 5 minutes)
1. Acknowledge the page and say in [channel] that you are looking.
2. Confirm impact on [dashboard]: [which panel, what bad looks like].
3. Check recent changes: [deploy log, config and feature flag changes].
4. Check dependencies: [status pages, owning teams' dashboards].
5. Declare an incident if [criteria]; otherwise continue below.
## Mitigation
Try these in order, most reversible first. Stop and escalate if a step
does not work within [N] minutes.
1. [Step, e.g. roll back the last deploy]
```sh
[command]
```
Expected: [output or signal that shows it worked]
2. [Next step]
3. [Do not: (actions that make things worse or need another team)]
## Escalation
- [Role or team], via [how], when [condition or time without progress].
- Dependencies: [service: owning team and how to reach them].
## Verify recovery
[Which signal returns to normal, and how long to watch before resolving.]
## Background
[Request path, dependencies, known failure modes, links to design docs.]
## After the incident
- Postmortem required if: [criteria from your severity or error budget policy].
- Fix anything in this runbook that was missing, wrong or slow to follow.
Filled example
A runbook for a fast-burn alert on the checkout API from the example SLO specification: 99.9% of requests succeed over a rolling 28 days. The burn rate and error ratio come from the burn-rate alerting page’s 2%-in-one-hour rule, scaled to a 28-day window; check them in the calculator. The service, channels, flag and commands are illustrative.
# Runbook: CheckoutAPIErrorBudgetFastBurn
Service: checkout-api Owning team: Checkout
Alert rule: alerts/checkout.yaml Dashboard: Checkout › SLO overview
Last reviewed: [YYYY-MM-DD] by [name]
## What this alert means
The checkout API is spending its error budget at least 13.44× faster than
its SLO allows, over both the last hour and the last 5 minutes.
SLO: 99.9% of valid requests return a non-5xx status, rolling 28 days.
Starting from a clean window with steady traffic, a burn at exactly the
threshold spends the 28-day budget in 2d 2h; a full outage spends it in
40m 19s. Check the observed burn rate and remaining budget on the dashboard.
## User impact
At least 1.344% of checkout requests fail. Customers see "Payment could not
be completed" and may retry or give up. Usual severity: SEV2.
Declare SEV1 if checkout fails for most users or for every payment provider.
## Triage (first 5 minutes)
1. Acknowledge the page and post "Looking at checkout fast burn" in #checkout-oncall.
2. On SLO overview, open "5xx by route" and "5xx by payment provider".
- One provider failing: go to Mitigation step 2.
- All routes failing: go to Mitigation step 1.
3. Check #checkout-deploys (the timestamped deploy log) for a checkout-api
deploy, config change or flag change in the last 2 hours.
4. Check the payment providers' status pages (links under Background).
5. Declare an incident in #incidents if the burn has lasted 15 minutes or is rising.
## Mitigation
Try these in order. Stop and escalate if a step does not bring the 5-minute
error ratio below 1.344% within 10 minutes.
1. Roll back a recent deploy, but only if the previous revision is known good
and works with the current database schema. If the deploy ran a migration,
do not roll back; page the database on-call and the payments lead.
```sh
kubectl -n checkout rollout history deploy/checkout-api
kubectl -n checkout rollout undo deploy/checkout-api
kubectl -n checkout rollout status deploy/checkout-api
```
Expected: "successfully rolled out", then 5xx falls within 5 minutes.
2. If one payment provider is failing, send its traffic to the fallback
provider: set the feature flag checkout.provider_fallback to on.
Expected: errors for that provider drop to zero within 2 minutes.
3. Do not restart the orders database or change its connection limits.
If its panel shows the connection pool exhausted, page the database on-call.
## Escalation
- Secondary checkout on-call: automatic after 5 minutes without acknowledgement.
- Database on-call: pager service "orders-db", for anything in the orders database.
- Payments lead: during business hours, or at any time for a SEV1.
## Verify recovery
The 5xx ratio stays below 0.1% for 15 minutes and the alert clears.
Post the budget spent (SLO overview › Budget remaining) in the incident channel.
## Background
- Request path: load balancer → checkout-api (Kubernetes, namespace checkout)
→ orders database (PostgreSQL) and two payment providers.
- Known failure modes: provider timeouts during provider maintenance; connection
pool exhaustion after a deploy that changes query plans.
- Design doc: [link] Provider status pages: [links]
## After the incident
- Postmortem required for SEV1 and SEV2, or if the incident spent more than 20%
of the 28-day error budget.
- Fix anything in this runbook that was missing, wrong or slow to follow.
What makes a runbook useful
- One entry per alert, linked from the alert. Put the link in the alert itself, for example in an annotation such as
runbook_url, so that the page leads straight to it. The SRE Workbook recommends a playbook entry for every alert. - Impact before diagnosis. Responders first need to know whether users are affected and how badly; that sets the severity.
- Safest, most reversible steps first. Rolling back, failing over or disabling a feature usually restores service faster than finding the root cause. Say when a step is not safely reversible: a Kubernetes rollback restores the application, not database or schema changes, and failing over to an asynchronous replica can lose recent writes.
- Say what success looks like. Give each step its expected output and a time limit, so the responder knows when to move on or escalate.
- Name what not to do. Actions that need another team, or that can make things worse, belong in the runbook as much as the safe ones.
- Make escalation explicit. Who to call, how, and when, including owners of dependencies.
General or step-by-step
The SRE Workbook describes this as contentious within Google. Some SREs keep entries general so they change slowly, for example one entry for all “RPC errors high” alerts read together with an architecture diagram. Others prefer step-by-step instructions to reduce variation between responders and recovery time. Whichever you choose, agree as a team on the minimum structured fields every entry must have; the template above is one starting point.
Keeping runbooks current
- Runbooks go out of date as fast as production changes. Ask on-call engineers to fix the entry whenever its alert fires and something was missing or wrong.
- Record an owner and a last-reviewed date, and review entries for alerts that fired in the weekly on-call handoff.
- Add “update the runbook” to postmortem action items when responders lacked instructions or found them wrong.
- Rehearse with game days or role-play exercises such as Google’s “Wheel of Misfortune”; a runbook nobody has followed is untested.
- Keep runbooks somewhere that still works during an outage, not only on the system they describe.
Automate the repeatable parts
If a runbook is a fixed list of commands that the on-call engineer runs every time an alert fires, automate it and keep the runbook for the judgment calls. Repeating the same manual fix is toil. For what still needs a human, writing it down pays off: Google’s SRE book reports that recording best practices in a playbook ahead of time gives roughly a threefold improvement in mean time to repair over improvising.
Sources: Google, Site Reliability Engineering, ch. 1 “Introduction” (playbooks and MTTR); The Site Reliability Workbook, ch. 8 “On-Call” (playbook contents, one entry per alert, maintaining playbooks); ch. 5 “Alerting on SLOs” (burn-rate parameters in the example); Kubernetes, Deployments: rolling back a Deployment.