Incidents Updated
Incident severity levels
A severity scale tells everyone how big an incident is, so they know who to wake up, how often to communicate and what follow-up is required. There is no universal standard; most organisations use three to five levels. Below is a four-level starting point that you can adapt.
Severity matrix
| Level | Impact | Examples | Response | Updates | Postmortem |
|---|---|---|---|---|---|
| SEV1 Critical |
Core functionality unavailable or badly degraded for most users; data loss or security breach; major revenue or legal exposure. | Site down; checkout failing for everyone; customer data exposed; corruption spreading. | Page immediately, 24/7. Dedicated incident commander. All needed teams pulled in. | Internal every 30 min; status page and customer comms; executives informed. | Required, reviewed. |
| SEV2 Major |
Significant degradation for many users, or a core feature down for a subset; no workaround or a painful one. | One region down; logins slow for 20% of users; payments failing for one provider. | Page immediately, 24/7. Incident commander assigned. | Internal every 60 min; status page if customers notice. | Required. |
| SEV3 Minor |
Limited impact: a non-core feature broken, or degradation with an easy workaround; error budget burning slowly. | Exports delayed; an admin page erroring; elevated errors on a low-traffic endpoint. | Business hours by the owning team; on-call may acknowledge out of hours. | In the incident channel or ticket as it progresses. | Optional; recommended if novel. |
| SEV4 Low |
No current user impact and no imminent risk; a cosmetic problem or a risk with time to fix it. | Certificate expiring in two weeks; a typo in an error message; one of several redundant nodes lost, with capacity to spare. | Normal ticket queue. | Ticket. | No. |
The update cadences and examples are starting points. Tune them to your product, contracts and team size, and write the final version down where on-call can find it in seconds.
How to classify
- Classify by impact, not cause. A one-line config typo that takes the site down is a SEV1; a dramatic-looking cluster failure with no user impact is a SEV4.
- When in doubt, go higher. Downgrading is cheap; under-reacting to a growing incident is not.
- Severity can change. Re-evaluate as you learn more, and announce changes in the incident channel.
- Severity is not urgency. Severity describes impact. Losing the last spare replica or a backup has no user impact yet but needs action now, because the next failure will; respond to the risk, not only the current state. The priority of follow-up work is a separate decision again.
- Make it measurable where you can. Tie levels to SLO burn rate (for example, a fast-burn burn-rate page is at least SEV2) or to the share of users affected.
- Declare early. Opening an incident is free; a team that hesitates to declare will respond late.
Incident roles
For SEV1 and SEV2, split responsibilities so that no one person both debugs and coordinates:
- Incident commander (IC)
- Owns the incident: sets priorities, assigns work, decides on escalation and severity. Does not debug.
- Operations or technical lead
- Leads the hands-on investigation and mitigation, often with other responders.
- Communications lead
- Writes internal updates, the status page and customer messages on the agreed cadence.
- Scribe
- Keeps a timestamped log of actions and decisions; this becomes the postmortem timeline.
In a small team one person may hold several roles, but the IC role should be explicit and handed over explicitly, for example “I am handing incident command to Sam”.
Response flow
- Detect: an alert, a customer report or an engineer notices.
- Declare: open an incident with a provisional severity and a channel.
- Mitigate first: roll back, fail over, shed load or disable a feature, following the alert’s runbook where one exists. Restoring service comes before finding the root cause.
- Communicate: on the cadence for the severity, even if there is nothing new.
- Resolve: confirm user impact has ended, then close the incident.
- Learn: write a blameless postmortem and track action items.
Further reading: Google, Site Reliability Engineering, ch. 14 “Managing Incidents”; The Site Reliability Workbook, ch. 9 “Incident Response”.