What counts as an incident
Every class before this one assumed the systems were up. This one begins at 2:14 on a Saturday afternoon when they are not — and the first thing you need is not a fix but a definition, because a word everyone uses loosely becomes useless at the exact moment you need it precise. A disk running warm is not an incident. A nightly report arriving late is not an incident. Checkout returning a blank page in November is. The line between them is not severity yet; it is whether something a customer or the business relies on has stopped working in a way that was not planned. Get that line agreed in daylight and you spend the bad Saturday responding. Leave it vague and you spend the first twenty minutes arguing whether you are even allowed to wake anyone.
- Incident
- An unplanned degradation or outage of a service the business or its customers depend on — measured by impact, not by cause. A failed disk, an expired certificate, and a bad deploy are three different causes of the same thing: something that worked no longer does, and it was not on the calendar.
The word that does the work in that definition is unplanned. A maintenance window where you take checkout down at 3am with notice is not an incident; the same downtime arriving without warning at 2pm is. The distinction matters because the two summon opposite behaviour — the planned outage is a runbook you follow calmly, the unplanned one is a loop you run frightened. Everything in this part exists to make the second feel more like the first: decide what an incident is, decide how bad each one is, and decide how you will hear about it — all three settled before the pager ever goes off.
Those three decisions are the spine of this part. Define the thing (this section), size it (§2), and detect it (§4), with the loop that carries you from the alert to the handover in the middle (§3). None of it is glamorous, and all of it is the difference between an outage that becomes a headline and one that becomes a footnote. The engineer who has done this thinking in advance is the one the room turns to and finds already calm.
How bad is it — the severity ladder
The single worst question to be negotiating at 2:19, with someone senior typing "???" into the channel, is "how bad is this?". So you answer it in advance, once, with a ladder everybody has agreed to: a small fixed set of severity levels, each attached to a concrete response expectation. Severity is not a measure of how upset anyone is; it is a measure of blast radius and business impact, and its only job is to convert a panicked judgement call into a lookup. You read the symptom, you find the row, you get the response — who to page, how fast, and whether anyone is allowed to keep sleeping.
Campux's ladder has four rungs. The top three it settled the week it went live; the fourth it added once it had enough small annoyances to need a rung that means "real, but it can wait." The numbers and thresholds here are Campux's, not a law — a bank and a games studio would draw the lines differently — but the shape is universal: one ladder, few rungs, each with a response the on-call engineer does not have to invent.1
| Level | What it means | Example | Response expected |
|---|---|---|---|
| Sev1 | Revenue or data actively impacted; a core service is dark for everyone | Checkout returning errors in November | Page everyone the runbook names, now; declare an incident with one owner |
| Sev2 | Significant degradation, but a workaround exists or one part still works | Search is down while browsing and checkout hold | Page the on-call engineer; let the rest of the team sleep |
| Sev3 | Annoying and contained; no revenue or data at risk | A back-office report renders the wrong totals | A ticket, fixed in business hours; no page |
| Sev4 | Cosmetic or low impact; noticed but harmless | A footer link points at last year's returns page | Logged to the backlog; scheduled, not paged |
Two habits keep the ladder honest. First, severity can move: you open at your best guess and re-declare the moment scope changes — a Sev2 search outage that turns out to be taking checkout with it is a Sev1, said out loud, and the extra people get paged. Under-calling to avoid waking someone is the expensive mistake, because the cost of a needless page is one grumpy engineer and the cost of a missed Sev1 is the November afternoon. Second, the ladder lives where the paged person can read it at 2am — in the runbook, not in someone's head — because a severity scheme only you understand is not a scheme, it is a bottleneck with your name on it.
The loop you run under pressure
Under an outage, with revenue burning and the channel filling, your working intelligence drops — sharply, measurably, for everyone in the room. You cannot train that away. What you can do is what aviation and medicine did: replace improvisation with a loop so practised it survives the adrenaline. Incident response is four verbs in a fixed order — detect, triage, mitigate, learn — and most of the damage done in real incidents comes from running them out of order, usually by diving for the root cause while the store is still dark.
This part owns the first verb and the setup for the second; Parts B and C take the rest at full depth. But you hold the whole shape now, because the order is the discipline. Detect: the machinery notices before a customer does and pages a named owner. Triage: three questions in five minutes — what is broken, for whom, and what changed — producing a severity call and one owner. Mitigate: make it stop hurting by the fastest safe path, which is explicitly not the same as fixing it. Learn: days later, the blameless postmortem that retires the whole class of failure. The figure below is that loop, drawn once against the outage the rest of this class refights.
Detect first. Understand last.
Detection — hearing it before a customer does
The detect box is where an incident is won or lost before anyone has typed a command, because the clock does not start when you notice — it starts when the service breaks, and everything between those two moments is customers hitting errors you have not seen. That gap has a name interviewers know: mean time to detect, MTTD, and the honest target is zero minutes of customer reports. If detection arrives by an angry email or a tweet, that is the first postmortem finding, logged before the incident is even over — not because someone failed, but because the machinery that should have paged first did not.
That machinery is the Class Twenty-Eight-to-Thirty toolkit, wired ahead of time to do one job: turn a broken signal into a paged human. Three layers carry most of the weight. The availability test — a synthetic probe that calls checkout every few minutes from outside your network — fails twice and knows the store is dark before real traffic has finished timing out. Metric alerts watch the numbers that bend when something breaks — the failed-request rate, response latency, queue depth — and fire when they cross a threshold you set in daylight. And the action group is the wiring that decides what "fire" means: it pages a named on-call engineer, not a shared mailbox nobody reads, so the alert lands on a person with a pager and not in a folder.
Two disciplines separate detection that works from a wall of noise. First, alert on symptoms, not causes: page when customers cannot check out, not when CPU touches eighty per cent — high CPU is often a healthy service under load, and paging on it teaches the on-call to ignore the pager, which is the one failure mode detection cannot survive. Second, the dashboard is for the human, the alert is for the machine: the Class Thirty dashboard is where you look once the page has already woken you, its two most useful views bookmarked — the availability chart and errors-by-operation — so a frightened engineer at 2am reads, rather than hunts. Alert fatigue is the quiet killer here; every page that turns out to be nothing spends a little of the trust the real page depends on.
What being on call actually asks of you
All of this converges on one arrangement most beginners meet only in production: the on-call rota. Being on call means that for a defined window — a week, usually — you are the human the action group pages, and you have agreed to carry a device that can wake you and to respond within a stated time. It is not a promise to fix everything alone; it is a promise to be the first responder — to acknowledge, to triage, to start the loop, and to escalate when the severity or the scope outgrows you. The person who owns the incident on a Saturday is not the most senior engineer available; it is whoever the pager reached.
A healthy on-call arrangement is a system, not an act of endurance, and the difference is worth naming because it is a thing you should ask about in an interview. It has a written runbook so the paged person is not improvising the severity ladder at 2am. It has a real escalation path — a second name to call when the first cannot answer or the incident is beyond them — so the rota survives one person being asleep or on holiday. And it counts the cost: pages that fire and turn out to be nothing are not free, they are trust and sleep spent, and an on-call rota that burns its people is one that will page an empty chair within two quarters. The senior signal is not the willingness to be woken; it is having built the machinery so that you rarely are, and so that when you are, the page is real.
The week Campux decided what an incident is
The disk in the closet was found by a customer. That is the sentence that hangs over this week: nine hours dark, and Campux learned about it from a shopper who could not pay, because there was nothing watching and nobody to wake. Newly on the cloud, with the reader — Campux's first cloud engineer — holding the pen, the company spends a quiet Tuesday doing the three things this part teaches. It writes down what counts as an incident, so a slow report and a dark checkout are no longer the same word. It draws the four-rung ladder in Table 1, so "how bad is it?" is a lookup and not an argument. And it wires the availability test to an action group that pages a named on-call engineer, so the next fault reaches a person in seconds.
None of it is heroic and all of it is cheap — an afternoon of writing against a nine-hour hole. When the next fault comes, and it does in the very next part, the store does not learn of its own outage from a customer tweet; a probe fails twice, a pager goes off, and an engineer is triaging before the first shopper has refreshed the page. Campux has bought itself the one thing the closet never had: it now finds out first.
The pager finds a person, not a mailbox
You are on call for the first time, and the arrangement you inherited tells you almost everything about the team. There is a runbook with a severity ladder you can read half-asleep; there is a second name to ring when a Sev1 is beyond you; the alerts that fire are the ones that mean customers are hurting, not the ones that mean a graph twitched. When your pager goes off, it is because something real broke — and because someone built the detection so that you, not a customer, found out first. Ask about all of this in the interview; a team that cannot answer is a team where being on call means being afraid.
Examination
Four drills, then two situations. Write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B — mitigate first; understanding is Wednesday's job. The loop's order exists precisely for this moment, because A is what your engineering instincts scream for, and A has a price tag: every minute in the diff is a minute of outage, and the diff will read exactly the same on Monday. Run the tape on A — twenty minutes of code archaeology, a theory, a forward-fix written under adrenaline and deployed untested — and now there are two changes in production, one of them brand new. C is a mitigation, but a superstitious one: nothing points at instance state, and restarting everything destroys the evidence. D confuses the loop's last verb with its third. The interview version — "walk me through your first fifteen minutes" — gets hired when it has a rollback in it before it has a theory.
B. Severity is blast radius, not the presence of any fault. Search being down is a real, customer-facing degradation — that is why D is wrong; unplanned degradation of a service customers use is an incident even when a workaround exists. But checkout holds and shoppers can still buy, so it is not the November-afternoon emergency that justifies waking everyone the runbook names: over-calling this to Sev1 spends the whole team's trust and sleep on something one rested engineer can handle in daylight-adjacent hours. C under-calls it — a page-nobody Sev3 leaves a broken shopping experience unowned overnight. The point of the ladder is exactly this lookup: read the impact, find the row, get the response, and do not re-argue it at 2am.
The availability test, the named action group, and the symptom-based alert. Each turns a broken signal into a real page to a real person about something a customer feels. The two rejects are the same mistake wearing different clothes: both manufacture noise. High CPU is often a healthy service under load — page on it and you cry wolf until the on-call mutes the wolf. A page per log warning does the same at volume, and "filter it later" means the one real alert arrives buried in fifty false ones at 2am. Alert fatigue is the failure detection cannot survive: every needless page spends a little of the trust the real page depends on, and a muted pager detects nothing.
# campux on-call & alerting — v1
1. Availability test calls checkout every 5 min from
outside the network; fails twice -> fires.
2. Alert routes to an action group that pages the
named on-call engineer for the week.
3. Also page on CPU > 80% on any instance, so we
never miss a busy period.
4. Escalation: if on-call does not ack in 10 min,
page the secondary; runbook holds the ladder.
Line three. High CPU is frequently a healthy service handling a busy period — exactly the load Campux sees every November — so this rule pages the on-call at the busiest, most normal times, for nothing. Within a fortnight the engineer learns that the pager mostly lies and starts glancing at it late, and the one page that means checkout is dark arrives at a person who has been trained to wait. You alert on the symptom customers feel — the failed-request rate — not on an internal number that is often fine. The other lines are healthy: a two-failure probe filters flukes without much delay (A), a named owner with a ten-minute escalation to a secondary is a rota, not a single point of failure (B and D misread standard practice).
Acknowledge first, decide second. The first move is to ack the page — not because you have assessed it, but because acking stops the escalation clock from paging your secondary over something you are already looking at, and it tells the system a human has the ball. Only then do you triage. Ambiguity is not a reason to dismiss and not a reason to panic; it is a reason to gather one or two more facts cheaply before you spend anyone's sleep.
Read the fixed shortlist, not the whole console. The availability test is green, which is real evidence checkout is serving; the error bump is small. Check the two bookmarked views: is the error rate trending up or holding flat, and is it one operation or spread across many? A small, flat, single-operation blip that the synthetic probe cannot even feel is most likely noise or a transient — you watch it for a few minutes rather than declaring. A rate that is climbing, even from a low base, is an incident in its first minute, and you declare and start the loop before the probe catches up.
Whatever you decide, write it down and set a checkpoint. Log the time, what you saw, and your call — "2:11, errors flat on one operation, avail green, watching, re-check 2:25." If it was nothing, that line tunes the threshold so this alert is quieter next week; if it grows, you have a timestamped start for the postmortem. The judgement being tested is the refusal to treat ambiguity as a coin-flip: you do not go back to sleep on a hunch, and you do not wake five people on one either — you spend five cheap minutes buying certainty.
Concede the cost before you defend the page. Your colleague is right that waking someone at 3am is expensive, and if you argue otherwise you sound like you enjoy heroics. So agree: a page is a real cost, which is exactly why you page on almost nothing — only the handful of symptoms that mean customers cannot transact. The disagreement is not about whether to protect people's sleep; it is about which failures are allowed to interrupt it.
Price the difference between a page and an email. Checkout's failed-request rate spiking is not "everything else" — it is revenue leaving in real time, at November prices measured in thousands of pounds an hour. An email that gets read at nine in the morning turns a twenty-three-minute incident into a six-hour one, and the company learns of its own outage the way it learned about the disk in the closet: from a customer who could not pay. The page exists to make mean-time-to-detect near zero for the one thing that must never wait. Everything genuinely lower — a slow report, a cosmetic bug — already goes to a ticket, exactly as they are proposing.
Reframe the ask as tiering, not paging-versus-email. The answer that ends the debate is that you already agree with the principle and have encoded it: the severity ladder is precisely the mechanism that sends Sev3 and Sev4 to the morning queue and reserves the page for Sev1. So the real question is only "is checkout being down a Sev1?" — and if the business will not fund a 3am page to protect its own checkout, that is a decision to make in daylight, in writing, not a default to reach by muting the one alert that pays for itself.
What Part I leaves you holding
- An incident is unplanned degradation of something the business or customers depend on — defined by impact, not cause. Agree the word before the Saturday, not during it.
- Severity is a lookup, not a debate: one ladder, few rungs, each with a response. Campux runs Sev1 to Sev4, and severity can move the moment scope changes.
- The loop is four verbs in order — detect, triage, mitigate, learn. Most incident damage comes from running them out of order, especially chasing the cause while the store is dark.
- Detection is machinery, not luck: an availability test, symptom-based alerts, and an action group that pages a named human. If a customer tells you first, that is a finding.
- Alert fatigue is the one failure detection cannot survive. Page on symptoms customers feel, never on internal causes that are often fine — a muted pager detects nothing.
- Treat the specific Sev1–Sev4 numbers, examples, and thresholds here as Campux's, not gospel — the direction is settled, the exact lines move by organisation. Sev1–Sev4 is general incident-management and SRE convention; note that Microsoft's support plans use a separate A/B/C severity for raising cases with Microsoft, which is a different scale for a different purpose. Agree your own ladder and write it in the runbook. ↩