The second system that fails is the channel
Part B left you mid-incident with the store coming back up. While you were reading the Activity Log, a second failure was building — not in the estate, in the channel. It fills with "any update?", and each one is a context switch for the person doing the work, each unanswered minute breeding two more. Nobody in the channel is trying to hurt you; they are frightened, and frightened people fill silence with imagination, and imagination during an outage runs dark. The fix is mechanical, and — this is the whole point of the part — it is the commander's job, not the debugging engineer's: a status update on a fixed clock, whether or not anything has changed.
"No change, next update 3:10" is a real update. It is the silence that costs you, not the lack of news, because a stated cadence tells everyone senior the one thing they actually need to know — that a process is running and they are inside it. The cadence buys the debugging engineer the only currency that matters mid-incident: uninterrupted minutes. That is why it is a role and not an afterthought. On a small team the commander sends the updates between mitigation calls; the moment the team is large enough, the work splits — one person owns the fix, another owns the words1. The engineer with their hands in the estate should never also be the one apologising to the VP.
- Commander
- Owns the incident: makes the mitigation calls and is the single decision-maker. On a Saturday it is whoever the alert paged, not the most senior person in the room.
- Comms lead
- Owns the words: sends the fixed-clock updates, shields the engineer from the channel, and translates raw findings into what each audience can act on. On a small incident, the commander wears both hats.
Two roles, one voice. The channel hears from the comms lead and only the comms lead; committees debug nothing and they broadcast even less. The failure mode this prevents is the one where three people post three half-true updates in four minutes and leadership, now certain the team is not in control, escalates — pulling the engineer off the fix to answer a director who was reassured by nobody. Fixed clock, single voice, and the estate gets its quiet back.
What a good update contains — by audience
Not everyone gets the same update, and getting that wrong is its own small disaster: the raw engineering log dumped on the board reads as panic; the four-sentence summary handed to the on-call engineer starves them of detail. Three audiences, three cadences, three contents. The table is worth committing to memory before any Saturday, because deciding "who am I writing for?" at 2:19 is a tax you pay in the currency you have least of.
| Audience | Cadence | What it contains |
|---|---|---|
| Incident channel | Continuous | The raw log — timestamps, findings, changes made, by whom. Written for Wednesday's postmortem as much as for now. |
| Leadership | Every 20–30 min, fixed | Four sentences: impact in business terms, what we know, what we are doing, when the next update comes. Never a promise you have not already tested. |
| Customers / status page | On state change | Honest, plain, no internals: "Checkout is currently failing; we are working on it" beats silence and beats fiction. Written by the comms lead, not by marketing at hour three. |
Note what the leadership row refuses to contain: an invented ETA. Under pressure a guessed time reads as a commitment, gets forwarded upward, and detonates when it passes unmet — the second update then opens in deficit, and trust spent there does not refund. The honest structure is a conditional: "the rollback completes in ten minutes; if it restores service we are done, if not the next path is failover and the next update is at 3:10." That sentence gives leadership what they are actually starved of — evidence there is a process — without borrowing against a future you do not control. Situation 01 below is this exact bind, live.
A guessed clock is a loan against your credibility.
The other line that never goes in a mid-incident update is a person's name. "The 2:07 deploy is the suspect" is a finding; "Priya's deploy broke it" is a verdict published before the trial, and it poisons Wednesday's postmortem four days early. Say what changed; never, mid-incident, say who. That discipline is not politeness — §3 is about why it is the load-bearing engineering choice in the entire class.
The blameless postmortem
Service is back. The incident is not over — it is over when the organisation has extracted the lesson, and whether that happens is decided by one design choice made long before Saturday: whether your postmortems are blameless. The logic is not kindness; it is engineering. If postmortems assign fault to people, people optimise for the postmortem — evidence gets thinner, timelines vaguer, "no idea what happened" becomes the safe answer — and the organisation goes blind exactly where it most needs sight.
Turn the rule around and it does real work. If the ground rule is that any failure a tired, competent human can trigger is a missing guardrail, then the incident's full anatomy comes out on the table, because nobody at the table is defending themselves.2 "Human error" stops being an answer and becomes the first question: a competent person did the reasonable-looking thing and the system let it through — so what let it through? The name of the person who shipped the change tells you nothing you can fix. Replace them with their equally competent successor and the same lane ships the same drift on their first tired Saturday. The guardrail is the finding; the human is just where the missing guardrail happened to surface.
- Blameless
- The ground rule that a postmortem's findings name systems, not people — because the goal is to retire a class of failure, and a name retires nothing. It protects mistakes made in good faith; it does not cover malice or a hazard ignored after the fix was known.
The document is one page and five headings, and the pattern is worth having by heart. It is a hang list, not prose, because each heading resists a different temptation.
- Impact
- Duration, users, money — Class One's native units. What makes leadership fund the actions rather than nod at them.
- Timeline
- Timestamped, lifted straight from Part A's running log, detection to resolution. The one honest witness; memory under adrenaline is fiction with confidence.
- Contributing causes
- Plural on purpose. Real incidents are a chain; "root cause" is usually just the last link anyone bothered to name. This is the heading that resists collapsing into one human's name — the hardest discipline in the document.
- What went well / got lucky
- The alert that fired; the fact it was a quiet Saturday and not Black Friday. Luck is a finding, because luck runs out.
- Actions
- Each with an owner and a date, reviewed like any other work. An action item without an owner is a wish.
The test of a good postmortem is brutal and simple: six months later, can this incident happen again? If yes, the meeting was theatre. And there is a second test, the one that decides whether the practice survives contact with power — does every sentence still retire the failure if you replace every human named in it? The finding that survives that swap is the one worth writing down.
The two numbers the postmortem hands forward
A good postmortem does not only close the last incident; it sizes the next one's defences. Somewhere in the Actions list, for an outage that touched data, sits an item that reads like housekeeping and is actually the hinge of everything that follows: agree an RPO and an RTO for each tier of service. Those two numbers are where the response half of this class hands the recovery half its brief. Everything in parts D, E, and F — backup policy, replication, the choice between a region pair and a zone — is bought to satisfy a number named here, at a desk, on a Wednesday, while nobody is frightened.
- RPO
- Recovery Point Objective — the maximum data loss you can tolerate, measured in time. An RPO of one hour says the last good recovery point may be up to an hour old; anything newer than that, you accept losing. It sets how often you must capture a recovery point.
- RTO
- Recovery Time Objective — the maximum downtime you can tolerate before the cost is unacceptable, again in time. An RTO of thirty minutes says recovery must complete inside half an hour. It sets how fast — and how rehearsed — the restore or failover has to be.
Read them together: RPO is how much of the past you are willing to lose, RTO is how long you are willing to be dark. They are not one number and they are not free — a tighter RPO buys more frequent backups and more storage; a tighter RTO buys a warm standby you pay for whether or not it is ever used. That is exactly why they are decided per tier and not once for the whole estate: checkout and the internal reporting dashboard do not deserve the same protection, and pretending they do either bankrupts you protecting a report or leaves the till exposed.3
Two numbers turn recovery into a budget.
Hold the definitions verbally for now; Part D draws them on a timeline, an outage marked in the middle with data loss falling to the left and downtime to the right, and the picture makes the trade-off obvious. The thing to carry out of this part is where the numbers come from: not a vendor default, not a gut feel, but an action item in a blameless postmortem, owned and dated like any other piece of work. An RPO nobody agreed is a number you will discover, badly, during the next incident.
Campux holds its first blameless postmortem
The Saturday incident from Part B is closed — twenty-three minutes, one change, one voice. The lesson is not extracted until the Wednesday meeting, and how that meeting is run is the whole difference between a fix and a trial. Campux has never held one of these before; the founding nine-hour outage, years earlier, ended with a shrug and a resolution to "be more careful", which is why it was always going to happen again.
This time the estate carries the loop, and the meeting carries the culture. It is where the abstract rule — systems not people — meets a real name in a real room, and where the two numbers that fund the rest of the class first get written down.
The Wednesday after the twenty-three minutes
The room holds the engineer who shipped the 2:07 hotfix, the commander, and a director who arrives believing the meeting is short: someone deployed on a Saturday without testing, name the name, done. The person running the meeting concedes the true half out loud — a change reached production without the check that would have caught it, and the document will say exactly that — then holds the ground rule as method, not mercy: the findings name mechanisms. There are two. The emergency lane, by design, skips the smoke test the standard lane enforces. And the production app settings drift from staging because they live outside the Bicep template. Two guardrails, not one culprit; the hotfix engineer presents the timeline personally, safely, because the rules held on the day power tested them.
The Actions list closes the loop and opens the next one. Settings move into IaC so drift is impossible (owner, date); the emergency lane keeps its speed but gains the ninety-second smoke test (owner, date). And the item that reaches past this incident into the rest of the class: agree an RPO and an RTO for each service tier — checkout, the storefront catalogue, internal reporting — owned by the reader, dated for the following week. The twenty-three-minute outage lost no data because the fault was a bad deploy, not a lost disk; the postmortem's honest question is what the numbers should have been if it had been the disk. Nobody at Campux can answer that yet. Parts D through F are the reader earning the answer.
The words are a role, not an afterthought
The incident is loud and you are not the one with your hands in the estate — you are the one holding the channel. You post on the clock: impact, what is known, what is being done, when the next update lands, even when the honest content is "no change". You keep the engineer's name for the deploy off the wire and out of the postmortem draft. When the meeting comes, you run it so systems take the blame and people keep working — and you make sure an RPO and an RTO leave the room with owners. Calm under fire is a skill; you practised it here.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored; this is between you and the page.
B — the silence is the cost, not the lack of news. A fixed-clock "no change" tells everyone senior the one thing they need: a process is running and they are inside it. A is the trap that feels responsible and detonates — the moment you go quiet, leadership fills the gap with imagination, and imagination during an outage runs dark; the "any update?" drumbeat starts, and each one is a context switch stolen from the fix. C manufactures a commitment you cannot keep and opens the next update in deficit. D collapses the two roles this part exists to separate: the engineer with their hands in the estate is precisely who must be shielded from the channel, not fed to it. Send the boring update on the clock.
RPO then RTO. RPO — Recovery Point Objective — is how much of the past you can afford to lose, measured in time: five minutes of orders here. RTO — Recovery Time Objective — is how long you can afford to be dark: thirty minutes. A inverts them, which matters because the two numbers buy different things — RPO buys backup frequency and storage, RTO buys a warm standby and rehearsal — so confusing them means funding the wrong defence. C erases the distinction that makes recovery a budget rather than a wish. D reaches for the availability vocabulary of a different class; SLAs are promises to customers, RPO and RTO are targets you set for recovery. Carry the pairing: RPO is the past you lose, RTO is the dark you sit in.
Impact, action-with-conditional, and the next-update time. Those three give leadership what they are actually starved of — evidence of a process — and the third one is the sleeper: a promised update time is what stops the "any update?" drumbeat that costs the debugging engineer their concentration. The rejects are the two classic self-inflicted wounds. An invented ETA feels kind and costs double: it gets forwarded upward as a commitment, and when it passes unmet, the second update opens in deficit — trust, once spent there, does not refund. And a name in a status update is a verdict published before the trial: it poisons Wednesday's blameless ground rules four days early, teaches every engineer watching that visibility equals exposure, and points at the wrong thing anyway, since the finding will be the guardrail that let a tired human ship it. Say what changed; never, mid-incident, say who.
# postmortem: checkout outage, Nov 14 — DRAFT
1. Impact: checkout unavailable 23 min; ~£95K of
carts abandoned; zero customer reports preceded
our detection.
2. Contributing causes: emergency deploy lane skips
the smoke test; production app settings drift
from staging because they live outside IaC.
3. Contributing causes (cont.): the incident was
caused by Priya rushing a hotfix without checking
the settings, contrary to team guidance.
4. Actions: settings into Bicep (owner: M.O., Dec 1);
smoke test added to emergency lane (owner: T.K.,
Nov 28); agree RPO/RTO per tier (owner: V.T., Nov 24).
Line three — and notice it adds no information; it only adds a defendant. Everything actionable in line three is already in line two: the lane skips the test, the settings drift. Adding Priya's name and "contrary to guidance" retires no failure mode — hire Priya's replacement and the same lane ships the same drift on their first tired Saturday. What the name does accomplish is corrosive and permanent: this document gets read by every engineer on the team, and each reads the same lesson — be invisible near incidents. The next timeline you collect will be vaguer, the next "what changed?" will meet more shrugs, and the organisation will have purchased one moment of accountability theatre at the price of its own eyesight.
The rewrite is mechanical, which is the skill this drill installs: delete the name, keep the mechanism — "the emergency lane permitted a deploy without the settings check the standard lane enforces." The distractors mark the opposite discipline: money figures are exactly what Impact is for (A — Class 1's native units, and what makes leadership fund the actions); naming the broken process is the entire point (B confuses blaming systems, which is required, with blaming people, which is banned); and the RPO/RTO item in line four is precisely the seam this part hands forward — owned and dated like any other work (D). The test for any postmortem sentence: does it still retire the failure if every human in it is replaced? Line three fails; line two is the finding.
The trap is the premise — that the only acceptable answer is a time. What the VP actually needs for the board is not a timestamp; it is evidence that the situation is controlled and a truthful thing to say upward. An invented ETA provides neither: it is a coin-flip that, if missed, converts "outage" into "outage plus a team that missed its own estimate" — and it will be missed, because forty minutes in with no root cause is precisely when estimates are fiction. The discipline is to refuse the number without refusing the person.
Send the structured answer, in business terms, in four sentences. What is true: checkout has been down since 2:08; the most likely cause was ruled out at 2:46 when the rollback did not restore service. What is happening: two paths are being worked in parallel — the configuration diff between staging and production, and failover of the checkout backend to the standby tier — with the first result expected within fifteen minutes. What the board can be told: "the team has a controlled process, has eliminated the primary suspect, and reports again at 3:20." And the commitment you can make honestly: the next update's time, which you will hit even if the content is "no change." A conditional plus a cadence is a professional answer; a guessed clock time is a loan against your credibility at the worst possible interest rate.
Then protect the machine that produces real answers. The moment a rollback fails is the moment pressure peaks and the Part B stampede risk returns — so the commander restates the rules in the channel: one change at a time, log everything, updates at :20 and :40. If the VP pushes again — they sometimes do — the sentence that ends it is Class 1's language, not engineering's: "Every minute is costing us about four thousand pounds, which is exactly why I won't spend the next ten of them manufacturing a number. You will have facts at 3:20." Executives respect the arithmetic of their own money; give them that instead of a guess.
A strong answer concedes before it argues. The director is not wrong that something failed and that accountability exists; steamrolling that instinct loses the room and the sponsor in one move. So concede the true half out loud: "You're right — a change reached production on a Saturday without the check that would have caught it, and this document will say exactly that." Then the pivot, stated as method rather than mercy: the document names mechanisms, because its job is to make this failure impossible, and the mechanism is fully identified without a name — the emergency lane permits deploys that skip the smoke test. Who approved it matters less than the finding that the lane exists and approving through it was normal; if guidance was truly violated, that is a management conversation with a manager, in private, on a different day — and saying that sentence plainly, in front of the room, is what gives the accountability question a legitimate home instead of denying it one.
Then price the alternative, because directors respond to consequences. If the name goes in the document, this is the last complete timeline the team ever produces: the next incident's log will be thin, the next "what changed?" will meet careful answers, and detection-to-mitigation time — the number that just fell from nine hours to twenty-three minutes — will climb back up for reasons no dashboard will show. Blameless is not a culture perk; it is the price of the data. The twenty-three minutes the director is proud of was purchased with exactly the ground rule they are about to break.
Close by redirecting the energy at the guardrails, with dates. "Here is what makes Saturday impossible to repeat: settings into IaC by December 1, smoke test in the emergency lane by November 28, and an agreed RPO and RTO per tier by next week — and I'd like your name on the review of all three." A director who arrived wanting a culprit and leaves owning a guardrail has been converted, not defeated — and the team watched the ground rules survive contact with power, which is the only demonstration of blamelessness that counts. The postmortem's authority was never the template; it was whether the rules held on the day someone senior tested them.
Five things worth carrying out of this part
- Two systems fail in every incident. The channel is the second one; a fixed-clock update — even "no change" — is what keeps leadership out of the engineer's way. Silence is what they fill with imagination.
- Split the roles the moment you can: a commander who owns the calls and a comms lead who owns the words. One voice in the channel; committees debug nothing and broadcast less.
- Three audiences, three updates: the raw log continuously, four sentences to leadership on the clock, plain honesty to customers on state change. Never an invented ETA; never a name mid-incident.
- Postmortems are blameless because blame poisons the data. "Human error" is the first question, not the answer: any failure a tired competent human can trigger is a missing guardrail. Five headings, plural causes, every action owned and dated.
- The postmortem produces the two numbers the recovery half spends: RPO, the past you can afford to lose, and RTO, the downtime you can afford to sit in — agreed per tier, owned and dated like any other action.
- On a two-person incident nobody splits anything — the same person mitigates and posts, and that is fine. The role split is a threshold, not a law: once the updates are stealing minutes from the fix, or once the audiences diverge enough that one message cannot serve all three, separate the words from the hands. Team sizes and titles vary; the principle — protect the person touching the estate — does not. ↩
- "Blameless" has an honest boundary: it covers competent people failing inside bad systems, which is nearly every incident — it does not cover malice, recklessness repeated after the guardrail existed, or covering up. Teams that pretend the boundary does not exist eventually meet a case that discredits the whole practice; teams that name it up front — "this protects mistakes, not misconduct" — keep the ground rules trusted precisely because they are known to have edges. ↩
- Treat any specific RPO/RTO figures in this part as illustration, not prescription — the right numbers fall out of what an hour of lost orders or an hour of downtime actually costs a given business, and that is Part F's tiering work, not a constant to memorise. What is settled is the shape: two independent numbers, priced against real loss, agreed per tier before the incident that tests them. ↩