What changed? — the question that finds most outages
The store was fine at eleven, and it is dark at eleven-fifteen. Between those two clock readings something moved, and the fastest route to the fault is almost never the code and almost always the calendar. Systems that ran untouched for a week do not usually break on their own; they break because a human or a pipeline changed them — a deploy, a config edit, a scaled instance count, a role reassignment, a certificate that quietly reached its expiry. So the sharpest question in triage is not "what is wrong?" but its blunter cousin: what changed in the last hour? Ask that first, and a startling share of incidents name themselves before you have read a single line of code.
This is the part of the loop that separates the engineer who flails from the one who is done in twenty minutes. Flailing looks like opening the application code and reading, hunting for the bug on the assumption that the software spontaneously went wrong. The disciplined move is to treat the outage as a suspect list of recent changes and work backwards from the moment the error rate bent. You already have the two instruments for it — the telemetry from Class Thirty that tells you when it bent, and a ledger, kept automatically, that tells you what changed just before. Line those two up and the cause usually falls out.
The instrument that answers "what changed" at the infrastructure layer is the Azure Activity Log: the subscription's record of every control-plane operation — who did what, to which resource, when, and from where.1 It is the ARM layer from Class Sixteen writing its diary, and it does so whether or not you ever configured a thing. At 2:20 on a Saturday, filtering it to the last two hours is worth more than any dashboard you built on a calm afternoon.
Reading the ledger — control plane versus data plane
The Activity Log records management operations — creates, updates, deletes, role assignments — the things done to your resources through Azure Resource Manager. It does not record what your application did with a request; that is the data plane, and it lives in the Class Twenty-Eight telemetry, collected only because you wired diagnostic settings before the incident. The distinction is the whole of triage in one line: the Activity Log tells you a deploy landed at 2:07; App Insights tells you checkout started returning 500s at 2:08. One names the change, the other times the wound.
- Activity Log
- The subscription's control-plane record — every resource create, update, delete, and role assignment, with the identity that made it, the timestamp, and the source. Collected automatically, kept for ninety days, and impossible to alter or delete.
Three facts about it decide whether it saves your Saturday. First, it is automatic: unlike resource logs, you configure nothing to have it — it was recording your estate before you knew you would need it. Second, it names the who: each entry carries the identity and the source address, so "the deploy at 2:07 came through the emergency lane, run by the on-call pipeline" is a fact you can read, not a theory you float. Third, it forgets: entries are retained for ninety days and then deleted, so an incident you want to study a quarter later must have been exported — into a Log Analytics workspace, where Class Twenty-Nine's KQL can join changes against errors — before the window closed.2
Under pressure you do not browse the whole log; you read a fixed shortlist of change sources against the bend time, and stop at the first that lines up. The list below is that shortlist, in the order a Saturday actually asks for it.
- Activity Log, last hour
- The control-plane diary: deploys, setting changes, scale operations, role edits — with who and when. The first place you look, because it is automatic and it names the change.
- Deployment history
- The pipeline's own record from Class Twenty-Two — what shipped, from which commit, approved by whom. Cross-check it against the Activity Log entry to see exactly which build is live.
- App telemetry, the bend
- The failed-request chart and one errors-by-operation query fix the minute it broke and which operation broke. This is the when you align every candidate change against.
- Change history
- For a suspected resource, the Activity Log's change view shows the property values before and after an operation — the config edit's actual diff, not just that one occurred.
Find when it bent, then find what changed just before. That pairing resolves more incidents in five minutes than any amount of code-reading resolves in an hour — which is why engineers who cannot write KQL from memory still survive Saturdays: the two views they need are bookmarked, and the Activity Log needs no query at all.
Mitigate first, root-cause second
Here is the instinct this section exists to overwrite. You have found the suspicious deploy, and every engineering cell in you wants to know why it broke — open the diff, read the code, understand. Meanwhile the store is dark at hundreds of pounds a minute, November Saturday prices. The professional move is the one that feels like surrender: reverse the change without understanding it. Understanding is Wednesday's work, at a desk, with coffee. Saturday's work is to make the harm stop by the fastest reversible path and let the revenue flow again while the cause waits, harmless, for a calmer day.
Saturday restores service. Wednesday explains it.
Phase Three built you an armoury of these fast reversible fixes, and mitigation is mostly the discipline of reaching for the right one. If a deploy broke it, the slot swap puts the last known good version back in seconds, or the pipeline redeploys the previous build. If load broke it, the scale set adds instances. If a new code path broke it, the feature flag turns that path off without a deploy at all. If a whole dependency broke it, you fail reads over to a replica so the site degrades instead of dying. None of these requires you to know the root cause — that is exactly their value. They convert "I understand the bug" from a prerequisite into a luxury you buy back later.
One rule guards the whole manoeuvre: mitigation must be reversible and singular. One change at a time, watch the metrics, write it in the log. The classic way a routine outage becomes a long one is the mitigation stampede — three people changing four things at once, no record, and now the estate sits in a configuration nobody has ever seen, with the original fault still in it somewhere. The Class Twenty lesson holds at 2:30 exactly as it held in the pipeline: state you cannot explain is worse than a fault you can see.
The levers — and when to reach for each
A mitigation lever is only as good as your certainty about when it applies. Reaching for a scale-out when the fault is a bad deploy wastes the minutes you do not have; swapping a slot back when the real problem is a dependency outage in Microsoft's half changes nothing and costs you a restart. So the levers are not interchangeable, and the senior skill is matching the lever to the shape of the change the Activity Log just showed you. The table below is that matching, held so it survives the adrenaline.
| Lever | Reach for it when | What it does, and the catch |
|---|---|---|
| Slot swap back | A deploy is the change, and it went out through App Service slots | Swaps the previous build back into production in seconds — the last known good site. The catch: the same swap in reverse is the undo, so know which slot now holds the good version. |
| Pipeline redeploy | A deploy broke it but there is no slot to swap | Re-runs the previous known-good release from Class Twenty-Two. Slower than a swap — minutes, not seconds — but always available. |
| Feature flag off | A new code path is the suspect, shipped dark or behind a toggle | Disables the path with no deploy at all. The catch: only works if the change was actually put behind a flag before it shipped. |
| Scale out | The change is load, not code — a traffic spike, a slow dependency saturating instances | Adds instances to absorb the pressure. Buys time; does not fix a bug, and costs money for every minute it runs. |
| Fail reads to a replica | A primary datastore is degraded but a read replica is healthy | Serves reads from the replica so the site degrades instead of dying. The catch: replica lag means slightly stale data — acceptable for browsing, rarely for checkout. |
Notice what every row shares: a way back. Each lever is chosen precisely because it can be undone if it makes things worse, which is what lets you pull it before you fully understand the fault. That is the difference between a mitigation and a gamble — a gamble you cannot reverse, and at 2:30 an irreversible change is the one move you are never allowed to make.
Four minutes, one swap, revenue back
2:14pm, and checkout starts returning 500s. The Class Thirty availability test has already paged the on-call engineer; no customer reported it first. Triage is quick: checkout only, all regions, and the bookmarked Class Twenty-Nine query shows failures bending sharply at 2:08. The commander opens the Activity Log, filters to the last hour, and reads one change in the window — a storefront deploy at 2:07, a cosmetic hotfix pushed through the emergency lane. That is enough. Nobody opens the diff.
2:26, the §3 call: swap the slot back. The previous build is still warm in the staging slot from the deploy minutes earlier, so the swap is near-instant — production and staging trade places, the last known good version returns to the live URL, and by 2:30 the error rate is at baseline and the availability test is green. Four minutes from decision to recovery, against nine hours the last time a faceless failure hit Campux. The why — an app setting that existed in staging but not production — waits for Wednesday's postmortem, where it belongs. Saturday restored service; nobody had to be brave to do it.
When the fix does not fix it, and when to stop
Sometimes the first lever fails. You swap the slot back and checkout still returns 500s. The dangerous instinct now is the same one from §3 in a new costume: to abandon mitigation and finally start reading code, on the reasoning that "the obvious fix did not work, so I must understand this properly." Resist it. A failed mitigation is information, not a verdict — it tells you the deploy was probably not the whole cause, and the answer is to work the next candidate change on the list, not to stop mitigating. The suspect list from §2 has more than one entry precisely for this moment.
But there is a real end to the mitigation phase, and knowing it is a senior skill. When the reversible levers are exhausted and the fault points at the platform itself — a region degraded, a managed service returning errors across every tenant — you stop reaching for your own controls and open a support case, then check the Azure status feed. This is Class One's line, remembered mid-fire: some of the stack is Microsoft's half, and no amount of your adrenaline fixes their region. Recognising that boundary early is not defeat; it is the difference between waiting out a provider incident with your customers informed and burning an hour changing your own healthy configuration in search of a fault that was never yours.
Everything in this part is one habit rehearsed: read the change ledger, reverse the change, hold the reversal to one reversible step, and know the moment your levers stop applying. Do that and the outage is a footnote. Skip straight to understanding and it is a headline — the same nine-hour headline Campux already survived once, and never intends to write again.
The engineer who reaches for the calendar first
Checkout is down and a manager is typing "???" into the channel. You do not open the code. You open the Activity Log, filter to the last hour, and read the one change that lines up with the bend in the telemetry — a deploy nine minutes ago. You swap the slot back, watch the error rate fall to baseline, and post the update. Total time: four minutes. Only then, with the revenue flowing again, do you start asking why the deploy broke — because on Saturday you restore service, and on Wednesday you explain it. That order, held under pressure, is the whole of this part.
Examination
Four drills, then two situations. Write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. Mitigate first, root-cause second. The single most expensive instinct in incident response is the urge to understand before acting — A is that instinct, and it keeps customers down for the full length of your investigation, priced in thousands of pounds on a November Saturday. A rollback or a slot swap restores service in minutes without any understanding at all, which is exactly its value. C outsources the same delay to someone else; D confuses Wednesday's job with Saturday's. Understanding is a luxury you buy back once the revenue is flowing again, not a prerequisite for stopping the bleeding.
The Activity Log — Class 16's control plane, writing its diary whether you asked or not. Every ARM operation — create, update, delete, role assignment — lands there with identity, timestamp, and source, no configuration required; at 2:20 on a Saturday it is the "what changed" answer in three clicks. The distractors each miss a plane: App Insights sees the application's behaviour (the bend, not the cause) — a deploy appears there only as its consequences; the workspace holds what Class 28 taught you flows only after you say so — and if you did wire the Activity Log into it, better still, because then Class 29's KQL can join changes against errors, which is the grown-up version of this drill. Advisor is a weekly gardener, not a flight recorder. The pairing to memorise: telemetry tells you when it bent; the Activity Log tells you what changed just before. One without the other is half a diagnosis.
The slot swap, the feature flag, and the replica failover. Each is fast, each is reversible, and none requires you to understand the fault — the three properties that define a safe mitigation. The two rejects are the anti-patterns. Deleting and rebuilding the resource group is irreversible and destroys the very evidence Wednesday's postmortem needs; at 2:30 an irreversible change is the one move you are never allowed to make. Editing three resources at once is the mitigation stampede — no record, several variables in flight, and now the estate is in a configuration nobody has ever seen with the original fault still buried in it. Mitigation is singular and reversible or it is a gamble.
# triage runbook — checkout is down
1. Confirm the symptom and scope from telemetry;
note the minute the error rate bent.
2. Filter the Activity Log to the last hour and
list every change against that bend time.
3. Open the suspected deploy's diff and read it
fully; do not touch anything until I understand
exactly why it broke.
4. Once the cause is understood and reversed,
post the leadership update.
Line three. It root-causes before it mitigates, which is the exact inversion this part exists to prevent. Once the Activity Log has named the suspect change, the next move is to reverse it — swap or redeploy — not to read the code; the store stays dark for every minute of that reading. Lines one and two are healthy: the bend time from telemetry and the change list from the Activity Log are the correct diagnostic pair, and the Activity Log is automatic — it needs no diagnostic settings, so B is simply wrong about how it works. Line four is fine; a leadership update naming a suspected cause and the action in progress is exactly right. The fault is the order, and the order is the discipline.
Read the failure as evidence, not as permission. The swap not restoring service tells you something precise: the deploy was probably not the whole cause. That is useful — it narrows the field. What it is not is a licence to abandon the discipline and start reading code, because the reason to mitigate before diagnosing has not changed one bit: customers are still down, and understanding still takes longer than reversing.
Work the next candidate, not the debugger. The suspect list from the Activity Log had more than one entry for exactly this moment. Go back to it — was there a config change, a scale event, a certificate, a dependency shifting — and try the next reversible lever against the next most likely change. You are still in the mitigate loop; you have simply moved to the second suspect.
Know where the loop actually ends. Mitigation stops when the reversible levers are exhausted and the fault points at the platform — a degraded region, a managed service erroring across tenants. That is when you open a support case and check the Azure status feed, not when the first fix misses. The end of mitigation is a boundary you recognise, never a mood you give in to.
Agree with the goal, split the timeline. They are right that the bug must be found and fixed once — you concede that immediately, or you sound like you want to paper over it. The confusion is between two clocks. Restoring service and fixing the bug are different jobs on different schedules: one is measured in minutes with revenue burning, the other in careful hours at a desk. Rolling back does not hide the problem; it stops the harm so you can study the problem safely.
Price the difference out loud. "Find the bug first" means every customer stays down for the length of the investigation — an unknown number of minutes at November-Saturday prices. Reversing the change restores service now and loses nothing: the failed deploy, the logs, the Activity Log entry are all still there for Wednesday. We do not trade the fix away by mitigating; we buy the calm to do the fix well.
Offer the sequence, not the standoff. "Roll back now, keep the broken build and every log, and I will book the root-cause session for Wednesday with all of it preserved." That gives the senior engineer exactly what they want — a real fix that stays fixed — without holding customers hostage to the diagnosis. It reframes the choice from mitigate-versus-understand into mitigate-then-understand, which is the whole discipline in one sentence.
The part in five lines
- The sharpest triage question is not "what is wrong?" but "what changed in the last hour?" — most outages are a change, not a spontaneous fault.
- The Activity Log answers it: the subscription's control-plane record of every operation, with who, what, when, and from where — automatic, kept ninety days, unalterable.
- Diagnosis is two questions joined: telemetry says when it bent; the Activity Log says what changed just before. Align them and the cause usually falls out.
- Mitigate first, root-cause second. Reverse the change by the fastest reversible path — slot swap, redeploy, feature flag, scale out, replica read — and understand it later.
- Every mitigation is reversible and singular, or it is a gamble. Stop mitigating when the fault points at the platform, not when the first fix misses.
- The Activity Log is scoped to the subscription and captures control-plane (management) operations only; data-plane actions — a read from a key vault, a query against a database — are not recorded there and require resource logs with a diagnostic setting. ↩
- Ninety days is the current default retention, and entries typically become available for analysis within three to twenty minutes of the event. Treat the exact figures as movable — Microsoft adjusts them, and your organisation may have exported the log to a workspace with far longer retention. Verify the current numbers on Microsoft Learn before you quote them in an interview. ↩