Two things you upgrade, and why it never ends
Every cluster has two surfaces that age, and confusing them is a common early mistake. The first is the Kubernetes version — the control plane and the kubelet on each node, moving together from 1.29 to 1.30 to 1.31. The second is the node image — the operating system on the node VMs, which needs security patches for the same reason any Linux box does, whether or not Kubernetes itself moves. You can patch the node image on Tuesday without touching the Kubernetes version, and you often should. They are two different clocks, and a mature team watches both.
The reason this is ongoing rather than a one-off is the release train. Kubernetes ships roughly three minor versions a year, and each version is supported for only a window before it falls off the back — AKS will eventually stop giving you security fixes for a version that has aged out, and can force an upgrade to keep the cluster in a supported state. So "we upgraded last year" is not a finished task; it is a lap. Plan for upgrading as a standing rhythm, not an event, and the day a version goes end-of-support arrives as a calendar entry you already knew about rather than a surprise.
Automating the rhythm — channels and maintenance windows
Doing every upgrade by hand does not scale past one cluster, so AKS gives you auto-upgrade channels: you subscribe the cluster to a cadence and it keeps itself current. The node-image channel takes the latest patched OS image as it becomes available; the patch channel takes Kubernetes patch releases within your minor version; stable and rapid move you across minor versions on different appetites for newness. Pick the channel that matches how much change you can tolerate, and the cluster stops being a thing you must remember to feed.
An upgrade that lands whenever Azure feels like it is not something you want on a Friday afternoon, so you pair the channel with a planned maintenance window — a recurring slot, say Sunday between 02:00 and 06:00, when disruptive changes are allowed and never outside it. The channel decides what to apply; the window decides when. Together they turn upgrades from an interruption you dread into a background process that touches the cluster only during hours you chose, when the fewest customers are watching and the most engineers are asleep.
The channel picks what. The window picks when.
Surge upgrades — adding a node so you never lose capacity
An upgrade replaces nodes: a node is cordoned so nothing new schedules on it, its pods are drained onto other nodes, and the machine is rebuilt on the new version. The naïve way to do that is to take a node out first — but then, for the length of the rebuild, your cluster is running one node short, and if it was near capacity you have just self-inflicted a shortage in the middle of a maintenance window. Surge upgrades fix this by working the other way round: they add a temporary extra node on the new version first, move work onto it, and only then drain and remove the old one. Capacity is held whole across the whole operation.
You control this with max surge — how many extra nodes the upgrade may add at once, as a count or a percentage of the pool. A larger surge upgrades faster because it rebuilds more nodes in parallel, at the cost of briefly renting more VMs; a surge of one is slow and cheap. The default is deliberately conservative. For a production pool you usually raise it so upgrades finish inside the window rather than crawling past it — a small, temporary bill to buy a faster, safer rollout.
Watching it without building the watchtower
You need to see logs, metrics, and dashboards, and the old reflex is to self-host that stack — run your own Prometheus, nurse your own Grafana, keep the log pipeline alive. On AKS you do not have to, and mostly should not. Azure gives you a managed trio that plugs into the cluster and is operated for you, so the observability layer is not one more thing on your pager. You wire it on at cluster create or after, and the pods, nodes, and control plane start reporting into tools you did not have to keep alive.
The division of labour is worth holding in your head. Container Insights collects the logs and the coarse metrics and lands them in Log Analytics, where the query language from Class Twenty-Eight already lives. Managed Prometheus scrapes the fine-grained time-series metrics in the format the Kubernetes ecosystem speaks — the same metric names your app and the community dashboards already emit. Managed Grafana draws it, using the dashboards the world already publishes for Kubernetes rather than ones you invent. Three services, one job each; the seam between them is clean.
| Service | Collects | You reach for it when |
|---|---|---|
| Azure Monitor Container Insights | Logs and coarse metrics, into Log Analytics | You need a pod's logs, or to query across the cluster in KQL |
| Azure Monitor managed service for Prometheus | Fine-grained time-series metrics, Prometheus format | You want per-second signal to alert on, or the metrics community dashboards expect |
| Azure Managed Grafana | Nothing — it draws the other two | You want dashboards and a wall of graphs without running Grafana yourself |
The bill is the nodes — and how to shrink it
Return to the fact from the hub: the control plane is free on the Free tier, and the nodes always bill. That single sentence tells you where every cost decision lives. You are not paying for Kubernetes; you are paying for virtual machines that run whether or not any pod is doing useful work on them. So cost control on AKS is almost entirely the discipline of not running more node than you need, for not one minute longer than you need it.
The levers are few and they compound. Right-size the pools so a node's CPU and memory roughly match what the pods actually request — an oversized VM bills for headroom nobody uses. Turn on the cluster autoscaler so the pool grows under load and, more valuably, shrinks when the load leaves. Put interruptible work — batch, CI, anything that can be restarted — on Spot node pools, which take Azure's spare capacity at a steep discount in exchange for the risk of eviction. And when a user pool has nothing to do, scale it to zero: the nodes are deallocated, the meter stops, and the pool spins back up when work arrives. Watch the whole thing in Cost Management, which Class Thirty-Two makes routine, so a pool someone left large shows up as a line on a report rather than a shock on the invoice.
The partner cluster learns to patch itself and sleep
The partner-integration cluster has been running for weeks, and now you make it boring to operate. You put it on the node-image auto-upgrade channel so the OS on its nodes stays patched without anyone remembering to do it, and you pin the disruption to a planned maintenance window on the weekend — the partner's traffic is a weekday integration, so a Sunday-morning node rebuild touches no one who matters. Managed Prometheus is already scraping the cluster, so you add one alert that pages if the gateway's pods stop being ready, because a security patch that quietly breaks the workload is worse than no patch at all.
Then you look at the bill. The integration does nothing between about 8pm and 6am, yet its user node pool was billing all night for VMs serving no requests. You set that pool to scale to zero when idle, so overnight the nodes deallocate and the meter stops, and the pool comes back before the partner's first morning call. The system node pool stays up — the cluster's own plumbing has to keep running — but the workload nodes now cost money only during the hours the contract is actually live. The cluster is still exactly the size of the contract; you have just stopped paying for the hours the contract is asleep.
The version end-of-support email arrives
Azure emails that your Kubernetes version goes end-of-support in six weeks. You do not upgrade in a panic on the live cluster. You read the release notes for the versions between here and current, upgrade a non-production cluster first and watch the workloads, then raise max surge on the production pools so the rollout fits the maintenance window, and let the change land inside that window rather than at noon. The upgrade is uneventful precisely because you treated a known deadline as scheduled work, not an emergency — which is the whole difference between operating a cluster and being operated by one.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. A surge upgrade adds an extra node on the new version first, moves work onto it, and only then drains and removes the old one — so the working node count never dips below where it started. A is a different tool: the autoscaler reacts to pending pods under load, not to the upgrade's own node churn, and cannot be relied on to cover the exact gap a rebuild opens. C only decides when the upgrade runs, not whether capacity holds during it — a well-timed shortage is still a shortage. D decides what version to apply, not how to apply it without losing nodes. The tell is "keeps from running short": that is the one thing surge exists to do.
B. Scaling a user pool to zero deallocates its nodes, the node bill stops, and the pool comes back when work returns — precise, reversible, and it leaves the system pool and control plane untouched. A throws away the whole cluster and its configuration nightly to save on nodes you could have simply parked; the rebuild risk dwarfs the saving. C still runs VMs all night, just smaller ones — a smaller bill, not a stopped one, and it also starves the pool when work does arrive. D misapplies Spot: Spot suits interruptible work that tolerates eviction, not a strategy for handling idle hours, and you cannot put a system pool on Spot anyway. Idle means zero, not smaller.
The two surfaces, channel-versus-window, and Grafana-draws-nothing. The two rejects invert the facts that matter most for the bill and the calendar. Kubernetes ships about three minor versions a year and ages old ones out of support, so upgrading is a standing lap, never finished — assuming otherwise is how a cluster drifts into an unsupported, unpatched state. And the control plane is free on the Free tier while the nodes always bill, so shrinking pools is nearly the entire cost lever, not a rounding error. Get those two backwards and you will both miss a forced upgrade and overpay for idle VMs.
# ops plan: partner-gw cluster
1. Subscribe the cluster to the node-image channel so
the OS stays patched automatically.
2. Pin upgrades to a Sunday 02:00-06:00 maintenance window.
3. Put the always-on gateway pods on a Spot node pool to
save money, since Spot is just cheaper VMs.
4. Scale the user pool to zero on the nights it is idle.
Line three. Spot is not merely "cheaper VMs" — it is Azure's spare capacity, offered at a discount precisely because Azure can reclaim it with little notice when it needs the hardware back. That trade is right for interruptible, restartable work: batch jobs, CI, the pos-batch style of load. It is exactly wrong for an always-on gateway that the partner contract depends on, because the day Azure evicts the node your integration goes down for reasons that look like nothing you did. The discount is real; so is the eviction, and putting availability-critical pods on Spot swaps a predictable bill for an unpredictable outage.
The other lines are sound. The node-image channel does keep the node OS patched (A is wrong). Maintenance windows are designed to recur weekly on a slot like this (B is wrong). And a user pool absolutely can scale to zero to stop the meter when idle — that is a headline cost lever, not an impossibility (D is wrong). The habit to carry: a discount that comes with a reclaim clause is only a saving on work that can survive being reclaimed.
The instinct is right; the "forget forever" is the trap. Automating upgrades is correct — doing them by hand does not scale, and a channel keeps the cluster from drifting into an unsupported state. So concede that first. The error is treating "automated" as "no longer my concern." An upgrade is still a change that lands on a running workload, and the fastest channel maximises how often and how disruptively change arrives, which for a workload someone depends on is the opposite of what you want.
Match the channel to your tolerance, and cage it in a window. Pick the channel by how much change the workload can absorb — a node-image or patch channel for something you want stable, not the most aggressive minor-version channel. Then pin it to a planned maintenance window so change only ever lands in hours you chose, when few customers are watching. The channel decides what; the window decides when; you have deliberately given up neither.
Then keep a way to notice it went wrong. Wire an alert — managed Prometheus is right there — that pages if the workload's pods stop being ready. A silent auto-upgrade that quietly breaks the app is worse than a manual one you watched, because nobody is looking. "Automate the toil, keep the visibility" is the whole answer: the channel removes the remembering, the window removes the surprise, the alert removes the silence.
Correct the premise: the bill is the nodes. Start by naming what money actually buys here. The control plane is free on the Free tier; what you pay for is virtual machines that bill whether or not any pod on them is doing useful work. "Make Kubernetes cheaper" is really "run less node-hour," so every lever is some form of not renting capacity you are not using. Saying this first stops the conversation from chasing phantom Kubernetes licence costs that do not exist.
Then walk the levers in order of payoff. First look for idle: a user pool doing nothing overnight should scale to zero, not bill until morning. Next look for oversizing: if nodes are half-empty against what pods request, right-size the pool or turn on the cluster autoscaler so it shrinks when load leaves. Then look at what the work tolerates: interruptible jobs can move to Spot for a steep discount, though never the always-on, availability-critical pods. Each lever is measured against the actual usage, not a guess.
Close by making it visible, not a one-off. A cluster gets expensive again the moment someone leaves a pool large and nobody notices, so the durable fix is Cost Management watching the spend, where an oversized pool shows up as a line on a report rather than a shock on the invoice. Cheaper is not a single edit; it is right-sizing plus autoscaling plus scale-to-zero, kept honest by a bill someone reads. That is what you tell finance you are setting up, not just what you cut today.
Five things worth carrying out of Part F
- Two upgrade surfaces age independently: the Kubernetes version (control plane and kubelet, ~3 releases a year, old ones lose support) and the node image (OS security patches). Upgrading is an ongoing lap, not a one-off.
- Auto-upgrade channels decide what to apply and on what cadence; planned maintenance windows decide when it may land. Pair them so change arrives only in hours you chose.
- Surge upgrades add a temporary node on the new version before draining the old one, so capacity is never reduced mid-upgrade. Raise max surge to finish inside the window.
- Use the managed trio — Container Insights for logs, managed Prometheus for metrics, Managed Grafana for dashboards — instead of self-hosting Prometheus and Grafana.
- The nodes are the bill: right-size pools, use the cluster autoscaler, put interruptible work on Spot, scale idle user pools to zero, and watch it all in Cost Management.
- The exact support window for a given Kubernetes version, and how aggressively AKS will auto-upgrade a cluster that has aged out, are the kind of detail Microsoft adjusts over time — check the current AKS supported-versions page rather than trusting a number memorised here. Treat the specifics as movable; treat the direction as settled: versions age out, support ends, and a cluster you never upgrade eventually stops getting security fixes. That last part does not change. ↩
- "Scale to zero" applies to user node pools, not the system pool — the cluster's own add-ons (DNS, metrics) have to keep running somewhere, so at least one system node stays up and keeps billing even when your workloads are asleep. The honest version of "the cluster costs nothing overnight" is "the workload nodes cost nothing overnight." A minimal system pool is the floor under the bill, and pretending it is zero is how a cost estimate quietly comes in low. ↩