ConfigMaps and Secrets — pulling config out of the image
An image should be built once and run anywhere, which means the things that change between environments — the database hostname, the log level, the API key — must not be baked into it. Kubernetes gives you two objects for that. A ConfigMap holds non-secret configuration as key–value pairs, and a pod reads it as environment variables or as files mounted into the container. A Secret is the same shape for sensitive values — passwords, tokens, connection strings — kept separate so you can treat it with more care. Both let you ship the identical image to dev and to production and change only the object beside it.
Here is the honest part, and it matters more than the tutorials admit: a Secret is not encrypted. Its values are stored base64-encoded, which is an encoding, not a cipher — anyone who can read the Secret can decode it in one command. By default it sits in the cluster's data store in that plainly reversible form. Kubernetes calls the object a Secret; that name is a promise about intent, not about protection. Treating base64 as security is how a token ends up in a screenshot in a support ticket, still perfectly usable.
So the real answer for real secrets is to not keep them in the cluster at all. In §27e you will wire Microsoft Entra Workload Identity to Azure Key Vault, so a pod fetches a secret at runtime with a federated token and nothing sensitive is ever written into a Kubernetes Secret. Use ConfigMaps freely for ordinary configuration; use Kubernetes Secrets for low-stakes values and while you are learning; reach for Key Vault the moment a leak would cost you something. Knowing that boundary — and saying it out loud in a review — is the difference between shipping and shipping a breach.
Probes — how the cluster tells alive from ready
A process that is running is not the same as a process that is working, and a process that is working is not the same as one that is ready to take traffic. Kubernetes will not guess the difference; you tell it, with three probes it runs on a schedule. A liveness probe answers "is this container still healthy, or has it wedged?" — if it fails, the kubelet kills and restarts the container. A readiness probe answers "should this pod receive traffic right now?" — if it fails, the pod is pulled out of its Service's endpoints but left running, so a pod that is warming up or briefly overloaded stops getting requests without being killed.
The third exists because slow starters get murdered by the first. A startup probe guards an app that needs thirty seconds to load before it can answer anything: while the startup probe is still failing, the liveness probe is held back, so a legitimately slow boot is not mistaken for a hang and restarted into an endless crash loop. Get these wrong in either direction and you feel it. No readiness probe, and traffic hits a pod before it can serve, so users see errors during every deploy. A liveness probe pointed at a dependency you do not control, and an outage in that dependency makes Kubernetes restart your healthy pods on a loop, turning a small problem into a self-inflicted one.
Requests and limits — rationing the node
A node has a fixed amount of CPU and memory, and every pod on it competes for the same pool. You govern that with two numbers per container. A request is a reservation: it is what the scheduler uses to decide which node a pod fits on, and it guarantees the pod that much CPU and memory even under contention. A limit is a ceiling: the most the container is allowed to consume before the node steps in. Set requests too high and pods will not schedule because the node looks full; set them too low and pods pile onto a node and starve each other under load. The request is a promise to the pod; the limit is a promise to its neighbours.
What happens at the ceiling differs by resource, and the difference is worth memorising because it decides how a failure looks. CPU is compressible: exceed the CPU limit and the container is throttled — slowed down, made to wait for cycles, but not killed. Memory is not compressible: exceed the memory limit and there is no "slower," so the container is OOMKilled — terminated outright, then restarted by its controller. That is why a memory leak under a tight limit shows up as a pod that restarts every few minutes with reason OOMKilled, while an under-provisioned CPU shows up as an app that is merely, mysteriously slow. Same idea, two very different symptoms.
Over memory you are killed; over CPU you are throttled.
Autoscaling — pods on one axis, nodes on another
Load rises, and there are two separate things you might need more of: more copies of your app, or more machines to put them on. Kubernetes keeps these on two different controllers, and confusing them is the classic interview stumble. The Horizontal Pod Autoscaler (HPA) watches a metric — CPU by default — and adds or removes pod replicas to hold that metric near a target. It reads utilisation as a percentage of the pod's request, which is why the HPA does nothing useful until you have set requests; with no request there is no denominator, and there is no signal to scale on. AKS provides the metrics-server the HPA reads from, so the plumbing is already there.
The HPA can ask for ten pods, but if the nodes are full those extra pods sit Pending with nowhere to land. That is the cluster autoscaler's job: it watches for pods that cannot be scheduled and adds nodes to the pool to fit them, then removes nodes later when they sit idle. So the two work as a pair — the HPA decides how many pods you need, the cluster autoscaler makes sure there is somewhere to run them. Say the distinction cleanly and it sticks: the HPA scales pods; the cluster autoscaler scales nodes.
One more name to know but not yet wire up: KEDA, an event-driven autoscaler add-on for AKS. Where the HPA scales on CPU and memory, KEDA scales on the size of a queue, the lag on a topic, or a scheduled time — and it can scale a workload to zero when there is nothing to do. It is the right tool for a job that is bursty and event-shaped rather than steadily loaded; note where it lives so you reach for it when a queue, not a CPU graph, is what should drive your replica count.
- Horizontal Pod Autoscaler
- Adds or removes pod replicas to keep a metric (CPU by default) near a target. Needs requests set and reads AKS's metrics-server. Answers "how many copies?"
- Cluster autoscaler
- Adds or removes nodes in a pool when pods cannot schedule or nodes sit idle. Answers "is there room to run them?"
- KEDA (add-on)
- Event-driven scaling on queues, topics, or schedules — and down to zero. For bursty, event-shaped work rather than steady CPU load.
Putting it on the workload — a Deployment that behaves under load
Take the objects from §1 through §4 and they are not four features; they are one hardened workload. Config and secrets come from outside the image, so the same build runs everywhere. Probes let the cluster route around a pod that is warming up and restart one that has truly wedged. Requests and limits make the pod a good tenant of its node, guaranteed its share and barred from eating its neighbours'. And an HPA turns a traffic spike into more replicas instead of a queue of dying pods. None of it is exotic; all of it is the difference between a demo and something you can leave running.
The order matters, too. You cannot sanely add an HPA before you have set requests, because the HPA measures against them; you should not set a memory limit without watching real usage first, or you are just guessing at the altitude where OOMKilled starts. So the honest sequence is: externalise config, add probes, observe real CPU and memory, set requests and limits from what you saw, then add the autoscaler on top. Each step earns the next, and doing them out of order is how a workload that looked fine in the demo falls over the first busy afternoon.
The partner gateway learns to survive a wholesale spike
The partner-gateway Deployment in the partner-gw namespace has run its two replicas quietly since Part A, and now the first real wholesale partner is going live. Their nightly order pushes are spiky — a quiet gateway, then a flood as a partner's own batch fires. So the Deployment gets a readiness probe on its health endpoint, so a warming pod never takes traffic mid-restart, and CPU and memory requests and limits set from a week of watched usage rather than from a guess. Then an HPA on the same Deployment: keep two replicas at rest, add more up to a modest cap when CPU climbs, and settle back down when the flood passes.
What they deliberately do not do is as important. The storefront stays on App Service; the nightly pos-batch job stays on Container Apps, scaling to zero as before. Nobody moves those onto the cluster because the gateway now has an autoscaler and "we have the room." The cluster is still exactly the size of the wholesale contract that justified it — it has simply learned to breathe with that one workload's load, so a partner's traffic spike stretches it instead of toppling it. Boring, on purpose, and now a little harder to knock over.
The pod that keeps restarting every few minutes
A pod is restarting on a loop and the app logs look clean. Before you touch the code you read the mechanism: kubectl describe pod and look at Last State. If the reason is OOMKilled, the memory limit is too low for real usage — raise it, or find the leak, but stop blaming the app. If instead the liveness probe is failing, ask what it points at: a probe wired to a downstream dependency turns that dependency's outage into your own restart storm. Two minutes of reading the exit reason saves an afternoon of changing things at random.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. Memory is not compressible — there is no "a bit slower" for RAM you do not have — so exceeding the memory limit gets the container terminated with reason OOMKilled, after which its controller restarts it. A describes what happens over the CPU limit, which is compressible and so is throttled rather than killed; swapping the two is the classic error. C invents a rescheduling behaviour that does not exist; the pod is killed in place. D confuses the request, which is the scheduling number, with the limit, which is a hard ceiling the node enforces. A pod that restarts every few minutes with OOMKilled is telling you the limit is below real usage.
B. Pending means the pods cannot be scheduled — the nodes are full — and adding nodes is precisely the cluster autoscaler's job: it watches for unschedulable pods and grows the pool to fit them. A misreads the symptom; the HPA already asked for eight and got them created, so raising its cap does nothing about the lack of room. C games the scheduler by shrinking limits, which packs pods tighter but courts OOMKills under load — solving a capacity problem by lying about how much memory the app needs. D is the wrong layer entirely; Service type governs exposure, not scheduling. The HPA scales pods; the cluster autoscaler scales nodes — this drill is that sentence made concrete.
The base64 Secret, the HPA needing requests, and readiness gating traffic. The two rejects swap pairs that matter. It is readiness, not liveness, that pulls a pod from its Service without killing it; a failing liveness probe restarts the container — mixing them up is how you wire a restart storm by mistake. And the last option has request and limit backwards: the request is the scheduling reservation, the limit is the enforced ceiling. The Secret line is the honest one to carry out of here — base64 is an encoding, not a cipher, so for anything that would hurt to leak you federate to Azure Key Vault with Entra Workload Identity in §27e.
# plan: make the gateway survive load
1. Move the API key out of the image into a Secret.
2. Add a readiness probe on /healthz so warming
pods do not take traffic.
3. Point the liveness probe at the upstream partner
API, so we restart if the partner is down.
4. Set CPU and memory requests, then add an HPA.
Line three. A liveness probe is a restart switch for "is this container itself wedged." Aim it at the upstream partner API and you hand that partner the power to reboot your pods: the moment their API has a bad ten minutes, your perfectly healthy gateway fails its liveness check and Kubernetes kills and restarts it on a loop — an outage you manufactured out of someone else's. Liveness should test only the pod's own health; a dependency's reachability belongs in readiness at most, or in your app's own retry logic.
The other lines are sound. Moving the API key into a Secret is the right first step even though a Secret is only base64 — better than baking it into the image, and the path to Key Vault later (A is wrong). Readiness probes are entirely normal on a Deployment's pods (B is wrong). And requests-then-HPA is exactly the correct order, because the HPA measures against requests and is useless without them (D is wrong). The lesson generalises: never let a thing you do not control hold the trigger on your own restarts.
Correct the fact before you touch the fix. A Kubernetes Secret is not encrypted by default — its value is base64-encoded, which is an encoding, not a cipher. Anyone who can read the Secret can decode it in a single command, and by default it sits in the cluster's data store in that reversible form. The name "Secret" describes what it is for, not what protects it. Say that plainly, because an audit answer built on a false premise is worse than none.
Then price the risk honestly, neither shrug nor alarm. This is not an immediate breach — reading the Secret still requires access to the cluster's API or its data store, which is itself gated. But "gated" is not "encrypted," and a database password is exactly the kind of value whose leak is expensive and permanent. So it is a real gap for an audit that asks about secrets at rest, and pretending base64 satisfies that question is how a finding becomes a headline later.
Close on the actual answer, and name it. Real secrets should not live in the cluster at all. The pattern is Microsoft Entra Workload Identity federated to Azure Key Vault: the pod fetches the password at runtime with a short-lived federated token, and nothing sensitive is written into a Kubernetes Secret. That is what you can put in front of an auditor. Offer it as the next step rather than a scolding — the teammate was right to pull the password out of the image; they just stopped one layer too early.
Separate the two numbers first. The colleague is treating the CPU limit as the thing standing between the app and enough capacity, but the limit is a ceiling on a single pod, not a supply of CPU. Under a spike the real question is whether there are enough replicas and enough nodes to carry the load — that is the request, the HPA, and the cluster autoscaler's territory, not the limit's. Reaching for the limit to solve a capacity problem is aiming at the wrong control.
Then say what removing the limit actually does. Drop the CPU limit and one busy pod can now consume the whole node's CPU, throttling and starving every neighbour on that node — you have not added capacity, you have removed the guardrail that kept one workload from eating the others. On a shared node that trades a slow gateway for a slow everything. CPU is compressible, so nobody gets OOMKilled, but the sluggishness just spreads to whatever else lives there.
Close on the correct move. If the pods are genuinely CPU-starved under load, the answer is more copies, not a higher ceiling per copy: confirm requests are set sensibly, let the HPA add replicas on CPU, and let the cluster autoscaler add nodes if those replicas cannot schedule. Keep a limit in place so no single pod can monopolise a node. "Scale out, do not un-cap" is the reasoning to say out loud — it shows you know the difference between rationing a pod and supplying the cluster.
Five things worth carrying out of Part B
- ConfigMaps and Secrets pull configuration out of the image so one build runs everywhere. A Secret is only base64-encoded, not encrypted at rest by default — for real secrets, federate to Azure Key Vault with Entra Workload Identity (§27e).
- Three probes: liveness restarts a wedged container, readiness pulls a pod out of its Service without killing it, startup protects a slow boot from the liveness probe. Never aim liveness at a dependency you do not control.
- A request is a scheduling reservation and a guarantee; a limit is an enforced ceiling. Over the memory limit you are OOMKilled; over the CPU limit you are throttled.
- The Horizontal Pod Autoscaler scales pod replicas on a metric and needs requests set; the cluster autoscaler adds and removes nodes when pods cannot schedule. HPA scales pods; cluster autoscaler scales nodes.
- KEDA is the event-driven add-on for queue-, topic-, or schedule-shaped work, and can scale to zero. Reach for it when a queue, not a CPU graph, should drive the replica count.
- You can turn on encryption at rest for Secrets — a KMS provider, or etcd-level encryption — and a managed control plane already encrypts its data store at the disk level. So "not encrypted" is shorthand for "not meaningfully protected from anyone who can read the Secret object," which is the threat that matters here: the value decodes from base64 in one command regardless of disk encryption. Treat the base64 point as settled and the disk-level nuance as a reason to use Key Vault anyway, not a reason to relax. ↩
- Treat any specific replica counts and limit values in this part as illustration, not prescription. The only honest way to set requests and limits is to watch a workload's real CPU and memory for a while and set them from what you saw, then revisit after load changes. Numbers copied from a blog post are guesses wearing a uniform; the method — observe, set, revisit — is the part worth keeping. ↩