The pod filesystem is a chalkboard — it wipes on restart
Everything you learned in Part A about pods being disposable has a cost you have not yet met. A pod's container filesystem is ephemeral: it exists for exactly as long as that pod instance does, and when the pod restarts — a crash, a node reboot during an upgrade, a rolling deploy — it comes back with a clean, empty disk. Anything the container wrote to its own filesystem is gone. For a stateless web app that is fine, even desirable; it is the same "cattle, not pets" honesty from Part A. For anything that must remember something between restarts — a database, an upload folder, a ledger — it is a trap waiting to spring at the worst hour.
So Kubernetes draws a hard line between the container and any storage that must survive it. Durable data does not live inside the pod; it lives in a volume that is mounted into the pod from outside, and the volume's lifecycle is deliberately separate from the pod's. The pod can die a hundred times and the volume stays put, ready to be re-attached to whatever pod takes its place. The whole of this part is the small set of nouns that make that separation work, and the single decision — one writer or many — that picks the right kind of disk underneath.
PV, PVC, StorageClass — a request, a resource, and a factory
Three nouns carry the durable-storage model, and they map cleanly onto a workflow you already know from Phase Three: declare what you need, let something provision it, bind the two. A PersistentVolume (PV) is a concrete piece of storage in the cluster — a real disk or share that exists. A PersistentVolumeClaim (PVC) is a pod's written request for storage: "I need ten gigabytes, writable by one node." The pod mounts the claim, never the volume directly, and Kubernetes binds the claim to a matching PV. That indirection is the same discipline as talking to a Service instead of a pod — the pod names what it wants, not the specific disk it gets.
Creating every PV by hand would be Phase-Two clicking in disguise, so a StorageClass automates it. A StorageClass is a named recipe — "an Azure managed disk, Premium SSD, delete it when the claim is deleted" — and when a PVC references that class, the matching CSI driver dynamically provisions a brand-new PV to satisfy it. You write a PVC, the StorageClass mints the disk, the pod mounts the claim. AKS installs several StorageClasses out of the box, so most days you never touch a PV directly; you ask, and the factory builds.
The pod names what it needs. The class builds it.
The CSI driver — Container Storage Interface — is the plug-in that actually talks to Azure to create, attach, and delete the storage. AKS ships three that matter, and choosing among them is the rest of this part. The names describe exactly what sits behind the volume: a managed disk, a file share, or object storage. The StorageClass picks the driver; the driver picks up the phone to Azure.
Disk versus Files — one writer or many
Two CSI drivers cover almost everything you will attach, and the choice between them comes down to a single question that has nothing to do with size or speed. Azure Disk CSI gives a pod a managed disk — the same kind of disk you attached to a VM in Phase Two — and a managed disk can be mounted read-write by exactly one node at a time. That access mode is called ReadWriteOnce (RWO). It is fast, it is cheap per gigabyte, and it is the correct home for anything with a single writer: a database, a message queue, any process that assumes it alone owns its files.
Azure Files CSI gives a pod an SMB or NFS file share, and a share can be mounted read-write by many pods across many nodes at once — the access mode called ReadWriteMany (RWX). That is the whole reason to reach for it: shared content that several pods must read and write together, uploads a fleet of web pods all serve, a directory two workloads hand files through. It costs more per gigabyte than a disk and a network share is slower than a locally-attached one, so you pay for the sharing and only buy it when you need it. There is also Azure Blob CSI, which mounts object storage for large unstructured data — media, backups, data-lake files an analytics job streams — where object semantics fit better than a filesystem.
Hold the rule as one line: one writer, Disk and RWO; many writers, Files and RWX. Reach for it before you think about performance tiers or price, because getting the access mode wrong does not make things slow — it makes them fail. Ask two pods to share a single ReadWriteOnce disk and the second pod simply will not schedule onto a different node; ask a database to share a ReadWriteMany file share with a copy of itself and you invite the kind of corruption no backup schedule fully saves you from.
| Driver | What it is | Access mode | Reach for it when | Cost intuition |
|---|---|---|---|---|
| Azure Disk CSI | A managed disk, attached to one node | ReadWriteOnce (RWO) | A single writer: databases, queues, anything that owns its files alone | Cheapest per GB; fastest; one node only |
| Azure Files CSI | An SMB / NFS file share | ReadWriteMany (RWX) | Many pods sharing one directory: uploads, shared content, hand-off folders | Dearer per GB; network-speed; scales to many pods |
| Azure Blob CSI | Object storage, mounted as a path | Many readers / writers | Large unstructured data: media, backups, data-lake files | Cheapest at scale; object, not true filesystem |
The layering, drawn — pod to PVC to PV to Azure
The four nouns stack in a fixed order, and seeing the stack is half of debugging it. The pod mounts a PVC; the PVC is bound to a PV; the PV is backed by real Azure storage — a disk or a share — that a CSI driver provisioned from a StorageClass. When a pod restarts, only the top of the stack changes: a new pod is created, it mounts the same PVC, which is still bound to the same PV, which is still the same Azure disk holding the same bytes. The data sat still while the pod was replaced. That is the entire point of the indirection.
When storage "does not work," the fault is almost always a mismatch somewhere in that stack — a PVC that requests an access mode no PV can satisfy, a StorageClass that provisions a disk in the wrong zone from the node, a pod scheduled onto a node the RWO disk cannot follow. Reading the stack top to bottom, and knowing which layer each object lives in, turns a vague "the volume is stuck" into a specific question you can answer with one kubectl describe.
StatefulSet — when a pod needs a name and a disk of its own
A Deployment treats its pods as interchangeable, and gives each replica a random name and a shared idea of storage. That is wrong for a database, where each replica must be a distinct member with its own durable disk that follows it across restarts. The object for that is a StatefulSet. It gives each pod a stable, ordered identity — db-0, db-1, db-2, not a random suffix — and a PVC of its own that stays bound to that identity. Delete db-1 and the replacement is still db-1, re-attached to db-1's exact disk. It is the honest exception to Part A's "cattle, not pets": some workloads genuinely need an identity, and pretending otherwise corrupts data.
This is the limit the footnote in Part A promised. Most of what you run is stateless and belongs in a Deployment; a small, important minority holds state and belongs in a StatefulSet on RWO disks. Knowing which is which — and resisting the urge to run a database as a Deployment because the YAML looks similar — is a senior judgement that shows up the first time a "simple" data workload loses everything on a routine node upgrade.
The partner integration needs to remember a small ledger
The partner-integration gateway now has to persist a small reconciliation ledger: a running record of which wholesale orders it has already handed off, so a restart mid-batch never double-sends or drops one. You reason it out loud rather than reaching for the first StorageClass in the list. Exactly one process writes this ledger — the gateway's reconciler — and nothing else should ever touch it. One writer means ReadWriteOnce, which means an Azure Disk via the Disk CSI driver, attached through a PVC to a single pod. Azure Files and its RWX sharing would be paying more for a capability that here is a liability, not a feature.
And you say the other half out loud too, because a colleague floats it: the storefront's real database is not moving onto the cluster to sit next to this. That database stays where it is, on managed Azure data services outside AKS, exactly as the estate has run it. The ledger is a few megabytes the integration owns; the storefront database is a production system of record with its own backups, scaling, and blast radius. Putting it on the cluster "while we have Kubernetes" would grow the cluster past the size of the contract that justified it — the same discipline as every part before this one.
The volume that will not mount
A pod sits in ContainerCreating and someone pages you. You read the stack, not the panic: kubectl describe pod shows a volume that will not attach, describe pvc shows the claim is bound, and the events name the cause — a ReadWriteOnce disk already attached to another node, because a second replica was scheduled where the disk cannot follow. In two minutes you have it: the workload wanted many writers but was given an RWO disk, or a Deployment was used where a StatefulSet belonged. The fix is a design decision, and you can name it because you know which layer each object lives in.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. A container's writable layer lives and dies with the pod instance; a restart gives it a clean, empty copy, and anything written only inside the container is lost. A is the exact misunderstanding that costs a team its data on the first routine upgrade. C imagines a safety net that does not exist — nothing copies the filesystem anywhere unless you explicitly mount durable storage. D confuses the wrong axis entirely: a Deployment governs how many pod copies run and how they roll out, not whether their data survives; a pod in a Deployment loses its ephemeral files exactly as readily. To keep data you mount a volume from outside the pod, which is what the rest of this part is about.
B. Many pods across many nodes writing one shared folder is the definition of ReadWriteMany, and that is Azure Files — an SMB or NFS share several pods mount at once. A cannot work: a ReadWriteOnce disk attaches to one node only, so the pods on the other two nodes would never schedule. C loses every upload the moment any pod restarts, and each pod would see a different folder anyway. D is a synchronisation nightmare you would be building by hand to avoid paying for the share that solves it out of the box. You pay more per gigabyte for Files than for a disk, but sharing is exactly the capability you are buying.
The PVC-mounts-claim rule, dynamic provisioning, and the StatefulSet. The two rejects are the errors that cause outages. ReadWriteOnce means exactly one node at a time — believing a single disk can be shared read-write across nodes is how a second replica silently fails to schedule and nobody can say why. And the container filesystem does not survive a restart; assuming it does is the original sin this whole part exists to correct. The indirection in the true answers is the same shape as talking to a Service instead of a pod: the pod names what it needs — a claim — and the platform binds it to the concrete volume.
# plan: run our reporting Postgres on AKS
1. Run it as a StatefulSet so each replica keeps
a stable identity and its own disk.
2. Back each replica with an Azure Disk PVC,
ReadWriteOnce, one disk per pod.
3. To let all three replicas share one data
directory, put it on a ReadWriteMany Files share.
4. Keep the storefront's production database off
the cluster, on managed Azure data services.
Line three. It contradicts everything the plan got right. Each database replica is a single writer that assumes it alone owns its files — which is precisely why lines one and two are correct, giving every replica its own ReadWriteOnce disk. Line three then tears that up by pointing all three replicas at one shared ReadWriteMany directory, which is how you get two processes writing the same data files at once and the kind of corruption no backup schedule fully undoes. Replicas share data through the database's own replication protocol over the network, never by mounting the same disk.
The other lines are sound. A database genuinely needs the stable identity and per-pod disk a StatefulSet provides, so A is wrong and line one is right. ReadWriteOnce is the correct mode for a single-writer disk, so B is wrong. And line four is the discipline from the case file — the storefront's system of record stays on managed data services, not migrated onto the cluster because Kubernetes is nearby. The trap is that lines one and two are so right they lend line three a false credibility.
The premise mistakes a wider door for a better one. ReadWriteMany is not "more flexible" in the abstract; it is one specific capability — letting many pods write one place at once — and you should buy it only when the workload actually needs many writers. The reconciliation ledger has exactly one writer, the gateway's reconciler. For it, RWX is not headroom; it is a door left open that nothing good walks through and something bad eventually does.
Name what the wider access actually costs. A Files share is dearer per gigabyte and slower than a locally-attached disk, so "just use Files for everything" is paying more for network-speed storage the ledger does not want. Worse, RWX removes the guardrail that RWO gives you for free: with a single-writer disk, the platform itself prevents a second pod from mounting it read-write, which is a safety property for data only one process should touch. Choose Files and you have hand-waved that protection away.
Close on the rule and its honest exception. The default is one writer, Disk and RWO; many writers, Files and RWX — pick by the access pattern, not by which sounds more capable. The ledger is single-writer, so it is a plain Azure Disk. Say the exception aloud so you are not dogmatic: the day a genuine shared-content need appears — several pods serving one upload folder — Files is exactly right and worth every extra cent. Flexibility you do not need is just cost and risk wearing a helpful face.
Separate the technical "can we" from the operational "should we." Yes, Kubernetes can run a database — a StatefulSet on ReadWriteOnce disks is exactly the tool, and this part taught it. So the objection is not that it is impossible. The objection is that running a production system of record on the cluster means you now own its durability, backups, failover, version upgrades, and disk performance by hand, on top of operating the cluster itself. Managed Azure data services already carry all of that for you, and moving the database onto AKS trades a solved problem for an unsolved one in the name of neatness.
Price the tidiness against the blast radius. "One place" sounds efficient until a routine node upgrade, a bad rollout, or a full disk touches the one system whose corruption ends the business. The storefront database is the record of every order and customer; its failure mode is not a restarted pod but a company-level incident. Keeping it off the cluster keeps that blast radius separate from the churn of a cluster you upgrade and redeploy constantly. The reconciliation ledger belongs on the cluster because the integration owns it and it is a few megabytes; the storefront database does not, because it is a different class of thing.
Close on the discipline the whole track has been teaching. The cluster should be exactly the size of the contract that justified it — the partner integration and nothing more. "We have Kubernetes, so put everything on it" is precisely the pressure that turns a small, boring, operable cluster into a sprawling one nobody can reason about. Say yes to the ledger, no to the database, and be able to explain that the line is drawn by ownership and blast radius, not by what happens to be nearby.
Five things worth carrying out of Part D
- A pod's container filesystem is ephemeral — it comes back empty on restart. Durable data lives in a volume mounted from outside the pod, with its own lifecycle.
- A PVC is the pod's request, a PV is the concrete storage, a StorageClass is the factory that dynamically provisions PVs. The pod mounts the claim, never the disk directly.
- The rule: one writer, Azure Disk and ReadWriteOnce; many writers, Azure Files and ReadWriteMany. Blob CSI covers large object data. Get the access mode wrong and it fails, not slows.
- The stack is pod → PVC → PV → Azure Disk/Files. Restart the pod and only the top box changes; the bytes underneath stay put. Debug storage by reading that stack top to bottom.
- A StatefulSet gives each pod a stable identity and its own PVC — the honest exception to "cattle, not pets," and the right home for databases. Most workloads are stateless and belong in a Deployment.
- Access modes have a subtlety worth the honesty: ReadWriteOnce is defined as "by one node," not one pod, so two pods scheduled onto the same node can in principle share an RWO disk — which is why a newer mode, ReadWriteOncePod, exists to mean strictly one pod. For everyday reasoning, treat RWO as "single writer, do not spread across nodes" and reach for Files when you genuinely need many; the node-versus-pod distinction matters mostly when you are debugging why two co-located pods behaved unexpectedly. ↩
- Running stateful workloads on Kubernetes at all remains a genuine debate, and the direction is worth more than any single verdict here: the platform can do it, managed data services usually do it better, and the right answer is workload-specific. Treat "databases on the cluster are always wrong" and "put everything on the cluster" as equally lazy. The discipline is to weigh ownership, blast radius, and who already carries the backups — as the second situation does — rather than to follow a slogan either way. ↩