Networking and storage for AI on Azure: feeding the GPUs fast enough.
A rack of GPUs is the most expensive thing in the building. The whole job of the infrastructure around it is to make sure it is never waiting.
New to cloud? CAMPUX is a free, build-first course. Start here →
An AI training job stalls two ways: a slow interconnect between the GPUs, and a storage layer that cannot feed them. On Azure you solve the first with InfiniBand-connected ND-series VMs and the second with a high-throughput parallel file system, Azure Managed Lustre, plus fast blob access. The goal, the one everything else serves, is to never leave an expensive GPU waiting on data. That last sentence is the whole discipline, and it is the part most guides skip while they list VM SKUs.
The honest version of the problem is the one the vendor pages will not lead with. When a training run comes in under the throughput you paid for, people reach for a bigger GPU. It is almost never the GPU. The GPU spends most of its idle time waiting for two things: gradients from the other GPUs it is training with, and the next batch of data from storage. Fix those two paths and the GPU you already have gets busy. Miss them and a rack of the fastest silicon on the planet runs at a third of its rate while the meter runs at full price. This note is about those two paths on Azure, written from the seat you will sit in — the cloud engineer who has to provision it, size it, and defend the bill.
If you want the wider map of what sits around the GPU (the VMs, the orchestration, the reference architecture), start with the hub piece, Azure AI infrastructure. This note zooms into the two lines on that map that decide whether the whole thing is fast: the wires between the chips, and the pipe that fills them.
The interconnect: why GPUs need to talk fast
Modern training does not run on one GPU. It runs on many, splitting the model or the data across them, and every training step they have to agree — each GPU computes a slice of the gradient, then all of them exchange and average what they found before the next step. That exchange happens every single step, thousands of times an hour. If the link between them is slow, the compute sits idle waiting for the sync. The interconnect is not a nice-to-have; it is in the inner loop.
There are two interconnects, and they operate at different scales. Inside a single VM, the GPUs on the board talk over NVLink, NVIDIA's direct GPU-to-GPU bus, which is far faster than anything that crosses a network. As long as your job fits inside one VM, say eight GPUs, that is the fabric doing the work and you do not think about the network at all. The moment the job spans more than one VM, the gradients have to cross the data-center network, and ordinary Ethernet is too slow and too jittery for a sync that happens every step.
That is where InfiniBand comes in, and yes, Azure supports it. The GPU-accelerated ND-series VMs ship with NVIDIA Quantum InfiniBand for VM-to-VM traffic. The numbers are worth stating precisely, then treating with the usual suspicion — Microsoft revises SKUs constantly, so verify the exact figure for your region and size at deploy time. On ND H100 v5 and ND MI300X v5, each GPU gets its own dedicated 400 Gb/s InfiniBand link, which works out to 3.2 Tb/s per VM. The older ND A100 v4 gives each GPU a 200 Gb/s HDR InfiniBand connection. Critically, the fabric is wired up automatically between VMs placed in the same virtual machine scale set, and it supports GPUDirect RDMA — the GPUs read each other's memory across the network without dragging the CPU into it. That RDMA path is what makes multi-node training scale to thousands of GPUs instead of collapsing under sync overhead. For the SKU-level detail on which VM to pick, see Azure GPU VMs for AI.
The interconnect is not a nice-to-have. It sits inside the training loop, and every slow step pays for it thousands of times an hour.
The storage path: keeping the pipe full
Now the other stall. A GPU chews through data faster than most storage can serve it, and a training run reads its dataset over and over, epoch after epoch. Point a rack of GPUs at a single blob download stream and you will watch utilization sag to nothing while every chip waits its turn for the same slow pipe. The fix is not one fast disk; it is a storage layer that can serve many readers at once at aggregate throughput measured in hundreds of gigabytes per second.
On Azure the workhorse for that is Azure Managed Lustre. Lustre is the open-source parallel file system that has run large HPC clusters for years; Managed Lustre is Microsoft running the cluster for you as a pay-as-you-go service. You provision a namespace sized in TiB and get throughput in the hundreds of GB/s with millions of IOPS — enough to keep a large multi-node job fed. It is priced per TiB per hour, and that price is real: it costs far more than blob per byte, which is exactly why you do not park cold data on it. The pattern is to stage a dataset onto Lustre for the run and tear it down after. Managed Lustre integrates with Blob Storage through hierarchical storage management, so you import the training set from cheap blob at the start and export trained models back to long-term storage at the end.
That leaves the supporting cast. Azure Blob Storage, ideally a premium or high-throughput account, is where the raw and archived data lives cheaply between runs; it is the cold pool the parallel file system draws from, and for smaller jobs you can sometimes read from it directly. Azure NetApp Files fits a different need: shared, low-latency file storage a whole team mounts, good for home directories, code, and checkpoints rather than raw training throughput. And every GPU VM ships with local NVMe — physically attached scratch disk that is the fastest storage you can touch, perfect for staging the current shard or spilling temporary data, with the catch that it evaporates when the VM stops. The skill is putting each kind of data on the tier that matches how it is read; if the storage-account choices feel fuzzy, Azure storage account types lays out the base menu.
Reading a dataset cold over the network in the middle of an epoch is how you starve a GPU. Before the run, copy the data onto the fast tier — import it from blob into Managed Lustre, or pull the current shard down to local NVMe — so that during training the GPU reads from something local and fast, not from a distant object store. The staging copy costs minutes once. Skipping it costs throughput every epoch, and you pay for that idle time at GPU prices.
The map: need, option, when
Five needs cover most of what an AI platform on Azure has to store and move. Match each to the option that fits, and resist the urge to buy the fastest tier for data that does not need it — the premium SKUs are where AI budgets quietly bleed.
| Need | Azure option | When to use |
|---|---|---|
| GPU-to-GPU interconnect | InfiniBand on ND-series VMs (NVLink within a node) | Any multi-node training run; NVLink handles inside a single VM automatically. Skip InfiniBand entirely for single-node jobs and inference. |
| High-throughput training data | Azure Managed Lustre (parallel file system) | Large multi-node jobs where many GPUs read the same dataset and need hundreds of GB/s. Stage in from blob, tear down after. |
| Shared team storage | Azure NetApp Files | Home directories, code, and checkpoints a whole team mounts with low latency — shared file access, not raw training throughput. |
| Cheap bulk / cold data | Azure Blob Storage (premium or high-throughput tier) | Raw datasets and archived models between runs; the cold pool the parallel file system imports from and exports back to. |
| Fast scratch | Local NVMe on the GPU VM | Staging the current shard and spilling temporary data during a run — fastest tier available, but wiped when the VM stops. |
The cloud engineer's cut: it is a provisioning and cost problem
Here is the part the HPC blogs and product pages leave out, because they are selling the silicon. On a real team, this is not a research problem — it is a provisioning and cost-control problem, and that is the job they are hiring you for. The InfiniBand-enabled ND SKUs are among the most expensive compute Azure rents, and Managed Lustre bills per TiB per hour whether the GPUs are busy or not. The two ways to lose money here are symmetrical: under-provision the data path and pay full GPU price for idle chips, or over-provision the fast tiers and pay premium storage rates for data that a cheap blob could have served.
So the work is boringly practical. Place the GPU VMs in the same scale set so the InfiniBand fabric forms between them — get that wrong and your expensive multi-node job silently falls back to slow networking. Size the Lustre namespace to the run and tear it down when the run ends; a parallel file system left idle over a weekend is a line item nobody will thank you for. Keep the durable data in blob and treat Lustre and NVMe as transient. Watch GPU utilization as your primary health metric — when it sags, the answer is upstream in the interconnect or the storage path, almost never a bigger GPU. None of that is exotic. It is the same discipline as the rest of cloud engineering: right-size the resource, kill it when it is idle, and put each kind of data on the cheapest tier that still meets the deadline.
Read the specific throughput and bandwidth numbers here as a snapshot rather than a spec sheet, even though the underlying direction holds. Microsoft renames and re-specs these SKUs on a schedule that outpaces any blog, so check the current figure for your region before you quote it to anyone. What does not change is the shape of the problem: an idle GPU is the most expensive thing on the invoice, and every design decision in this note exists to keep it fed.
Questions people also ask
Does Azure support InfiniBand?
Yes. The GPU-accelerated ND-series and the HPC-focused HB-series VMs ship with NVIDIA Quantum InfiniBand for VM-to-VM traffic. On ND H100 v5 and ND MI300X v5, each GPU gets its own dedicated 400 Gb/s InfiniBand link, for 3.2 Tb/s per VM; ND A100 v4 uses 200 Gb/s HDR InfiniBand per GPU. The fabric is provisioned automatically between VMs in the same scale set and supports GPUDirect RDMA.
What storage is best for AI training on Azure?
For large multi-node training where many GPUs read the same dataset, a parallel file system — Azure Managed Lustre — is the usual answer, because it delivers hundreds of GB/s of throughput and keeps the GPUs fed. Keep the raw and archived data cheaply in Azure Blob Storage and stage it into Lustre for the run; use local NVMe on the VM for scratch. Azure NetApp Files is a good fit when a team needs shared, low-latency file storage rather than raw training throughput.
What is Azure Managed Lustre?
Azure Managed Lustre is a managed, pay-as-you-go parallel file system based on the open-source Lustre used across HPC. Microsoft runs the cluster; you provision a namespace sized in TiB and get throughput in the hundreds of GB/s with millions of IOPS. It integrates with Azure Blob Storage through hierarchical storage management, so you import a dataset from blob for a training run and export trained models back to cheap long-term storage.
How do you avoid GPUs sitting idle?
You make sure nothing upstream of the GPU is slower than the GPU. In practice that means a fast interconnect so multi-GPU gradients sync quickly (InfiniBand, and NVLink inside a node), a storage layer with enough throughput to feed every GPU at once (a parallel file system, not a single blob stream), and staging the dataset onto fast storage or local NVMe before the run rather than reading it cold over the network mid-epoch. Watch GPU utilization; if it sags, the fix is almost always the interconnect or the data path, not a bigger GPU.
Do you need InfiniBand for AI on Azure?
Only for multi-node training. If your job fits inside a single VM, the GPUs talk over NVLink on the board and InfiniBand is irrelevant. The moment training spans more than one VM, gradients have to cross the network every step, and standard Ethernet becomes the bottleneck — that is where InfiniBand's low latency and RDMA earn their keep. For single-GPU or single-node work, or for inference, you do not need it and should not pay for it.