Running AI workloads on AKS: GPU node pools, and why teams pick Kubernetes.
You can train and serve models on a plain GPU VM, so before you reach for a cluster it is worth being clear about what Kubernetes buys you — and what it costs you in operational weight.
New to cloud? CAMPUX is a free, build-first course. Start here →
To run AI or ML on Azure Kubernetes Service you add a GPU node pool (NC- or ND-series nodes), install the NVIDIA device plugin or GPU operator so pods can request GPUs, and scale the pool with the cluster autoscaler; AKS gives you scheduling, bin-packing, and scale-to-zero that raw VMs don't — the part the quickstart guides skip is that it also adds real operational weight you should only take on when you need it.
Almost every "how to run AI on AKS" article stops at the commands: add the node pool, apply a YAML, watch a pod land on a GPU. Those steps are correct and I'll give them to you plainly. But the commands aren't the hard part. The hard part is the judgment call underneath them — Kubernetes is a fleet of machinery you now have to operate, and for a single model it is more than you need. This piece covers both: the concrete setup on AKS, and the honest boundary of when it earns its place. It's a spoke of our larger Azure AI infrastructure guide, so if you're still choosing between a VM, a cluster, and a managed service, start there and come back.
The GPU node pool is the whole idea
An AKS cluster is a set of node pools, and a node pool is just a group of identical VMs. A GPU node pool is one backed by GPU-capable VM sizes — the NC series for general GPU compute and single-node training, the ND series for large-scale and distributed training with high-bandwidth interconnect. You keep GPU nodes in their own pool, separate from the cheap CPU nodes that run your ingress, your logging, and your control-plane-adjacent bits. That separation is the point: the expensive hardware is isolated, scales on its own schedule, and can be tainted so that only pods which explicitly want a GPU ever land on it.
Getting the pool ready to accept GPU work is where you pick one of two paths. With an AKS-managed GPU node pool, AKS installs and maintains the NVIDIA driver, the Kubernetes device plugin, and the DCGM metrics exporter for you — one step, and Microsoft owns keeping that stack coherent. That is the path Microsoft recommends for most people. The other path is to create the pool with the driver skipped (the --gpu-driver none flag) and install the NVIDIA GPU operator, which manages the driver, container toolkit, device plugin, and monitoring itself. You reach for the operator when you need a specific driver version or the CUDA tooling the operator ships; you take on maintaining it in return. Either way, the outcome that matters is the same: the node advertises nvidia.com/gpu as a schedulable resource.
A GPU is not automatically visible to Kubernetes. The device plugin running on the node is what registers each GPU and makes the node advertise nvidia.com/gpu to the scheduler. No device plugin, no advertised GPUs, and your pods sit Pending forever with no obvious error. If a GPU pod won't schedule, check that first: is the device plugin running on the pool, and is the node reporting nvidia.com/gpu in its allocatable resources.
nvidia.com/gpu, the scheduler places them on a node with a free GPU, and the cluster autoscaler grows the pool under load and scales it back toward zero when idle so the expensive nodes are not paid for while quiet.Asking a pod for a GPU
Once the node advertises the resource, a pod claims it in the container's resources block. You set a limit on nvidia.com/gpu — resources.limits.nvidia.com/gpu: 1 asks for one whole GPU. GPUs in standard Kubernetes are countable and non-divisible: you request whole units, and the scheduler finds a node with that many free. You cannot ask for half a GPU out of the box; fractional sharing needs extra tooling like time-slicing or NVIDIA's multi-instance GPU. Pair the request with a node selector or toleration that matches the taint on your GPU pool, so the pod targets GPU nodes and nothing else accidentally squats on them.
This is also where the bin-packing payoff shows up. If you have eight-GPU nodes and a stack of jobs that each want one or two GPUs, the scheduler packs them onto shared nodes instead of stranding a whole machine per job. On raw VMs you'd be doing that arithmetic by hand. That density is a real reason teams move GPU work onto Kubernetes once they have more than a couple of jobs in flight.
The scheduler packs many small jobs onto shared GPU nodes — the arithmetic you'd otherwise do by hand, per machine.
Autoscaling, scale-to-zero, and spot: the cost story
GPU nodes are the most expensive compute you can rent on Azure, which makes idle time the enemy. Three levers keep the bill honest, and all three are things a cluster does that a standing VM does not.
The cluster autoscaler watches for pods that can't schedule because there's no free GPU and adds nodes to the pool to fit them; when GPU nodes sit empty, it removes them. Configure the pool with a minimum of zero and it will scale to zero — no GPU nodes running, no GPU bill, until a pod that requests nvidia.com/gpu goes pending and pulls a node back up. That's the single biggest reason to run bursty training on AKS instead of a VM you keep forgetting to deallocate. The trade is a cold start: expect a wait while the node provisions and the driver initializes before your pod runs.
Spot GPU node pools take it further. Spot nodes run on Azure's spare capacity at a steep discount, with the catch that Azure can evict them when it wants the capacity back. For fault-tolerant training that checkpoints, or for batch inference you can retry, that trade is often worth it. You create the pool in spot mode and combine it with autoscaling; you design the workload to survive an eviction. It is not for anything that must stay up — put your latency-sensitive serving on regular nodes and your interruptible batch work on spot.
What this looks like as a table
| Task | How on AKS | Why it helps |
|---|---|---|
| Provision GPUs | Add a GPU node pool on an NC/ND-series VM size; use an AKS-managed pool or install the NVIDIA GPU operator | Isolates expensive hardware in its own pool; the driver and device plugin are installed and maintained for you |
| Schedule to GPUs | Request nvidia.com/gpu in the pod's resource limits, with a toleration for the GPU pool taint | The scheduler places and bin-packs jobs onto shared GPU nodes instead of one machine per job |
| Autoscale | Enable the cluster autoscaler on the GPU pool with a minimum of zero | Adds nodes when GPU pods are pending, removes them when idle — no manual capacity math |
| Keep costs down | Scale-to-zero on the GPU pool plus spot GPU node pools for interruptible work | You pay for GPUs only while a job runs; spot cuts the rate further for fault-tolerant training and batch |
| Serve / inference | Run inference pods on regular GPU nodes; use KAITO (the AI toolchain operator) for model inferencing, fine-tuning, and RAG | Rolling updates and horizontal scaling for serving; KAITO wires up common model workloads without hand-building the stack |
KAITO and the higher-level tooling
You do not have to assemble the model-serving stack by hand. AKS supports KAITO, the AI toolchain operator, an add-on that runs inferencing, fine-tuning, and retrieval-augmented generation on the cluster — it handles GPU provisioning and model deployment so you describe the model you want rather than every YAML underneath it. In larger, multi-team shops you'll also see GPU-orchestration layers such as Run:ai sitting on top to fair-share GPUs across teams and squeeze utilization higher. Treat both as what they are: conveniences that pay off once you already have the cluster and the multi-tenant problem. They don't change the fundamentals below them — a node pool, a device plugin, a resource request, an autoscaler.
When AKS beats a VM, and when it's overkill
Here's the part the quickstarts won't tell you, because it argues against using the thing they're documenting. For one model, AKS is overkill. If you have a single training job or one model to serve, a GPU VM — or a managed service like Azure Machine Learning — gets you there with a fraction of the moving parts. You skip the cluster upgrades, the node-pool taints, the RBAC, the CNI decisions, and the class of failures that only exist because Kubernetes exists. Reaching for AKS to serve one model is how teams end up operating a platform to run a script.
AKS earns its weight when the shape of the problem changes. Pick it when you have many GPU jobs to bin-pack onto shared hardware; when multiple teams or tenants share a GPU pool and need fair, isolated scheduling; or when you're building an MLOps pipeline where consistent scheduling, autoscaling, rolling deployments, and scale-to-zero are the product, not a side quest. That's the honest line: AKS is the right tool for a fleet of GPU work and a platform team to run it, and the wrong tool for a single job. If you're not sure which side you're on, you're probably on the VM side for now.
Two operating notes for the cloud-engineer running this, because they bite in that order. First, GPU quota: NC/ND capacity is quota-limited per region and per family, and a fresh subscription often starts at zero for the family you want, so request the increase before you plan the launch, not during it. Second, tune the autoscaler for GPU reality — cold starts are slow and GPU nodes are dear, so set sensible min/max bounds and scale-down delays rather than thrashing nodes up and down. Get quota and the autoscaler right and the rest is ordinary Kubernetes.
The takeaway
Running AI on AKS is four ideas stacked: a GPU node pool on NC/ND VMs, a device plugin or the NVIDIA operator so nodes advertise nvidia.com/gpu, pods that request that resource, and a cluster autoscaler with scale-to-zero and spot to keep the most expensive compute you can rent from sitting idle. The mechanics are not hard. The judgment is: take on the cluster when you have many jobs, many tenants, or a real MLOps pipeline — and stay on a GPU VM when you have one model and one job. If Kubernetes itself is still fuzzy, what AKS actually is is the right prerequisite before any of the GPU parts land.
Questions people also ask
How do I run GPU workloads on AKS?
Add a GPU-enabled node pool using an NC- or ND-series VM size, make sure the NVIDIA device plugin is present so the nodes advertise their GPUs to Kubernetes, then in your pod spec request the nvidia.com/gpu resource. With an AKS-managed GPU node pool, AKS installs and maintains the NVIDIA driver, device plugin, and metrics exporter for you; alternatively you can create the pool with the driver skipped and let the NVIDIA GPU operator manage the full stack. Once a pod requests nvidia.com/gpu, the scheduler places it on a GPU node.
What is a GPU node pool in AKS?
A GPU node pool is a group of AKS worker nodes backed by GPU-capable VM sizes — the NC series for training and general compute, the ND series for large-scale training and distributed jobs. You keep it as a separate node pool from your CPU nodes so the expensive GPU capacity is isolated, can be tainted so only GPU pods land on it, and can scale independently. The GPU nodes advertise nvidia.com/gpu as a schedulable resource that pods request explicitly.
Do I need Kubernetes to run AI on Azure?
No. For a single model or one training job, a GPU VM or a managed service like Azure Machine Learning is simpler and cheaper to operate than a cluster. Kubernetes earns its keep when you have many GPU jobs to bin-pack onto shared hardware, multiple teams or tenants sharing a GPU pool, or an MLOps pipeline that needs consistent scheduling, autoscaling, and rollouts. Run AKS when you have that shape of problem, not because it is the default.
Can AKS scale GPU nodes to zero?
Yes. The AKS cluster autoscaler can scale a GPU node pool down to zero nodes when no pods need it and back up when a pod that requests nvidia.com/gpu is pending. This matters more for GPUs than for anything else, because GPU nodes are the most expensive compute you can rent; scale-to-zero means you pay for them only while a job is running. Expect a cold-start delay while a node provisions and the driver initializes.
How do you request a GPU in a Kubernetes pod?
In the container's resources block, set a limit on nvidia.com/gpu — for example resources.limits.nvidia.com/gpu: 1 for a single GPU. GPUs are a countable, non-divisible resource in standard Kubernetes, so you request whole GPUs and the scheduler places the pod on a node that has that many free. You cannot request a fraction of a GPU without extra tooling such as time-slicing or multi-instance GPU. The node only advertises nvidia.com/gpu if its device plugin is running.