Kubernetes wraps a language model the way it wraps any app: a Deployment, a Service, pods the scheduler places. But a GPU pod isn't like a web pod. It needs a GPU the cluster may not have, an image that's mostly model weights, and minutes of loading before it can answer.
This story follows one Monday-morning traffic spike on a vLLM service. Watch the autoscaler ask for help in a second, then watch what it takes for that help to arrive. Then replay the same morning three ways.
You'll learn
- Why LLM serving scales on the request queue, not on CPU or GPU utilization.
- What
Pending with Insufficient nvidia.com/gpu means, and what the cluster autoscaler does about it.
- Why a new GPU node can be Ready and still have no GPU to offer.
- Why a running vLLM pod isn't ready, and how the readiness probe keeps traffic off it.
- The trade between waiting, an always-on spare, and streaming weights.
How to watch
The batchis the grid in each pod: about 64 requests a replica works on at once. When it's full, new requests wait.
The lineby the Service is requests waiting for a slot. The chart beside it is how long a request arriving now waits for its first token.
The clockis the incident's real time. Minutes pass in seconds here, and it tells you when.