Self-Hosted LLM Inference Serving with vLLM in Production

What vLLM Inference Serving Actually Fixes

Write a naive inference server around model.generate() and you will get maybe 10-20% utilisation out of an expensive GPU. The reason is structural, not a bug you can optimise away: requests finish at different times, so a statically batched group runs until its slowest member is done while every finished sequence sits there holding memory and doing nothing. vLLM inference serving exists because that waste is enormous and entirely avoidable.

Two ideas do most of the work. Continuous batching evicts a sequence the moment it emits its stop token and admits a queued request into that slot on the very next forward pass, so the batch is re-formed every iteration instead of every request. PagedAttention fixes the memory side. Together they routinely turn a low-utilisation GPU into a saturated one, which is the difference between needing four cards and needing one.

PagedAttention Is Virtual Memory for the KV Cache

This is the part worth understanding properly, because it explains most of vLLM’s tuning surface. Every token a model generates must keep its key and value tensors around for attention over the rest of the sequence — that is the KV cache, and it is the thing that eats your VRAM.

The naive implementation reserves one contiguous block per sequence, sized for the maximum possible length. A request that might generate 4,096 tokens reserves for 4,096 and typically uses 200. The rest is stranded: too fragmented to reuse, too reserved to release. Published measurements of that approach put real utilisation of the reserved cache somewhere in the 20-40% range.

PagedAttention borrows the operating system’s answer. It chops the KV cache into fixed-size blocks and keeps a per-sequence block table mapping logical positions to physical blocks, which need not be contiguous. A sequence allocates blocks as it grows, so nothing is reserved for length it never reaches. The payoff beyond density is sharing: two requests with the same prompt prefix point at the same physical blocks, copy-on-write style, which makes parallel sampling and beam search dramatically cheaper.

Abstract AI visualisation representing vLLM inference serving on GPUs — PagedAttention pages the KV cache like an OS pages memory, so nothing is reserved for unused length.

Standing One Up

vLLM speaks the OpenAI API, which is the pragmatic detail that makes migration cheap — existing client code changes a base URL and nothing else.

pip install vllm

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --enable-prefix-caching \
  --served-model-name gpt-3.5-turbo   # lie to legacy clients if you must

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarise this incident report."}],
    max_tokens=512,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

Three of those flags carry real weight. --gpu-memory-utilization is the fraction of VRAM vLLM claims for weights plus KV cache; push it to 0.95 and you gain cache blocks but leave nothing for the fragmentation spike that will OOM you at 3am. --max-model-len caps context, and lowering it from a model’s maximum to what you actually send is often the single highest-leverage change available, because it directly multiplies out into how many sequences fit. --enable-prefix-caching is close to free money when requests share a long system prompt.

Tensor Parallelism and Multi-GPU

When weights exceed one card, shard them. Tensor parallelism splits individual layers across GPUs and needs fast interconnect, because every forward pass performs an all-reduce.

# 70B across 4 GPUs on one node
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192

# Check what you actually got
curl -s localhost:8000/metrics | grep -E \
  'vllm:num_requests_(running|waiting)|gpu_cache_usage_perc'

Set --tensor-parallel-size to a divisor of the attention head count, or vLLM will reject it. And keep tensor parallelism inside a node: NVLink makes the all-reduce cheap, whereas spanning nodes over Ethernet makes the interconnect your bottleneck and can leave you slower than a smaller model on one machine. Those metrics matter more than they look — gpu_cache_usage_perc pinned near 100% with a rising num_requests_waiting means you are KV-cache bound, and the fix is a lower --max-model-len or quantisation, not more replicas.

Data centre server racks hosting GPUs for vLLM inference serving — Keep tensor parallelism within a node — the all-reduce needs NVLink, not Ethernet.

The Trade-off Nobody Mentions

Continuous batching buys throughput by spending single-request latency, and that is not a flaw to be tuned away — it is the deal. A request sharing a GPU with 30 others generates each token slower than one sitting alone on the card. Your aggregate tokens-per-second goes up; that one user’s experience goes down.

So decide which you are optimising before you tune anything. A batch summarisation job wants maximum throughput and does not care about per-request latency. An interactive chat UI cares intensely about time-to-first-token and will need --max-num-seqs capped well below what the hardware could technically hold. Benchmark with your real prompt-length distribution, because a workload of 200-token prompts behaves nothing like one of 8,000-token prompts, and a benchmark that averages them describes neither. Once it is serving traffic, put proper tracing in front of it — the tooling in our LLM observability guide covers what to record per request.

When Self-Hosting Is the Wrong Answer

Here is the uncomfortable arithmetic. An A100 or H100 costs roughly $2-4 per hour on demand, and it costs that whether or not you send it a single token. At low or bursty volume, a per-token API is simply cheaper, and it comes with someone else’s on-call rota attached. Self-hosting starts winning at sustained high utilisation — and “sustained” is the operative word, because a GPU idling overnight is money burning.

The real reasons to self-host are usually not cost. Data residency, a fine-tuned model that no API hosts, hard latency requirements that rule out a network hop, or freedom from someone else’s rate limits — those justify it. If your driver is a fine-tune, our LoRA and QLoRA fine-tuning guide pairs directly with this, since vLLM can serve LoRA adapters against a shared base model. And if you are chasing cost on an API you already use, exhaust the cheap wins first: the techniques in our prompt caching optimisation guide often cut the bill more than a GPU lease would, with none of the operational weight.

Budget for that weight honestly. Self-hosting means owning GPU driver versions, CUDA compatibility, model weight distribution, autoscaling something that takes minutes to load, and a failure mode where a bad request pattern OOMs the server for everyone. That is a platform team’s worth of work, not a weekend.

Neural network graphic illustrating self-hosted model serving decisions — Self-hosting wins on residency, custom weights, and latency — rarely on cost alone.

vLLM inference serving is the strongest open-source answer to GPU utilisation right now, and PagedAttention plus continuous batching is why: the first stops you reserving memory for tokens you never generate, the second stops finished sequences squatting in a batch. Start with --gpu-memory-utilization at 0.90, cap --max-model-len at what you genuinely send, turn on prefix caching, and watch gpu_cache_usage_perc against num_requests_waiting to know whether you are cache-bound or compute-bound. Then be ruthless about the prior question — check the vLLM documentation for the tuning knobs, but check your utilisation curve before you sign for the GPUs at all.