The first serious LLM bill tends to arrive as a surprise, and the first reaction — “switch everything to the cheapest model” — is usually the wrong lever pulled too hard. LLM cost optimization starts with understanding what you are actually paying for, because the bill is rarely where intuition says it is. Once you can read it, the biggest savings often come with no quality loss at all.
Input and output are not priced the same
The first thing to internalise: output tokens cost several times more than input tokens, typically three to five times, depending on the provider and model. This asymmetry shapes everything. A prompt that stuffs in ten thousand tokens of context to get back a fifty-token answer is cheap on the expensive side and expensive on the cheap side — so the optimisation is not the same at both ends.
It means two different disciplines. On the input side, you are managing volume: how much context you send, and how often. On the output side, you are managing generation: how much you ask the model to write. Telling a model “be concise” or capping max_tokens directly cuts the most expensive tokens, and for many tasks the verbose version was not adding value anyway.
The hidden cost: you resend the whole conversation every turn
The single most surprising line item, once you see it, is that a chat application resends the entire conversation history on every turn. The model is stateless; it has no memory between calls. So turn ten of a conversation sends turns one through nine again as input, plus the new message.
This means a long conversation’s cost grows roughly quadratically. Each turn is a little longer than the last, and you pay for the whole accumulated history every single time. A twenty-turn support chat can cost far more than twenty times a single exchange, and nobody notices because each individual call looks cheap. This is frequently the largest and most invisible chunk of a chat application’s bill.
The mitigations are concrete. Summarise old turns once a conversation gets long, replacing ten verbose exchanges with one compact summary. Trim history to a sliding window when the full transcript is not needed. And for a retrieval system, resist the urge to keep stuffing more retrieved chunks into context “to be safe” — every chunk is input tokens on every call, and the diagnosis in our RAG retrieval guide is as much a cost lever as a quality one.
Prompt caching: the button almost nobody presses
If your prompts share a long, stable prefix — a big system prompt, a fixed set of instructions, a document you ask many questions about — prompt caching can cut the cost of that prefix dramatically. The provider caches the processed prefix and charges a small fraction to reuse it, so you pay full price for the shared context once instead of on every call.
This is close to free money for the right workload and it is routinely left switched off. A customer-support bot with a two-thousand-token system prompt, called thousands of times a day, is paying to reprocess those same two thousand tokens every time when it could pay for them once. The techniques and the ordering rules that make it actually hit are covered in our prompt caching guide — the key constraint being that the cached part must be the prefix, so structure your prompt with the stable content first and the variable content last.
Model routing: not every request needs your best model
The instinct to route everything through one capable, expensive model is comfortable and wasteful. Most applications have a wide spread of request difficulty: a lot of easy classification and extraction, a little genuinely hard reasoning. Sending the easy majority to a small cheap model and reserving the expensive one for the hard minority can cut costs substantially while barely touching quality.
The trick is deciding which is which. A cheap first pass can classify difficulty, or you can route by task type — extraction and classification to a small model, open-ended reasoning to a large one. The failure mode is over-routing to the cheap model and eating a quality regression you cannot see, which is exactly what an LLM-as-judge evaluation is for: measure quality on both routes before you trust the savings.
When self-hosting changes the maths, and when it does not
At high, sustained volume the per-token economics of an API can lose to running your own inference on open models — the point where a dedicated GPU running near capacity beats paying per token. But that crossover is higher than most teams think, and it comes with an operational bill that does not show up in the per-token comparison. The honest trade-offs, including why “sustained” is the load-bearing word, are in our guide to self-hosted inference with vLLM. For most applications, exhausting the cheap wins above beats a GPU lease, because caching and history management cut the bill with none of the operational weight.
Make the bill observable before you try to cut it
The theme running through all of this is that the biggest costs are invisible until you measure them, so the highest-value early move is not any single optimisation — it is instrumentation. You cannot manage a bill you cannot see broken down, and the provider’s monthly total tells you nothing about where the money went.
Log, per request, the model used, the input token count, the output token count, and a tag for which feature triggered the call. With those four fields you can answer the questions that actually reduce a bill: which feature is most expensive, whether a specific endpoint is sending far more context than it needs, whether one runaway conversation type dominates the total. Almost always this surfaces a surprise — a single feature quietly responsible for most of the spend, a background job re-embedding data it did not need to, a prompt that grew over months and nobody trimmed. This is the same discipline as the tracing in our LLM observability guide, viewed through the lens of cost rather than quality — and the same log serves both.
Set a budget and alert on it, the way you would for any cloud spend, because LLM costs scale with usage and a bug — an infinite retry loop, a runaway agent that keeps calling itself — can multiply a bill overnight with no ceiling. An agent that loops without a hard cap on its own calls is a genuine financial risk, not just a correctness one, and the fix is a spend limit that trips before the invoice does. A cost alert is cheap insurance against the failure mode where you discover the problem when the bill arrives. Observability first, then optimisation, because the optimisations you would guess at are rarely the ones the data tells you to make.
The order to work in
Measure before you optimise: log input and output tokens per request and find where the money actually is, because it is rarely where you would guess. Then, in rough order of payoff, turn on prompt caching for stable prefixes, manage conversation history so you are not resending a growing transcript, cap and trim output on tasks that do not need length, and route easy requests to cheaper models with an evaluation guarding the quality. Reaching for a cheaper model first — the instinct — is usually the smallest of these levers and the one most likely to cost you quality. The bill rewards understanding it far more than it rewards fear of it.