What LLM inference actually costs (and how to cut it)
The token price on the pricing page is the smallest part of the bill. Here's where LLM cost actually comes from in production, and the levers that move it without hurting quality.
Teams estimate their AI bill by multiplying a token price by a request count, then get a surprise at the end of the month. The published price is real, but it is the least interesting number in the equation.
The actual cost of running an LLM feature is driven by how many tokens you push through it, how often, and how much of that work you are needlessly repeating. Almost all of that is under your control once you can see it.
Input tokens are usually the hidden bulk
Everyone watches the output. But in most production systems the input dwarfs it — a long system prompt, a stack of few-shot examples, retrieved context and a growing conversation history, all resent on every single turn.
A chatbot that replays its full history sends a little more context with each message, and cost grows with the square of the conversation length. The fix is rarely a cheaper model; it is sending fewer tokens. Trim the system prompt, retrieve less but better, and summarize old turns instead of resending them verbatim.
Cache the things you keep resending
If the front of your prompt is identical across requests — and for a fixed system prompt and instructions, it is — prompt caching lets the provider skip recomputing it. The savings on the cached portion are large, and the change is mostly about ordering your prompt so the stable parts come first.
Beyond the provider cache, a plain response cache in front of the model pays for itself fast. A surprising share of real traffic is near-duplicate questions. Serving the repeats from an exact or semantic cache is both cheaper and faster than any model call.
- Put stable content (system prompt, tools, examples) at the front so it can be cached
- Cache identical requests outright — the cheapest inference is the one you skip
- Consider a semantic cache for near-duplicate queries, with a similarity threshold you have tested
- Set and monitor a per-request token budget so a runaway prompt cannot quietly balloon
Right-size the model per call, not per app
Picking one model for the whole product means overpaying on easy requests or underserving hard ones. Most workloads are a mix: a lot of simple classification and extraction, a little genuinely hard reasoning.
Route by difficulty. Send the easy majority to a small, cheap model and reserve the frontier model for the cases that need it. A good router plus a cheap default often cuts cost by more than half while leaving quality on the hard cases untouched. Your eval set is what tells you where the line sits.
Shorten the output, and stop generating early
Output tokens are typically the most expensive per token and the slowest to produce, because they are generated one at a time. A prompt that asks for a terse structured answer costs less and returns faster than one that invites the model to think out loud in prose.
Ask for exactly the format you will parse. Set a sensible max-token cap. Use stop sequences so generation ends the moment the useful part is done. And when you do not need streaming, batch requests together — batch pricing is materially cheaper for anything that is not user-facing in real time.
Measure cost as a first-class metric
You cannot optimize a bill you cannot attribute. Log tokens in and out, cached versus uncached, and model used — tagged by feature and by customer. Cost per request belongs on the same dashboard as latency and quality, not in a monthly invoice.
Once cost is visible per feature, the expensive outliers reveal themselves and the fixes are usually obvious. The goal is not the lowest possible bill; it is the lowest bill that still clears your eval bar. Cost, latency and quality are one trade-off, and you should be tuning all three on purpose.