LLM cost: why the bill shows up after rollout

The prototype costs a few euros a day. Ten test users, short prompts, nobody checks the bill. Then the system goes live, the context grows because every answer gets better with more background, and the bill grows faster than the user count.

This is where FinOps for LLMs stops looking like classic cloud FinOps. The levers are different, and they hang directly on answer quality, not just infrastructure.

Where the money actually goes

Context length is the biggest lever: every extra chunk in the prompt costs on every call, not once. A RAG system that pulls ten chunks instead of three, because it raises the hit rate, triples the input cost per request just like that. On top of that come reranking and other intermediate calls, a reranker model, a validation call, a second look through a smaller model. Each of these steps is its own line item on the bill.

A third driver gets overlooked often: model choice without feedback. A team switches to a bigger, more expensive model because the text sounds better. Whether quality actually improved or only the style did is something nobody can say without evaluation. And then there are retries on failure, a parsing error that triggers a second call. Invisible at a low error rate. At 5% error rate and high volume, a visible line on the bill.

What I measure first

Not the total at month's end. Cost per request, broken down by step: retrieval, reranking, main call, validation. Without that breakdown, "costs went up" is an observation with no lever attached.

Request: "Which standard applies to additive X?"
  Retrieval:      $0.0003  (embedding + search)
  Reranking:      $0.0012  (cross-encoder, 20 candidates)
  Main call:      $0.0180  (context: 6 chunks, answer: 200 tokens)
  Validation:     $0.0040  (smaller model, source check)
  Total:          $0.0235

That breakdown shows immediately where an optimization actually works. If the main call carries 75% of the cost, a cheaper reranker buys little. If retries make up 30%, the parsing error rate is the real cost problem, not the model.

The mistake I see most often

Teams optimize the model before they optimize the pipeline. A cheaper model for the main call saves money if the context stays the same. But if the context drags in unnecessary chunks, you pay for dead weight no matter which model sits behind it. The order should be reversed: trim the context first, then think about the model.

A second mistake: looking at cost per token instead of cost per request relative to the value of the answer. A request that costs ten cents but replaces an hour of manual standards research is cheap. A request asking "what's the weather" at the same price is not.

Caching, before you touch the model

Some requests repeat, word for word or close to it. A standard gets asked about often, a standard phrasing shows up across several sessions. A cache at the embedding level that recognizes similar requests and reuses the last answer plus its source cuts cost without a user noticing anything except a faster reply. The catch: the cache has to know when a source changed, or it serves a stale, already wrong answer faster than before. Without an invalidation rule, caching is a new hallucination risk with better latency.

What multi-tenant changes about the cost question

For a multi-tenant system like the food-science workspace, a total at month's end is not enough. Without cost per tenant you cannot build a pricing model that does not lose money on every new customer. It is the same point as tenant isolation, just on the cost side instead of the security side.

The tenant tag belongs in the same place as the retrieval filter: in the request context, not in a heuristic assigned after the fact from timestamps. If you split cost across tenants after the fact through guesswork, you get numbers that will not hold up in a pricing negotiation.

Self-hosted model or API

Past a certain request volume, the question of whether a self-hosted model beats a provider's API on cost becomes seriously relevant. The answer rarely comes down to price per token alone. A self-hosted model brings its own infrastructure cost, GPU instances that have to run even at low load, and operational effort that sits with the provider in the API case. At low, irregular volume, the API is almost always cheaper, because nobody pays for a GPU sitting idle most of the time. At high, predictable volume, the math flips. Without the cost-per-step breakdown from earlier, you cannot make that call seriously, you just guess with more confidence.

Who actually sees the bill

In many teams, finance sees the cost number first, at month's end, with no context. The technical side only gets the number once it is already uncomfortably large. A budget alert at the request or tenant level, firing before the bill becomes a surprise, works better. That is not complicated infrastructure, just a threshold and someone reading it.

What the check does with it

Cost sits on day 4 of the LLM readiness check, next to operations and tenants. I look for whether cost per request is even measurable, and whether anyone knows which pipeline step carries the biggest share before they touch the model.

The service behind this sits under AI engineering. Related: evaluation in production, reconstructing answers.