LLM Cost per Inference: How to Actually Measure It

LLM cost per inference sounds like it should be a single number, and that is exactly the trap. In a real system, one user action rarely maps to one model call, cheap and expensive calls hide behind the same feature, and an average smooths away the very cases that are eating your margin. Measuring cost per inference properly means defining the unit carefully, computing it from the right raw data, and resisting the averages that make the number look tidy and useless. This post walks through how to do it.

It is the metric that sits underneath everything in per-user cost tracking and reducing your API bill: you cannot price, optimise, or forecast what you have not measured at the unit level.

Define the unit before you measure it

“Per inference” is ambiguous, and the ambiguity is where analyses go wrong. There are at least three candidate units:

• Per API call. The raw model invocation. Precise, but often meaningless to the business — one user action might be five calls.
• Per user action. The cost of one thing a user actually did: a search, a summary, a chat turn. Usually the right unit for product decisions.
• Per completed task. For agents and multi-step workflows, the cost of finishing a whole task — planning, tool calls, retries, and all. The truest unit-economics signal, because it captures the steps that per-call figures ignore.

Pick the unit that matches the decision you are making. For pricing a feature, per user action. For catching a degrading agent, per completed task. Mixing them — comparing a per-call cost against a per-task budget — produces nonsense.

Compute it from token counts, not from the bill

The bill is an aggregate arriving weeks late; it is useless for per-inference measurement. The correct source is the token usage returned on every call. For each inference:

cost = (input_tokens x input_rate) + (output_tokens x output_rate)

Both terms matter, because output tokens usually cost several times more than input tokens — a fact that a single blended rate would hide. Compute this at the call site, tag it with the feature, model, and user, and you have per-inference cost as a live signal rather than a month-end reconstruction. Token cost tracking captures the token split and computes cost per call across providers. Because per-token rates change and differ by model, keep the rate table current — check the provider’s pricing page rather than trusting a hard-coded constant.

The averaging trap

Once you have per-inference cost, the strongest temptation is to reduce it to a single average, and that average will lie to you the same way it does with per-user cost. LLM cost distributions are skewed: most inferences are cheap, a few are dramatically expensive, and the mean sits in a gap where few actual inferences live.

Watch the distribution, not just the mean:

• Track the tail. The p95 and p99 cost per inference often reveal the runaway cases — the agent task that took forty steps, the prompt that pulled in an enormous context. These are where margin dies, and the average hides them.
• Segment before you average. A blended cost per inference across a cheap classification feature and an expensive reasoning feature describes neither. Average within a feature and model, never across.
• Weight by volume when you roll up. A high per-inference cost on a low-volume feature matters less than a modest cost on your highest-traffic path.

Cost per completed task: the metric that catches rot

For anything multi-step, cost per completed task is the number to anchor on, because it is the one that exposes slow degradation. A monthly total can stay perfectly flat while cost per task quietly triples — because tasks are taking more steps, retrying more, re-sending more accumulated context on each call. That is a workflow rotting in slow motion, and only the per-task view catches it before the total does.

Tracking it requires grouping all the calls that belong to one task under a shared identifier and summing their cost. The payoff is an early-warning signal: when cost per completed task starts climbing, a workflow is degrading, and you have found it weeks before the invoice would have. A steady climb here is exactly the pattern anomaly detection is built to flag automatically.

Turning the number into decisions

A well-measured cost per inference feeds directly into action:

1. Pricing. If a feature costs a known amount per user action, you can price it with a real margin instead of a hopeful guess.
2. Routing. If per-inference cost on a feature is high because it defaults to a frontier model, routing simpler calls down is the obvious lever — see model routing for cost savings.
3. Budgeting. Per-inference cost times expected volume is the basis of any honest forecast, which is where cost forecasting begins.
4. Guardrails. A per-task cost ceiling enforced through budget alerts stops a stuck agent from failing expensively.

As an illustrative example only, with invented figures: a summarisation feature might cost a small fraction of a rupee per user action on a cheap model, while an agentic research task runs to several rupees per completed task across dozens of steps — the two are not comparable, and only per-unit measurement makes that legible. The numbers are made up; the discipline of measuring each unit separately is the point.

FAQ

What is LLM cost per inference? It is the cost of a single unit of AI work — a model call, a user action, or a completed multi-step task — computed from that unit’s input and output token counts rather than read off an aggregate bill.

How do I calculate it? Sum input tokens times the input rate and output tokens times the output rate for the unit, using current per-model rates. Capture the token split at the call site with token cost tracking; do not use a single blended rate, since output usually costs several times more than input.

Why is the average cost per inference misleading? LLM cost distributions are skewed — most inferences are cheap and a few are very expensive — so the mean describes almost no real inference. Track the p95/p99 tail and segment by feature and model before averaging.

What is cost per completed task and why does it matter? It is the total cost of finishing a multi-step task, including retries and re-sent context. It exposes slow workflow degradation that a flat monthly total hides, making it the truest unit-economics signal for agents.

---

Cost per inference is the number that turns “AI is expensive” into a decision you can actually make. See how AI Vyuh FinOps computes it per call and per task with token cost tracking, flags degradation with anomaly detection, and enforces ceilings with budget alerts — or read more on the blog. Working out unit economics across teams? Email finops@aivyuh.com or see enterprise.