AI Spend Is a Capacity Problem, Not a Billing Problem

Every organisation running models in production eventually reaches the same point: the AI portion of the cloud bill grows faster than expected, and the immediate response is to invest in visibility. Calls are tagged, spending is attributed, dashboards are created, and the results are shown to the teams responsible.

That work is valuable, but it does not reveal where the cost was actually determined. By the time a token appears on a dashboard, the decision that priced it was made weeks earlier during an architecture review that nobody treated as a financial discussion. AI cost is fundamentally a capacity discipline disguised as a billing problem.

They tell you what already happened, attributed accurately.

Tools that break spending down by team, feature and model [HOST LINK] solve a genuine problem. Without them, you cannot tell whether the AI bill was created by one runaway batch job or by a thousand well-behaved features. You also cannot have a useful conversation with the team responsible for that spending.

What these tools cannot tell you is whether the total should ever have reached that level. A dashboard can accurately report $40,000 in inference spending. It cannot show that the same workload might have cost $12,000 under a different capacity model, because that alternative was never billed and therefore never appeared in the data.

Three inputs, and only one of them is a price.

AI infrastructure cost is determined by three variables: monthly token volume, the blended tariff paid for those tokens and the utilisation achievable on any reserved capacity.

Procurement teams usually focus on negotiating the second variable. In practice, the third often has a greater effect on the final bill, and utilisation is an operational decision rather than a purchasing one.

Start with where the two models cross over.

A dedicated H100 costs approximately $2,000 per month, consistent with a market rate of two to four dollars per GPU-hour when used continuously.

Flagship open-weight models are available through APIs for roughly $1.00 per million input tokens and $4.00 per million output tokens. With an input-to-output ratio of 85/15, which is typical of production workloads, the blended cost is approximately $1.45 per million tokens.

Dividing the monthly GPU cost by the blended token price places the break-even point at approximately 1.38 billion tokens per month. Below that volume, the API is less expensive. Above it, dedicated capacity begins to offer better economics.

Because 1.38 billion tokens a month requires only about 525 tokens per second sustained, which any modern accelerator delivers without effort.

Capacity is rarely the limiting factor in an inference workload. Cost is. The organisation usually runs out of budget long before the hardware runs out of processing capacity.

The important question is therefore not whether the GPU can handle the workload. It is how much of each month the GPU spends idle.

Because a reserved GPU bills continuously and your traffic does not arrive continuously.

A workload limited to standard business hours uses approximately 176 of the 730 hours in an average month. That represents utilisation of only 24.1%.

Under those conditions, every delivered token effectively carries around four times the cost suggested by the headline calculation. The true break-even point moves from 1.38 billion tokens to something closer to 6 billion.

This is where many capacity decisions go wrong, and mistakes happen in both directions. Some teams reserve dedicated hardware at volumes that cannot justify it and end up paying for idle GPUs. Others remain on per-token pricing long after a mostly utilised reserved instance could have reduced the bill by half.

Neither mistake is obvious on a spending dashboard. In both cases, the dashboard simply shows a number increasing.

The useful metrics are not in your billing system, they are in your serving layer, and if you run an open-source stack they are already being emitted.

vLLM exposes a Prometheus endpoint at /metrics, with metric names prefixed by vllm:. Two of those metric series can predict your infrastructure costs more effectively than an invoice.

The first is KV cache utilisation. Memory rather than compute usually limits the number of concurrent requests, and the cache holds the context for every conversation currently in progress.

At startup, vLLM reserves a percentage of GPU memory—90% by default—loads the model weights into that allocation and assigns the remaining space to the cache. When KV cache utilisation approaches its limit, the system has reached its concurrency ceiling, regardless of what the general GPU utilisation graph shows.

The second metric is queue depth. A growing queue while the cache remains half empty usually indicates a scheduling problem. A growing queue combined with a full cache points to insufficient capacity.

These situations may look identical on a spending report, but they require opposite responses.

Track both metrics alongside the number of tokens served per hour. Together, they reveal something that a billing dashboard cannot: how much of the capacity you are paying for is actually being used.

Caching first, because it is nearly free.

Agent loops, RAG pipelines and applications with large system prompts repeatedly send the same prefixes. Providers often price repeated input very differently from new input.

For example, DeepSeek publishes $0.14 per million input tokens on a cache miss against $0.0028 on a cache hit, creating a fiftyfold difference for the repeated part of the request.

Despite the potential saving, most teams have never measured what percentage of their input tokens could be cached.

Batching comes second. Throughput per GPU-hour depends partly on the number of requests processed concurrently. Anything that safely increases concurrency reduces the cost per token when using reserved capacity.

Scheduling comes third, and it is the area operations teams are best positioned to manage. Batch enrichment, embedding regeneration, evaluation jobs and offline summarisation do not need to run at 11 a.m.

Moving these workloads to overnight periods costs nothing and improves the utilisation of capacity that has already been purchased. This single change has moved more workloads across the break-even point than most price negotiations.

Commitment comes last, after caching, batching and scheduling have stabilised utilisation. Negotiating a long-term rate for a workload that has not yet been optimised locks the organisation into the wrong capacity profile.

At the point where the answer stops being a number and starts being a constraint.

Several factors can turn capacity from a procurement issue into an architecture decision: sustained volume above the break-even point, a workload capable of keeping reserved hardware busy, latency requirements that a shared endpoint cannot meet, or regulations requiring data to remain within a defined jurisdiction.

Any one of these constraints can justify moving away from standard consumption-based APIs.

The market now offers intermediate options that were not widely available eighteen months ago. Between using public APIs and purchasing hardware, organisations can choose enterprise GPU platforms with published hourly pricing. The cost is known in advance, while the provider remains responsible for the operational burden.

Whichever option you select, the calculation uses the same three variables: volume, blended tariff and achievable utilisation. Perform that calculation before the architecture review, not after receiving the invoice.

Four things, none of which require a procurement cycle.

First, measure the percentage of input tokens that can be cached. Caching is usually the least expensive saving available, yet most organisations do not know their cacheable share.

Second, instrument KV cache utilisation and queue depth in the serving layer.

Third, calculate the break-even point using your actual traffic volume and contracted prices rather than relying on a vendor’s example.

Finally, examine the shape of your traffic and determine what percentage of the month any reserved capacity is genuinely working.

Keep the spending dashboard. It tells you who is spending money and what they are spending it on. These four measurements explain why the cost is that high—and that is the answer that can actually change the bill.