Running LLM Workloads in Production: An Operations Playbook for Teams That Did Not Sign Up for This

Somewhere in the past two years, AI quietly became an operations problem. The proof of concept your product team shipped — a support-ticket summarizer, a natural-language search box, a code-review assistant — graduated into a production dependency, and now it pages you. The failure modes are unfamiliar: latency distributions with tails measured in tens of seconds, upstream providers that throttle without warning, costs that scale with user enthusiasm rather than infrastructure size, and outputs that can be wrong in ways a health check will never catch. This is a playbook for operations teams inheriting LLM workloads, written from the perspective of keeping the thing up, observable, and affordable.

Treat Model Providers Like Any Other Third-Party Dependency — Only Less Reliable

The first mental shift is accepting that a hosted AI model is an external dependency with availability characteristics closer to a beta service than a database. Providers ship breaking changes on short notice, deprecate models with a few months' warning, tighten rate limits during demand spikes, and occasionally degrade output quality without any versioned change you can pin. Your standard playbook for third-party dependencies — timeouts, retries with backoff, circuit breakers, and a fallback path — applies in full, with parameters you will need to retune. A 30-second timeout that would be absurd for a REST API is normal for a long-context completion. A retry storm against a rate-limited provider makes the throttling worse, so exponential backoff with jitter and a hard retry budget is non-negotiable.

The fallback path deserves the most thought. When your primary model is down or throttled, the least-bad option is rarely "return an error." Teams that handle this well define a degradation ladder in advance: fail over to a second provider running a comparable model, then to a smaller and cheaper model that produces acceptable results for most requests, and only then to a cached response or an honest unavailability message. Multi-provider failover used to mean maintaining several SDK integrations and credential sets. Most teams now route through an aggregation layer instead — a marketplace such as APIMart exposes models from many labs behind one OpenAI-compatible endpoint, so failing over from one provider's model to another is a routing rule rather than an integration project, and the per-model pricing is visible on one bill.

Observability: Tokens Are Your New Golden Signal

Standard HTTP metrics miss most of what matters in an LLM workload. Two requests with identical status codes and endpoints can differ by two orders of magnitude in cost and latency depending on token counts. Instrument every call with input tokens, output tokens, model name, latency broken into time-to-first-token and total generation time, and the request's business context — which feature, which customer tier. Time-to-first-token is your user-experience metric for streaming interfaces; total tokens are your cost metric; and the ratio of output to input tokens is a surprisingly effective anomaly detector, because prompt-injection attempts and runaway generations both distort it.

Log full request and response payloads somewhere queryable, with retention and access controls that reflect the sensitivity of the content, because "why did the model say that" is an incident question you will eventually face. Sampled logging is acceptable at scale; zero logging is how you end up unable to reproduce the exact prompt that produced a harmful output in front of a customer.

Cost Control Is Capacity Planning

LLM spend behaves like a utility bill with no breaker: every feature that calls a model is a meter that spins faster the more users like it. Treat cost the way you treat capacity. Set per-feature and per-tenant budgets with alerts at thresholds, not just a monthly invoice review. Route aggressively by job difficulty — autocomplete and classification belong on small fast models at a fraction of a cent, while long-document reasoning earns the premium model. Cache at multiple levels: identical-request caching is trivial and catches more traffic than teams expect, while semantic caching — serving a cached answer for a paraphrased repeat question — cuts costs meaningfully for support and search workloads. And measure cost per business transaction, not per API call; "each resolved support ticket costs $0.04 of inference" is a number your finance team can act on.

Quality Is a Production Metric, Not a Launch Gate

Model behavior drifts — because providers update weights, because your users find new edge cases, because your own prompt templates evolve. Quality evaluation therefore belongs in your operational cadence, not just your release pipeline. Maintain a golden set of a few hundred representative prompts with known-good responses, run it against your production configuration on a schedule, and score with a mix of exact checks and a judge model. Alert on regression the way you alert on error-rate increase. When you switch models or providers — which the cost section above will eventually make you want to do — the golden set is what turns a risky migration into a measured one. Treat the golden set itself as a living asset: rotate in prompts from real production traffic monthly, prune cases that no longer represent how users actually talk to the system, and version it alongside your prompt templates so a regression can be traced to the exact change that caused it.

The Failure Modes Nobody Warns You About

A few incidents recur across every team running these workloads. Silent context overflow: a growing prompt template plus a long user input exceeds the model's context window and truncates in the middle of instructions, producing subtly broken outputs with no error. Guard with explicit token counting before dispatch. Provider-side model swaps: an alias like "latest" starts pointing at new weights and your carefully tuned prompts behave differently overnight — pin model versions where providers allow it. Thundering-herd retries after a provider incident: when the upstream recovers, every queued request fires at once and you get throttled into a second outage — drain queues with rate-limited workers. And cost incidents from feature loops: an agentic feature that calls itself recursively can turn a bug into a five-figure invoice in an afternoon, which is why hard per-request and per-tenant spend caps belong in code, not in dashboards.

Start Boring

None of this requires exotic tooling. Timeouts, budgets, structured logs, a golden set, and a failover route cover the large majority of LLM production incidents. The teams that struggle are the ones that treated the model as magic and skipped the boring parts. The teams that sleep are the ones that treated it as one more unreliable dependency — instrumented, budgeted, and swappable like everything else in the stack.