AI in Production — Costs and Scaling

Your prototype works. It answers questions, it calls tools, the demo went well. Now you need to run it at scale without a surprise five-figure monthly bill. The gap between "works in development" and "sustainable in production" is where most LLM projects stall, and it's almost always about costs, latency, and the lack of evaluation.

Cost Drivers

The formula is simple: (input_tokens x input_rate) + (output_tokens x output_rate) per request. What makes it expensive is the multiplication effect of conversations.

Every turn of a multi-turn conversation resends the entire history. Turn 1 sends your system prompt plus the user message. Turn 10 sends the system prompt, 9 previous turns, all tool call results, and the new message. This is the context tax — costs grow quadratically with conversation length if you're not careful.

A real example: a customer support agent with a 2,000-token system prompt, averaging 8 turns per conversation, with tool calls returning 500 tokens each. By turn 8, you're sending ~12,000 input tokens per request. At $3/M input tokens, that's $0.036 per turn — which sounds trivial until you multiply by 100,000 conversations per month.

Cost Optimization Levers

Prompt caching — providers cache your system prompt prefix across requests. If every request starts with the same 2,000-token system prompt, you pay full price once and ~10% for cached reads after that. This is the single highest-ROI optimization for most applications.

Model routing — not every query needs your most capable (and most expensive) model. Route simple queries to a smaller, cheaper model and escalate to the larger one only when needed. A classifier or set of heuristics can route 70% of traffic to a model that costs 10x less.

Shorten system prompts — that 3,000-token system prompt you wrote? Trim it. Every token is sent with every request. Eliminate redundancy, use concise instructions, remove examples that aren't pulling their weight.

Cap max_tokens — set output limits appropriate to the task. A classification task doesn't need 4,096 output tokens. Set it to 50.

Semantic caching — if users ask similar questions repeatedly, cache the responses. Hash the query embedding, check for a cache hit within a similarity threshold, return the cached response. This works well for FAQ-like traffic patterns.

Batch API — for offline processing (document summarization, evaluation runs, data enrichment), batch APIs offer 50% cost savings. The trade-off is latency: results arrive in hours, not seconds.

RAG instead of stuffing — instead of cramming everything into the context window, retrieve only the relevant chunks. This reduces input tokens dramatically for knowledge-heavy applications.

Latency

Two metrics matter: TTFT (time to first token) — how long the user waits before anything appears — and total generation time — how long until the full response is complete.

Streaming makes TTFT the metric users feel. A 200ms TTFT with streaming feels fast even if total generation takes 5 seconds. Without streaming, the user stares at a spinner for the full 5 seconds. This is why streaming is a production requirement, not a nice-to-have.

Other latency levers: use faster models for latency-sensitive paths, reduce prompt length (fewer input tokens = faster processing), set lower max_tokens, and co-locate your application with the API provider's infrastructure when possible.

Reliability

LLM APIs go down. Rate limits get hit. Models return malformed outputs. Your production system needs to handle all of this.

Multi-provider fallback — if your primary provider returns a 500 or times out, fall back to a secondary provider. This means maintaining compatible prompts across at least two providers.

Graceful degradation — when the LLM is unavailable, show a useful fallback. "I'm unable to help with that right now, here are some relevant docs" is infinitely better than a stack trace.

Circuit breakers — if a provider is failing consistently, stop sending it traffic for a cooldown period rather than burning through your retry budget. Standard circuit-breaker pattern from distributed systems.

# Simplified fallback pattern
providers = [anthropic_client, openai_client]
for provider in providers:
    try:
        return provider.chat(messages, timeout=10)
    except (APIError, Timeout):
        continue
return fallback_response()

Evaluation

You cannot improve what you cannot measure, and LLM outputs are notoriously hard to measure. But you have to try, because without evaluation, every prompt change is a gamble.

Golden datasets — curate a set of input/expected-output pairs that represent your key use cases. Run every prompt change against this set and compare. This is your regression test suite for prompts.

LLM-as-judge — use a strong model to evaluate the outputs of your production model. Define rubrics (accuracy, completeness, tone, safety) and have the judge score each response. Not perfect, but scales better than human review.

CI integration — run your evaluation suite in CI before merging prompt changes. If recall drops below a threshold, block the merge. Treat prompts with the same rigor as code.

# Prompt evaluation in CI (conceptual)
- name: Run eval suite
  run: python eval/run_evals.py --dataset golden_set.jsonl
  threshold: 0.92

Observability

Log everything, per request: the prompt sent, the completion received, token counts (input and output), latency (TTFT and total), cost, model used, and any errors. This is your debugging surface and your cost attribution data.

Aggregate these into dashboards: cost per feature, cost per user, p50/p95/p99 latency, error rates by provider, token usage trends. You'll need this data when someone asks why the AI bill tripled last month — and someone will ask.

Trace multi-step agent workflows end to end. When an agent takes 15 steps to answer a question that should take 3, you need the trace to understand why.

Guardrails

Input validation — reject or sanitize inputs before they reach the model. Length limits, content filtering, format validation.

Output filtering — scan model outputs before showing them to users. Check for PII leakage, harmful content, hallucinated URLs, or data from other tenants.

Cost caps per user — set maximum daily/monthly spend per user or per organization. When an agent loop goes haywire, this is what prevents the $40k bill. This is not hypothetical — teams have discovered five-figure bills from unbounded agent loops running overnight.

PII handling — strip PII from inputs before sending to the API if your data handling agreement doesn't cover it. Redact PII from logged prompts and completions.

Key Takeaways

• Conversation costs grow with turn count due to history resending — the context tax is the primary cost driver
• Prompt caching and model routing are the two highest-ROI cost optimizations for most applications
• Streaming is a production requirement, not a feature — TTFT is the latency metric users feel
• Multi-provider fallback and circuit breakers are essential; LLM APIs are not as reliable as traditional APIs
• Evaluate prompts like code: golden datasets, LLM-as-judge, and regression tests in CI
• Log prompts, completions, tokens, latency, and cost per request — you will need this data

Next, we'll look at what happens when things go wrong from a security perspective — the threat model for LLM applications.