LLM APIs — OpenAI, Anthropic, Gemini

You're going to spend a lot of time calling these APIs, so it's worth understanding the common patterns and the places where providers diverge. The good news: they've converged on roughly the same shape. The bad news: "roughly" does real work in that sentence.

The Common Shape

Every major LLM API follows the same basic structure: you send a messages array with roles, specify a model, and get a completion back. The core parameters are consistent across providers:

# The shape every provider shares (conceptually)
response = client.chat(
    model="the-model-name",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain DNS in one paragraph."},
    ],
    max_tokens=500,
    temperature=0.7,
)

Messages are an array of objects with role and content. Roles include system (instructions), user (human input), and assistant (model output). You include previous turns to maintain conversation history — the API is stateless, so you resend everything each time.

Temperature controls randomness (0 = deterministic, higher = more varied). Max tokens caps the response length.

Provider Differences

The divergences are small but will bite you if you're building a provider-agnostic abstraction.

Anthropic puts the system prompt in a separate top-level system parameter, not inside the messages array. Messages must start with a user role. Content can be an array of content blocks (text, images, tool results) rather than a plain string.

OpenAI puts the system message inside the messages array with role: "system". Supports a response_format parameter for JSON mode. Tool calls use a tools parameter with function definitions.

Google Gemini uses contents instead of messages, with parts instead of content. Roles are user and model (not assistant). System instructions go in a separate system_instruction field.

These differences are why abstraction libraries exist — and also why they're always slightly leaky.

Streaming

Streaming is non-negotiable for user-facing applications. Without it, the user stares at a blank screen for 2-10 seconds while the model generates the full response. With streaming, tokens appear as they're generated, and perceived latency drops to the time-to-first-token (TTFT), typically 200-500ms.

All providers use Server-Sent Events (SSE). You open a connection and receive a stream of small JSON payloads, each containing one or a few tokens:

# Anthropic streaming
with client.messages.stream(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain TLS."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Handle stream interruptions gracefully. Network drops happen, and you'll want to show the partial response rather than an error.

Tool/Function Calling

Tool calling follows the same pattern across providers: you describe available tools in a schema, the model decides when to call one and emits structured arguments, your code executes it and sends the result back.

tools = [{
    "name": "get_weather",
    "description": "Get current weather for a city",
    "input_schema": {
        "type": "object",
        "properties": {
            "city": {"type": "string"},
        },
        "required": ["city"],
    },
}]

The model's response will include a tool-use block instead of (or alongside) text. You execute the tool, then send a new request with the tool result appended to the messages. The model uses that result to formulate its final answer.

Structured Outputs and JSON Mode

When you need the model to return data in a specific schema (not just free text), you have options. OpenAI offers a response_format parameter that constrains the model to valid JSON matching a provided schema. Anthropic achieves similar results through tool calling — define a tool whose schema matches your desired output format, and the model will "call" it with structured data.

JSON mode is essential for any pipeline where model output feeds into downstream code. Parsing free text with regex is fragile; a guaranteed JSON structure is not.

Token Accounting and Cost

LLM pricing has input/output asymmetry: output tokens typically cost 3-5x more than input tokens. This matters because it changes your optimization strategy.

# Example pricing (illustrative)
Claude Sonnet: $3/M input, $15/M output
GPT-4o:        $2.50/M input, $10/M output
Gemini Pro:    $1.25/M input, $5/M output

Every request includes token counts in the response metadata. Log them. Multiply by rates. Aggregate by user, feature, and endpoint. Without this, your first bill will be a surprise.

Prompt caching reduces input costs for repeated prefixes. If your system prompt is 2,000 tokens and you're sending it with every request, cached reads can cost 90% less. Anthropic and OpenAI both support this — the first request pays full price, subsequent requests with the same prefix get cache hits.

Rate Limits

Every provider enforces rate limits on two dimensions: RPM (requests per minute) and TPM (tokens per minute). You'll hit one or the other depending on your traffic pattern — many short requests hit RPM, fewer long requests hit TPM.

When you get a 429 response, implement exponential backoff with jitter:

import random, time

def call_with_retry(fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fn()
        except RateLimitError:
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
    raise Exception("Max retries exceeded")

The jitter (random component) is important. Without it, all your retrying clients will hammer the API at the same intervals, creating thundering-herd problems.

Batch APIs

For offline workloads — processing a backlog of documents, running evaluations, generating training data — batch APIs offer 50% cost savings in exchange for higher latency (results in hours, not seconds). Upload a file of requests, get results back later. Worth using whenever you don't need real-time responses.

Key Management

API keys go on the server. Never in the client. Not in environment variables that get bundled into a frontend build. Not in mobile app binaries. Server-side only, loaded from a secrets manager, rotated regularly.

If you need to call an LLM from a client application, proxy through your own backend. This also lets you enforce per-user rate limits, log usage, and swap providers without shipping a client update.

Key Takeaways

• All major LLM APIs share the same basic shape (messages array, model, max_tokens, temperature) but diverge in specifics — Anthropic separates the system prompt, Gemini uses different field names
• Streaming via SSE is non-negotiable for UX — it reduces perceived latency from seconds to milliseconds
• Output tokens cost 3-5x more than input tokens — log token counts and costs per request from day one
• Prompt caching can cut input costs by up to 90% for repeated system prompts
• Rate limit handling requires exponential backoff with jitter, not fixed delays
• API keys are server-side only — proxy client requests through your backend

Next, we'll tackle the gap between a working demo and a system you can afford to run — costs, latency, reliability, and evaluation in production.