Tokens, Embeddings, and Context Windows

If you want to understand why LLMs behave the way they do in practice — why they cost what they cost, why they forget things, why they're bad at spelling — you need to understand three concepts: tokens, embeddings, and context windows. These are the mechanical constraints that shape every interaction with a language model.

Tokens: The Atomic Unit

LLMs don't process words or characters. They process tokens — subword units created by a tokenizer (usually Byte Pair Encoding, or BPE). The tokenizer scans the training corpus, finds the most frequently occurring character sequences, and builds a vocabulary of subword units.

Common words become single tokens. Rare or long words get split into pieces:

"the"          →  [the]              (1 token)
"unbelievable" →  [un|bel|iev|able]  (4 tokens)
"ChatGPT"      →  [Chat|G|PT]        (3 tokens)

A rough rule of thumb for English: one token is about 4 characters, or roughly 0.75 words. But this ratio varies a lot. Code tokenizes less efficiently than prose — whitespace, brackets, and unusual variable names each eat tokens. Non-English languages often tokenize much worse because the training corpus was English-heavy, so fewer common subword units exist for other scripts.

This is why "count the r's in strawberry" trips up LLMs. The model never sees individual characters. It sees tokens like str|aw|berry and has no built-in mechanism to inspect what letters are inside a token. It has to reason about it indirectly, and that reasoning is unreliable.

Embeddings: Tokens as Vectors

Once text is tokenized, each token is mapped to a high-dimensional vector — a list of hundreds or thousands of floating-point numbers. This is the embedding.

These vectors aren't hand-designed. They're learned during training, and they end up encoding semantic relationships as geometric relationships. Tokens with similar meanings land near each other in this vector space. The classic example:

king - man + woman ≈ queen

This works because the vectors for "king" and "queen" differ in roughly the same direction as the vectors for "man" and "woman." The model didn't learn an explicit rule about gender — this structure emerged from predicting next tokens across billions of sentences.

Embeddings are why LLMs can generalize. When the model learns something about "dogs," it partially applies to "puppies" and "canines" because their embeddings are nearby in vector space. This is also the foundation of semantic search: you can find related documents by comparing the geometric distance between their embedding vectors, even if they share no words in common.

Context Window: The Hard Limit

The context window is the total number of tokens the model can process in a single call — input and output combined. GPT-4o has a 128K token window. Claude has up to 200K. Some models go further.

This window is everything the model "sees." There is no persistent memory between API calls. Each request starts from scratch. If your prompt plus the model's response would exceed the context window, you either truncate or you get an error.

The context window matters for two practical reasons:

Cost scales with tokens. API pricing is per-token, both input and output. A 50K-token prompt costs real money, especially at high volume. This makes context management an engineering concern, not just a UX concern.

Attention cost is quadratic. Self-attention compares every token to every other token. Doubling the sequence length quadruples the computation. This is why longer contexts are slower and more expensive, even with optimizations like FlashAttention.

Lost in the Middle

Context windows have a quality problem, not just a size problem. Research shows that LLMs have strong recall for information at the beginning and end of the context, but performance degrades for content buried in the middle. This "lost-in-the-middle" effect means that simply cramming more text into the context doesn't guarantee the model will use it effectively.

This has practical implications for RAG systems and long-document processing. If you're stuffing retrieved documents into a prompt, the order matters. Key information should go at the beginning or end of the context, not sandwiched between less relevant chunks.

Practical Implications

Estimate token counts before sending requests. Most APIs provide a tokenizer endpoint, or you can use tiktoken (OpenAI) or the relevant provider's library locally. A 10-page document is roughly 3,000-4,000 tokens. A full codebase can blow through a context window fast.

Manage context deliberately. In multi-turn conversations, the full history is re-sent every call. Conversations get expensive and eventually hit the window limit. Strategies include summarizing older messages, dropping low-value turns, or using a sliding window over recent history.

# rough token estimation
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("your text here")
print(f"Token count: {len(tokens)}")  # plan your budget

Non-English and code need more tokens. Budget 2-3x more tokens for the same content length when working with CJK languages, code with verbose naming, or heavily formatted text.

Key Takeaways

• Tokens are subword units (roughly 4 English characters each), not words or characters — this explains most "dumb" LLM failures
• Embeddings map tokens to high-dimensional vectors where distance encodes meaning — the foundation of semantic search and generalization
• The context window is a hard token budget for input + output combined, with no memory between calls
• Attention cost is quadratic in sequence length, so longer contexts are disproportionately expensive
• Models recall information at the start and end of the context better than the middle
• Cost management is token management — estimate, monitor, and trim

Next, we'll look at how to use these constraints to your advantage with prompt engineering.