Fine Tuning vs RAG

Every team building an LLM application hits the same question: should we fine-tune the model or use RAG? The answer is usually decided by gut feeling, which is why it's usually decided wrong. There's a clean rule that gets you to the right answer most of the time: RAG for knowledge, fine-tuning for behavior.

What Fine-Tuning Actually Does

Fine-tuning takes a pretrained model and continues training it on your own dataset of prompt/response pairs. The model's weights are updated to reflect the patterns in your examples. This changes how the model behaves — its tone, format, structure, and task-specific patterns.

Fine-tuning is good at teaching:

• Format and structure. Always respond in a specific JSON schema. Always produce a three-section report.
• Tone and style. Write like our brand voice. Be terse and technical, not conversational.
• Task-specific patterns. Given a code diff, produce a review comment. Given a log line, extract the error category.

Fine-tuning is bad at teaching facts. If you fine-tune on your knowledge base, the model might memorize some facts, but it will also hallucinate confidently about things it partially learned. There's no way to update a single fact without retraining, and no source to cite when the model produces an answer.

Full Fine-Tuning vs LoRA

Full fine-tuning updates every parameter in the model. For a 70B model, this requires enormous GPU resources — think multiple A100s for days or weeks. The result is a new model checkpoint that's expensive to store and serve.

LoRA (Low-Rank Adaptation) takes a smarter approach. Instead of updating all parameters, it freezes the original weights and trains small adapter matrices that modify the model's behavior. These adapters are typically 0.1% the size of the full model. You get most of the benefit of fine-tuning at a fraction of the cost.

QLoRA goes further: it quantizes the base model to 4-bit precision and trains LoRA adapters on top. This lets you fine-tune a 70B model on a single GPU. The quality tradeoff is surprisingly small for most tasks.

Full fine-tuning:  All 70B parameters updated     → weeks on 8x A100s
LoRA:              ~70M adapter parameters trained → hours on 1-2 A100s
QLoRA:             4-bit base + LoRA adapters      → hours on 1 GPU

What You Need for Fine-Tuning

High-quality examples. You need hundreds to thousands of input/output pairs that demonstrate exactly the behavior you want. Data quality dominates quantity — 500 excellent examples beat 50,000 sloppy ones. Each example should be a gold-standard demonstration of the task.

Evaluation set. Hold out 10-20% of your data for evaluation. Without this, you have no way to know if fine-tuning helped, hurt, or just memorized the training examples.

Awareness of catastrophic forgetting. Fine-tuning can degrade the model's general capabilities. If you train heavily on medical report formatting, the model might get worse at general conversation or coding. LoRA mitigates this because the base weights are frozen, but it's still something to test for.

What RAG Costs

RAG avoids retraining but introduces its own infrastructure and ongoing costs:

• Vector store infrastructure. You need to run and maintain a vector database. This is operational overhead — indexing pipelines, monitoring retrieval quality, managing document updates.
• Retrieval latency. Every query now includes an embedding step and a similarity search before the LLM call. This adds 50-200ms depending on your setup.
• Token cost per query. Retrieved chunks are stuffed into the prompt, which means more input tokens per request. If you're retrieving 5 chunks of 500 tokens each, that's 2,500 extra tokens per query. At scale, this adds up.
• Chunking and indexing pipeline. Documents need to be processed, chunked, embedded, and indexed. When documents change, the pipeline needs to re-run. This is an ongoing engineering effort, not a one-time setup.

The Decision Framework

Here's the practical framework:

Need current, private, or frequently changing facts? Use RAG. The model gets fresh information at query time, you can update documents without retraining, and answers come with citations.

Need consistent output format, domain-specific tone, or task-specific behavior? Fine-tune. You're teaching the model how to respond, not what to know.

Need both? Do both. Fine-tune for behavior, RAG for knowledge. A fine-tuned model that also retrieves from your knowledge base is a perfectly valid (and common) architecture.

Not sure yet? Start with prompt engineering. It's free, fast to iterate, and often sufficient. Many teams that jumped to fine-tuning or RAG would have been fine with a well-crafted system prompt and a few-shot examples.

Is the problem about WHAT the model knows?     → RAG
Is the problem about HOW the model responds?   → Fine-tune
Is the problem about BOTH?                     → Both
Is the problem unclear?                        → Start with prompting

The Mistake Most Teams Make

Most teams that fine-tuned for knowledge should have used RAG. Fine-tuning feels more "serious" and "custom," but it produces a model that confidently states things it half-learned, can't cite sources, and requires retraining to update. RAG gives you verifiable, updatable, permission-controlled knowledge at the cost of some infrastructure.

The reverse mistake is rarer but does happen: teams that use RAG when they actually need consistent behavior changes. If every response needs to follow a rigid format and your prompt engineering attempts keep producing inconsistent results, fine-tuning is the right tool.

A Practical Example

Imagine you're building a customer support bot for a SaaS product.

• Product documentation, pricing, feature details change regularly and are factual. This is a RAG problem — index your docs, retrieve relevant sections when users ask questions.
• Response style — always empathetic, always includes a ticket number, always ends with "Is there anything else I can help with?" — is a behavior problem. If prompting alone can't nail it consistently, fine-tune.
• Escalation logic — complex routing rules for when to hand off to a human — might be best handled outside the model entirely, in deterministic code.

Key Takeaways

• RAG for knowledge (facts, docs, data), fine-tuning for behavior (format, tone, task patterns) — this rule gets it right most of the time
• LoRA and QLoRA make fine-tuning accessible without massive GPU budgets by training small adapter matrices
• Data quality dominates quantity for fine-tuning — 500 gold-standard examples beat 50,000 mediocre ones
• RAG has real costs: vector infrastructure, retrieval latency, and extra tokens per query
• Start with prompt engineering before committing to either approach — it's free and often enough
• Most teams that fine-tuned for knowledge should have used RAG

Next, we'll explore AI agents — systems that go beyond single-turn generation to take actions and use tools.