Fine-tune or RAG? Choosing the right way to teach a model your business
Both make a general model behave like it knows your domain, but they solve different problems. A practical guide to picking the right one — and knowing when you need both.
Every team adopting LLMs hits the same question within the first month: the base model doesn't know our products, our policies, our tone. Do we fine-tune it, or do we build retrieval around it? The question gets framed as a rivalry, and it isn't one.
Fine-tuning and RAG solve different problems. One teaches the model how to behave; the other tells it what is true right now. Most of the wrong decisions we see come from using one to do the other's job.
Which problem are you solving?
Teaches it
- RAG
- Knowledge — facts it should cite
- Fine-tuning
- Behaviour — format, tone, task shape
Updating it
- RAG
- Re-index; minutes
- Fine-tuning
- Retrain; a cycle
Citations
- RAG
- Natural — you have the sources
- Fine-tuning
- Not available
Start here when
- RAG
- Almost always
- Fine-tuning
- RAG works but the style or format won't hold
| RAG | Fine-tuning | |
|---|---|---|
| Teaches it | Knowledge — facts it should cite | Behaviour — format, tone, task shape |
| Updating it | Re-index; minutes | Retrain; a cycle |
| Citations | Natural — you have the sources | Not available |
| Start here when | Almost always | RAG works but the style or format won't hold |
RAG is for knowledge, fine-tuning is for behavior
The cleanest way to decide is to ask what kind of gap you're closing. If the model gives well-formed answers that are factually wrong about your domain — wrong prices, outdated policies, products it has never heard of — that is a knowledge gap, and retrieval closes it. The facts live in a store you control, get fetched at query time, and the model reasons over them.
If the model knows enough but answers in the wrong shape — wrong tone, wrong format, ignores your conventions, needs three paragraphs of instructions to produce one clean output — that is a behavior gap, and fine-tuning closes it. You are not adding facts; you are baking in a skill so you stop paying for it in prompt tokens on every call.
Why RAG should almost always come first
Start with retrieval even when you suspect you'll fine-tune later. RAG is cheaper to stand up, updates instantly when the underlying documents change, and — critically — it can cite its sources. When a user challenges an answer, you can show the passage it came from. A fine-tuned model can only shrug.
There is also a staleness problem fine-tuning cannot escape. Facts trained into weights are frozen at training time. If your pricing changes on Tuesday, a retrieval index reflects it Tuesday afternoon; a fine-tuned model repeats the old number until you retrain, redeploy and re-evaluate. For anything that changes faster than your training cadence, weights are the wrong place for it.
When fine-tuning genuinely earns its cost
Fine-tuning stops being optional in a few recognizable situations. The pattern they share: the requirement is about consistent behavior at scale, not about facts.
The prerequisite people underestimate is data. A useful fine-tune needs hundreds to thousands of high-quality input-output examples that show exactly the behavior you want. If you cannot produce that set — or the examples you have are inconsistent — fix the data problem first, because a model trained on noise learns noise.
- A style or format the prompt can't reliably pin down — domain-specific structure, a regulated tone, strict output schemas
- Prompt bloat: your instructions and few-shot examples have grown so long that latency and cost hurt on every call
- A smaller model that needs to match a bigger one on a narrow task — fine-tuning is how you buy that gap down
- Vocabulary and phrasing the base model consistently mangles — internal jargon, product codes, non-English domain terms
The strongest systems use both
In mature deployments the two stop being alternatives. A model fine-tuned to reason well over retrieved passages — to cite properly, abstain when the context doesn't support an answer, and follow your output format without being told — sitting on top of a live retrieval index is better than either technique alone.
The division of labor is clean: retrieval owns the facts, the fine-tune owns the behavior. Knowledge stays fresh without retraining, and behavior stays consistent without a thousand-token prompt.
Which one, for which symptom
- It doesn't know your factsRAG
- Facts change weeklyRAG
- You need citationsRAG
- It knows the facts but won't hold the formatFine-tune
- You need a house voice, consistentlyFine-tune
Decide with an eval, not a debate
Whatever you choose, the decision should be a measurement. Build a golden set of real queries from your domain first, score the base model with a good prompt, then score RAG on top of it. Only reach for fine-tuning if a gap remains that retrieval and prompting demonstrably cannot close — and score that too.
This ordering saves teams from the most expensive mistake in the space: spending weeks on a fine-tune to fix a problem that was actually a retrieval problem, or that a better prompt would have solved for free. Cheapest lever first, and let the numbers move you to the next one.