What Problem Are You Actually Solving?
Before choosing between RAG and fine-tuning, get specific about what you’re trying to fix. Most AI customisation projects fail not because the wrong technique was used, but because the team never precisely defined the failure mode they were solving.
The two most common problems are: the model doesn’t know the right answer (knowledge gap), or the model doesn’t behave the right way (style or format gap). RAG primarily addresses knowledge gaps. Fine-tuning primarily addresses behaviour gaps. Conflating the two leads to expensive solutions that only partially work.
How RAG Works
Retrieval-Augmented Generation adds a retrieval step before inference. At query time, the system searches a knowledge base — typically a vector database of embedded documents — and injects the most relevant chunks into the model’s context window alongside the user’s question.
The model then answers using both its pre-trained knowledge and the retrieved context. The key insight is that you never modify the model itself; you just give it better information to work with.
RAG is essentially structured prompting at scale. The sophistication is in the retrieval pipeline — chunking strategy, embedding quality, re-ranking, and query reformulation — not the model itself.
How Fine-Tuning Works
Fine-tuning modifies the model’s weights on a curated dataset of input-output pairs. The model learns to replicate patterns it sees in training data, effectively updating its internal knowledge and behaviour at a fundamental level.
This is powerful for teaching a model a consistent persona, a specific output format, a domain vocabulary, or task-specific reasoning patterns — things that are difficult to convey through prompting alone. It is substantially more expensive and slower to iterate on than RAG.
When to Choose RAG
RAG is the right choice when:
- Your knowledge base changes frequently (product documentation, policies, live data feeds)
- You need citations and source attribution for compliance or trust reasons
- You need to be able to audit or update specific facts without retraining
- Your use case involves large bodies of text that won’t fit in a context window
- You want to get to production quickly — RAG pipelines can be built and iterated on in days, not weeks
When to Fine-Tune
Fine-tuning earns its cost when:
- You need a specific communication style or tone that prompting can’t reliably enforce
- The model needs to output a precise, structured format consistently (e.g., domain-specific JSON schemas)
- You’re dealing with a highly specialised domain where the base model’s vocabulary and reasoning patterns are inadequate
- Latency is critical and you need to move complex instructions out of the prompt and into the weights
- You have thousands of high-quality labelled examples to train on
When to Use Both
Many mature production systems use both techniques together. Fine-tuning establishes the model’s behaviour (tone, format, reasoning style); RAG provides the knowledge (current facts, proprietary data, source documents).
A legal AI assistant, for example, might be fine-tuned on thousands of legal briefs to write in the correct format and reasoning style, while RAG gives it access to current case law and the client’s specific documents at query time.
A Decision Framework
When a client asks us which approach to use, we start with three questions:
- Does the problem change over time? If yes, lean RAG — you don’t want to retrain every time your data updates.
- Is this a knowledge problem or a behaviour problem? Knowledge → RAG. Behaviour → fine-tuning.
- How much labelled data do you have? Fine-tuning without sufficient high-quality examples produces worse results than good prompting. If you have fewer than ~1,000 examples, start with RAG and prompting first.
In our experience, roughly 70% of enterprise AI customisation use cases are best solved with RAG alone or RAG plus careful prompting. Fine-tuning is the right answer for the remaining 30% — and worth every dollar when it is.