Fine-tune or retrieve: the decision is about what changes, not what you prefer

Two mechanisms that get confused for each other

Fine-tuning changes the weights. Retrieval changes the prompt. They are frequently discussed as competing techniques for the same job, which is why so many teams pick one on instinct and then spend a quarter discovering why it does not fit.

They solve different problems. Fine-tuning is good at form: tone, output shape, task adherence, domain idiom. Retrieval is good at content: specific facts, documents that did not exist at training time, anything you need to cite.

The decision rule

Start with two questions. First: how often does this knowledge change? Anything on a cadence shorter than your retraining cycle belongs in retrieval, because a fine-tuned model goes stale silently and you will not notice until a user does. Second: do you need to show where the answer came from? If yes, retrieval is not optional — a weight update cannot produce a citation.

Only after both questions come back favourable should you consider fine-tuning. The remaining case is narrow but real: you need a stable behaviour or format that prompting cannot reliably produce, at a token budget or latency that retrieval makes worse.

The failure mode of each

Fine-tuning fails quietly. The model becomes fluent in your domain vocabulary while remaining confidently wrong about current facts, and because it sounds right, nobody checks. It also creates a maintenance obligation: every base model upgrade means repeating the training and re-running the evaluation.

Retrieval fails visibly, which is a feature. Bad retrieval produces obviously irrelevant context, and you can measure it. The cost is that you now own a pipeline: parsing, chunking, embedding, index freshness, and permission filtering. Most quality problems in RAG systems are in that pipeline, not in the model.

What usually works

Retrieval first, for nearly everything, because it is measurable and reversible. Then fine-tune narrowly for form, once you have traces showing a consistent behavioural gap that prompting and few-shot examples cannot close. Doing it in that order means you will have the evaluation set ready before you need it, which is the thing most teams lack when they start.