Should we fine-tune a model or use RAG?

Use retrieval when the correct answer changes as your documents change, and fine-tune when the facts are stable but the model writes them in the wrong shape, tone, or schema. The test is one question: when the underlying truth changes, does a document have to change, or only the phrasing? A document means retrieval. Phrasing means fine-tuning. Most systems that have to cite a source end up retrieval-first, with fine-tuning added later only if the leftover errors are about form.

Is fine-tuning cheaper than RAG?

Sometimes per call, rarely over two years. Fine-tuning can shrink a long few-shot prompt into a short one, which cuts token cost at volume, though inference on a fine-tuned model carries a premium on some lines: OpenAI lists fine-tuned gpt-4.1 at 3.00 dollars per million input tokens against 2.00 dollars for the standard model, read 4 August 2026. The costs that decide it are recurring ones, chiefly retraining when the data moves or a new base model ships, hourly hosting for a deployed custom model, and a full retrain when the base model retires.

Can you do both RAG and fine-tuning on the same system?

Yes, and vendor documentation treats the combination as a normal use case rather than an exotic one. Microsoft's fine-tuning guidance lists enhancing retrieval-based performance as a reason to fine-tune, on the grounds that a tuned model uses retrieved passages more reliably and filters out irrelevant ones. The sensible order is retrieval first, measure, then fine-tune only if the remaining errors are about form rather than fact. Running both doubles the upkeep, so it should be earned by a measured gap and not assumed at the start.

RAG vs Fine-Tuning: Which One Does Your Business Need?

The short answer

Fine-tuning changes how a model behaves. Retrieval changes what it can see. If the right answer changes when your documents change, you need retrieval, because a fine-tuned model's knowledge is frozen at the moment you trained it. If the answer is stable and the trouble is that the model says it in the wrong shape, tone, or schema, fine-tuning is the cheaper fix.

Most explainers stop there. The expensive decision is the next one: what each option costs to own for two years, who has to retrain it, and what happens the morning your base model gets a shutdown date. Both approaches sit inside the larger problem of stopping an AI system from making things up, and neither one solves it alone.

The mechanics, in one table

	Retrieval (RAG)	Fine-tuning
What it changes	What the model sees at question time	How the model responds, before any question
Where your knowledge sits	In an index you control	In the model weights
How you update it	Edit or add a document, reindex it	Assemble new examples, retrain, redeploy
Lag from change to correct answer	Minutes to hours	Days to weeks
Can it cite a source	Yes, that is the point	Not from what it was trained on
Typical failure	Retrieves the wrong passage, answers from it confidently	Learns the style but invents the facts

Two vendor sources are worth reading first, because both sell fine-tuning and both hedge it. OpenAI's optimization guide puts prompting ahead of tuning, saying the prompt engineering process "may be all you need", and the benefit it claims for fine-tuning is efficiency rather than knowledge: "You can use shorter prompts with fewer examples and context data, which saves on token costs at scale" (OpenAI, model optimization guide 1). Microsoft's list of good fine-tuning uses runs to style and tone, output schemas, tool-calling consistency, and distilling a large model into a small one (Microsoft, fine-tuning considerations 2). Teaching a model facts that keep changing is on neither list.

A decision rule you can apply in ten minutes

Ask one question about your workflow: when the underlying truth changes, what has to change? If the answer is "a document", you want retrieval. If the answer is "nothing, the model just has to say it differently", you want fine-tuning.

Your situation	Start with	Why
A human must be able to open the source behind an answer	Retrieval	Weights have no source to point at
The facts move weekly or faster	Retrieval	No retraining cadence keeps up
Access rules differ by user, tenant, or role	Retrieval	Permissions live on records, not on weights
Output must match a strict schema on every call	Fine-tuning	Prompts drift, weights do not
A house voice that prompting keeps losing	Fine-tuning	Style is exactly what training moves
High volume behind a long few-shot prompt	Fine-tuning	Shorter prompt, smaller model, lower unit cost
Classification into a fixed label set at scale	Fine-tuning	Finite answers, stable over time
Expertise lives in people's judgement, not documents	Fine-tuning	Nothing to retrieve
Changing facts and a fixed output shape	Both, retrieval first	Measure the gap before paying for two systems

Check the door is still open before you budget

This is new and it changes the arithmetic. OpenAI is winding down its self-serve fine-tuning platform on a published schedule. Since 7 May 2026, organisations that had never fine-tuned cannot start. Since 2 July 2026, organisations that have not run inference on a fine-tuned model in the past 60 days cannot create new jobs. On 6 January 2027, "Active existing customers will no longer be able to create new fine-tuning jobs" (OpenAI deprecations 3, read 4 August 2026). Inference on models you already trained keeps running until the base model itself is deprecated.

That does not end fine-tuning. It relocates it. Azure's Foundry fine-tuning and Amazon Bedrock's model customization are both still open, and open-weight models can be tuned on hardware you rent or own. What moves is the operating model. A self-serve button was one vendor relationship. A Bedrock customization job or a self-hosted tune is a training pipeline your team owns, with the on-call rota that implies. Price the second thing, because that is what is on offer now.

Retraining cadence: the line item nobody quotes

Microsoft states the upkeep plainly in its own product documentation: "Fine-tuning may need to be repeated whenever the data is updated, or when an updated base model is released. This involves monitoring and updating regularly" (Microsoft 2). Read that as a recurring cost with two independent triggers. Your data moves on your schedule. Your base model moves on the vendor's, and you do not get a vote.

Retrieval has upkeep too, and anyone who says otherwise is selling. You maintain an ingestion pipeline, a chunking strategy, and an index that drifts as documents are added, superseded, and deleted. The difference is granularity: correcting a fact touches one document, not the whole model.

What happens when the base model is deprecated

The dates are public, and most pilots have not planned for them.

OpenAI commits to "At least 6 months" of notice before retiring a generally available model, and states that "Inference on fine-tuned models will continue to be available until the base models are deprecated". Your custom model dies with its parent. The same page lists the fine-tuned bases "ft-gpt-3.5-turbo" and "ft-gpt-4" shutting down on 23 October 2026.

Amazon Bedrock runs a comparable clock with a sting in it. A model "will be in the Legacy state for at least 6 months before the EOL date", and "once the model is in Legacy state, you cannot create new fine-tuning jobs on that model". During the extended-access part of that window, Bedrock warns, "you should expect higher pricing, which will be set by the model provider" (AWS, Bedrock model lifecycle 4). Staying put gets more expensive first, then stops being possible.

For a retrieval system, a base model retirement is a configuration change and a rerun of the eval suite. For a fine-tuned system it is a fresh training run against a new base, new hyperparameters, and full revalidation, all inside the notice window. Budget it as a project, not a ticket.

The honest counterweight: retrieval has the same problem in its embedding model. Change it and you re-embed the whole corpus, and embeddings are usually a third-party dependency too. Anthropic states that it "does not offer its own embedding model" and points customers at Voyage AI (Anthropic, embeddings 5). Two vendors, two clocks, neither yours.

What it costs to correct one wrong answer

Ask a vendor for this number, because it is the one you will pay most often.

Under retrieval, someone finds the passage the answer came from, corrects or adds the document, reindexes it, and reruns the eval case. Hours, usually, and often done by whoever owns the content rather than an engineer. The citation trail is what makes it hours instead of days: you know which record produced the claim.

Under fine-tuning, someone writes new training examples, runs a training job, evaluates the whole model against the full battery, and redeploys. Days to weeks, an ML engineer, and the fix is probabilistic. You cannot promise that one added example produces one specific answer.

Neither fix is safe without an eval suite already in place, or you are guessing whether you improved anything. That measurement problem is its own discipline: how to evaluate a RAG system.

Two-year total cost of ownership

Prices below were read on 4 August 2026 and will move. The last column matters most: it tells you which way your own numbers slide.

Cost line	Retrieval	Fine-tuning	What swings it
Build	Ingestion, chunking, index, retrieval evals	Dataset construction and labelling, training runs, evals	Corpus messiness against label quality
Per-token inference	Base model list price	A premium on some lines: OpenAI lists fine-tuned gpt-4.1 at 3.00 in and 12.00 out per million tokens, against 2.00 and 8.00 standard	Provider and tier; a smaller tuned model can undercut a larger base one
Training	None	OpenAI lists 25.00 dollars per million tokens to train gpt-4.1	Dataset size times epochs, times attempts
Standing infrastructure	Index hosting, embedding calls on new content	Azure charges "an hourly hosting cost regardless of whether" calls are made; Bedrock charges 1.95 dollars a month to store each custom model on its Llama line	Idle time punishes fine-tuning, corpus size punishes retrieval
Keeping content current	Reindex changed documents, cheap per change	Retrain when the data moves	How often your facts actually move
Base model retirement	Change a model id, rerun evals	Retrain on a new base, revalidate, redeploy, inside a six-month window	Vendor cadence, not yours
Fixing one wrong answer	Hours, document level, traceable	Days, model level, probabilistic	Whether the eval suite exists yet
Who is on call	Whoever owns content and retrieval	Someone who can run and judge a training job	Whether you have that person in-house
The hidden one	Swapping the embedding model re-embeds everything	A retired base model forces a rebuild	Both are timed by a vendor

Sources: OpenAI pricing 6, Microsoft fine-tuning how-to 7, Amazon Bedrock pricing 8.

When fine-tuning is the right call

Fine-tune when the output shape is the problem and the facts are not. A classifier over a fixed label set. A strict schema that prompting keeps drifting away from. A house voice. Tool selection that a long tool list keeps getting wrong. High volume where the few-shot prompt is the real cost, and a smaller tuned model on a short prompt wins on unit economics even after the tuned-inference premium. And the case retrieval cannot serve at all: expertise that lives in people's judgement and was never written down.

Do that arithmetic yourself. A 4,000-token prompt on every call, cut to 400 by fine-tuning, absorbs a per-token premium and still finishes ahead. Run it on your own traffic, and put the retraining cadence in the two-year figure rather than the first-month one.

Doing both, and what neither one fixes

Combining them is ordinary practice, not a hedge. Microsoft lists enhancing retrieval-based performance among its reasons to fine-tune, because a tuned model can learn to use retrieved passages well and ignore irrelevant ones. The order that keeps costs sane is retrieval first, measure, then fine-tune only against a gap you can point at. Starting with both means paying two upkeep bills to fix a problem you have not diagnosed. Building and operating that is our AI agents and grounded RAG work.

One limit applies to both. Grounding and citations do not make mistakes impossible; they make them visible and traceable instead of silent. Fine-tuning does the opposite, making a system fluent about being wrong. Neither buys accuracy on its own. Evals and an honesty gate do, and that is the part most teams skip and rebuild a year later.

RAG or fine-tuning: what each one costs to own