The short version
- The question is not which technique is better; it is what kind of truth you need the model to speak. Changeable facts want retrieval. Live per-user records want request-time reads. Format, labels, and voice want prompts first, fine-tuning second.
- Fine-tuning's hidden price is operational: every content change becomes a training run, a wrong sentence becomes a tendency you cannot delete, and you still need fallbacks it cannot provide.
- The case study: a support bot whose answers change when a person edits an FAQ from Slack and whose refund verdicts come from database reads — two needs fine-tuning serves badly, which is why no support model was ever trained.
Every AI application hits this fork within its first month: the model sounds generic, someone says "we should train it on our data," and someone else says "no, we should use RAG." Both camps are usually arguing past each other, because "our data" is not one thing — it is several kinds of truth with different shapes, and each shape has a different right tool.
This guide gives you the decision framework, the honest costs of each path, the cases where fine-tuning genuinely wins, and then the real decision as it played out on a production support bot — including what we did instead of the training run everyone expects.
What each technique actually does
Fine-tuning continues training a model on your examples, nudging its weights so the behaviors in your examples become its defaults. The knowledge and style end up inside the model. Changing them means producing new examples and training again; removing one bad behavior means figuring out which examples taught it — there is no delete key for a weight.
RAG (retrieval-augmented generation — full guide here) leaves the model untouched and changes what it reads: fetch your relevant content at question time, paste it into the prompt, instruct the model to stay inside it. The knowledge lives outside the model, in a store you can edit, audit, and delete from.
There is a third option both camps forget, and for product work it does the heaviest lifting: request-time reads. Facts about this user right now — their order, their account state, the flag on their record — belong in neither weights nor a document index. They are a database query whose result you place in the prompt, in code.
How you should choose
Split "our data" into the kinds of truth it contains, then match each to its tool:
Match the technique to the kind of truth
Choose RAG when the truth is editable prose
Policies, product docs, FAQs, pricing rules — content that changes on its own schedule and must be correctable the day it is wrong. Retrieval gives you three properties training cannot:
- Same-day updates. Edit the entry; the next question uses the new text. No data pipeline, no training job, no deploy.
- A delete button. A wrong answer traces to a retrieved entry id, and that id can be removed. In a fine-tuned model, a wrong answer is a tendency — you retrain and hope.
- An audit trail. You can log exactly which passages produced an answer. No one can tell you which training examples produced a completion.
Choose request-time reads when the truth is a record
No document pile can know which tickets this user holds at this moment, and weights are worse — they would freeze Tuesday's truth into Wednesday's answers. If the answer depends on who is asking and what their account says, the fact must be fetched in code when the question arrives. Stuffing more documents into the prompt does not create this fact; this is the error behind most "our RAG isn't working" complaints.
Choose fine-tuning when the task is a skill, not a fact
The legitimate cases are real:
- Format discipline at volume — a classifier or extractor that must emit exact JSON labels, where instructions underperform examples and you run it millions of times (the fine-tuned model can also be smaller and cheaper per call).
- Voice you cannot specify — a style that survives every phrasing, demonstrated rather than described.
- Latency/cost compression — distilling a prompted behavior into a smaller model once the behavior is stable.
Note what is on that list: formats, styles, skills. Not facts. The practical ordering: prompt first (temperature 0, schema, examples in the prompt), measure, and fine-tune only when the prompted version demonstrably fails at your scale.
Price the operations, not just the API bill
The comparison that matters is loops, not invoices. RAG's loop: a person edits text; seconds; non-engineers can own it. Fine-tuning's loop: collect examples → clean → train → evaluate → deploy; days; engineers own it forever; and it reruns every time your product changes a rule. And one more asymmetry — when the provider is down, a RAG system still has its authored fallback text to serve. A fine-tune gives you no fallback at all; it is the model that just went down.
How the decision played out on a real project
The support bot on the events platform I work on never got a fine-tuned model — and not because we benchmarked one and it lost. We never ran the experiment, because mapping our truths to the matrix above settled it before any training job was worth its pipeline.
Truth one: policy prose that changes from Slack
The platform's support knowledge is FAQ pairs — how publishing works, what co-hosts can do, how organizer-set refund policies behave. Those sentences change when the product changes, and the team edits them from Slack: faq edit <id> replaces one vector in the index, and the very next question answers from the new text. Had those sentences been trained into weights, every policy tweak would be a training run, and the bot would keep an old policy as a tendency until the next one. The same-day delete button was a hard requirement; training cannot provide it.
Truth two: records read when the question arrives
Refund questions are answered from the account: the specialist agent loads the user's upcoming ticketed events and reads the event's refund flag, resolves a state in code, and only then lets the model write — with authored fallback paragraphs per state for when the model call fails. The split, in one picture:
Where each kind of truth lives
Stable sentences
live in retrieval
Product rules that hold across events. Edited from Slack, deleted by id, found by meaning. A training run is not needed to change a sentence.
Live records
read at request time
This person’s tickets, this event’s refund flag. No document pile can know them, and weights would freeze yesterday’s values.
Written fallbacks
no model at all
Each state has an authored paragraph for when the provider fails. Fine-tuning gives you neither a fallback nor a delete button.
That third column is the quiet argument-ender. Each refund state has a hand-written fallback with the real link, so a provider outage degrades the bot into accurate static text. No amount of fine-tuning buys that property, because the fallback's whole point is to not involve the model.
The one classifier that still is not a fine-tune
The platform does run a model as a strict classifier — a dispute-risk check that must answer JSON-only with a confidence score (the tiered pipeline is its own guide). That is exactly the shape fine-tuning serves. It is still a prompted hosted model, temperature 0, because at our volume the prompted version meets the bar. The day it misses at a scale that matters, fine-tuning that classifier — small, evaluable, format-bound — would be the first legitimate training run in the system. The support bot's knowledge would still not be one.
Pitfalls to avoid
- Fine-tuning on support transcripts to "learn our answers." You bake yesterday's policies and one-off exceptions into permanent tendencies — plus whatever PII was in the transcripts.
- Expecting RAG to know per-user facts. That is a database read. Retrieval fetches prose, not state.
- Treating the techniques as rivals. Mature systems use both: retrieval for facts, with maybe a fine-tuned classifier at the edges. The mistake is using either where the other belongs.
- Skipping the prompted baseline. If you have not tried temperature 0 with a schema and good examples, you do not yet know whether you need a fine-tune.
FAQ
Per call it can be — no retrieved tokens, possibly a smaller model. But add the recurring pipeline: every content change is a new training run. For knowledge that changes, the operational loop usually erases the per-call savings.

Written by
Rushit Jivani
Senior Software Architect & AI Application Engineer. Architecture that shows up in crash rate, performance and how fast the team ships.