RAG vs fine-tuning: which approach fits your product?

Hephaestus Technology7 min read

In the RAG vs fine-tuning choice, use retrieval-augmented generation (RAG) when the model needs to know your facts, and fine-tuning when it needs to behave in a specific, consistent way. RAG fetches relevant passages from your content at question time and adds them to the prompt; fine-tuning changes the model itself by training it on example inputs and outputs. Most products should start with good prompting plus RAG, and consider fine-tuning only for narrow tasks where examples in the prompt are not enough.

Key takeaways

  • RAG adds your current facts to the prompt at question time; fine-tuning teaches a model a behaviour, style or format from examples.
  • If your whole knowledge base fits comfortably in the context window, Anthropic suggests you can skip RAG and put it in the prompt, ideally with prompt caching.
  • Fine-tuning availability is shifting: OpenAI says it is winding down its fine-tuning platform, while Google's Vertex AI lists supervised, preference and reinforcement tuning for several Gemini models.
  • RAG makes GDPR deletion and access requests far easier, because personal data stays in a database you control rather than inside model weights.
  • Measure both options on the same evaluation set before you commit; retrieval quality is often the real bottleneck.

What is the difference between RAG and fine-tuning?

RAG changes what the model sees; fine-tuning changes what the model is. Anthropic describes RAG as a method that retrieves relevant information from a knowledge base and appends it to the user's prompt. The model is unchanged, so updating its knowledge means updating your documents.

Fine-tuning trains an existing model further on your own examples. OpenAI's guidance lists the benefits as fitting more example inputs and outputs than could fit within the context window, using shorter prompts, and training a smaller, cheaper and faster model to excel at a particular task. Google's Vertex AI documentation recommends supervised tuning for tasks such as classification, sentiment analysis, entity extraction.

How do RAG and fine-tuning compare?

The two approaches solve different problems, which the table makes clear.

RAG Fine-tuning
Best for Answering from your documents, product data, policies Consistent format, tone, classification, narrow repeatable tasks
Keeps facts current Yes, update the index No, retrain to change facts
Shows sources Yes, can cite retrieved passages No
Upfront work Chunking, embeddings, search, access rules Collecting and labelling hundreds or more examples
Running cost Extra input tokens per question, plus search Shorter prompts; tuned model prices vary by provider
Provider lock-in Low, works with most models Higher, the tuned model lives with one provider
GDPR requests Delete or correct the source record Personal data may be embedded in weights
Main failure mode Retrieves the wrong or no passage Confident answers from stale or wrong training

When is RAG the right choice?

Choose RAG when answers depend on information that is private, large or changing. Typical cases are support assistants over a help centre, internal policy search, product catalogues and contract or document Q&A. RAG also lets you enforce permissions at retrieval time, so a user only gets passages from documents they are allowed to see.

Retrieval quality is where RAG projects succeed or fail. In Anthropic's published experiments, adding context to each chunk before embedding reduced failed retrievals by 35%, combining that with keyword (BM25) search reduced them by 49%, and adding reranking reduced them by 67%. The lesson for a small team is that search engineering matters at least as much as the choice of model.

Embedding and storage costs are usually modest. OpenAI, for example, lists text-embedding-3-small at $0.02 per million tokens, and hosted vector storage at $0.10 per GB per day with 1 GB free.

When can you skip RAG altogether?

You can skip RAG when your knowledge base is small enough to send in full. Anthropic's guidance is that for a knowledge base under about 200,000 tokens, roughly 500 pages, you can just include the entire knowledge base in the prompt. Context windows have grown since: Anthropic's pricing page states that Claude 4.6 and later models (except Haiku 5.5) include a 1M token context window at standard pricing. Sending the same large prefix on every request is expensive without caching, so pair this approach with prompt caching; our guide to reducing LLM costs explains how.

When is fine-tuning worth it?

Fine-tuning is worth it when prompting and examples cannot get a narrow task right consistently, or when you need a smaller model to do one job cheaply at high volume. Good candidates are classifying tickets into your own categories, extracting fields into a strict schema, matching a house style, or correcting a specific instruction-following failure. Google lists preference tuning for subjective qualities like tone, and reinforcement tuning for multi-step reasoning, complex domain rules, verifiable tasks such as code or SQL generation.

Fine-tuning is a poor way to add facts that change, because every change means retraining. OpenAI's guidance places current or private information in the prompt instead, describing data from private databases or current, up-to-the-minute information as context to include.

Check availability before you plan around it. OpenAI's model optimisation guide states that it is winding down the fine-tuning platform, that it is no longer accessible to new users, and that existing fine-tuned models remain available until their base models are deprecated. Google's Vertex AI currently lists supervised tuning for models including Gemini 3.5 Flash and 3.1 Flash-Lite. Provider support changes, so confirm it on the day you decide.

What does GDPR change about the choice?

GDPR generally favours RAG when personal data is involved. With RAG, customer data stays in a database you control, so you can delete, correct or restrict a record and the next answer reflects it. Ireland's Data Protection Commission asks organisations using AI whether they can respond to requests to access or delete personal data held within an AI system.

Fine-tuning on personal data is harder to undo. The European Data Protection Board's Opinion 28/2024 states that AI models trained with personal data cannot, in all cases, be considered anonymous; anonymity has to be shown case by case. If you fine-tune, strip or pseudonymise personal data from training examples first. Our guide to GDPR and LLMs covers the wider obligations.

How do you decide in practice?

Work through the options from cheapest to most involved:

  1. Prompt first. Write clear instructions and a few examples. OpenAI notes this may be all you need.
  2. Add context. If the model lacks facts, put the documents in the prompt (small corpus) or build RAG (large or changing corpus).
  3. Measure. Run an evaluation set and read the failures. Wrong facts point to retrieval; wrong format or behaviour points to prompting or tuning.
  4. Tune narrowly. Fine-tune only for a well-defined task with enough good examples, and keep RAG for facts.
  5. Recheck costs and lock-in. A tuned model ties you to one provider's platform and roadmap.

Many production systems end up combining both: RAG supplies current facts, while a tuned or carefully prompted model handles format and tone.

Frequently asked questions

Can you combine RAG and fine-tuning?

Yes. A common pattern is RAG for current facts and a fine-tuned or carefully prompted model for format and tone. Start with RAG and prompting, and add tuning only if an evaluation shows a specific behaviour you cannot fix any other way.

Does fine-tuning stop an LLM making things up?

No. Fine-tuning shapes behaviour but does not give the model a reliable, updatable source of facts. Grounding answers in retrieved passages, and instructing the model to say when the passages do not contain the answer, does more to reduce invented answers.

How much data do I need to fine-tune a model?

It depends on the task and provider, and the providers give method-specific guidance rather than one number. Plan on collecting and checking hundreds of high-quality examples for a narrow task, and test whether a well-written prompt with a few examples already meets your bar.

Is RAG expensive to run?

The search and storage parts are usually cheap; the main cost is the extra input tokens each retrieved passage adds to every question. Retrieve fewer, better passages, and cache the fixed parts of the prompt to keep costs predictable.

Which is better for GDPR, RAG or fine-tuning?

RAG is usually easier, because personal data stays in a store you control and can be deleted or corrected. The EDPB's Opinion 28/2024 says models trained on personal data cannot in all cases be treated as anonymous, which makes fine-tuning on customer data harder to justify and to unwind.

If you are deciding between RAG and fine-tuning for your product, book a free 30-minute call to talk it through.

Cover photo by SHVETS production on Pexels.

Hephaestus Technology

We help founders and businesses validate, build and launch products across software, AI and hardware, from offices in Ho Chi Minh City, Dublin and Delaware. See how we work.

Tell us what
you're building.

A free 30-minute call. You'll get an honest view of the idea, the risks we see and what we'd do first, whether or not we end up working together.