RAG vs fine-tuning vs prompt engineering: which does your product need? (2026)

Most teams that ask "should we fine-tune?" actually have a knowledge problem, and fine-tuning is the wrong tool for that. Prompt engineering, retrieval augmented generation (RAG) and fine-tuning each change a different part of an AI feature. Pick the wrong one and you pay for training runs that fix nothing, or for a vector database you didn't need.

Below: what each technique changes, a decision table by problem type, a RAG architecture with a small Node.js example, where fine-tuning stands at OpenAI, Anthropic (via Amazon Bedrock) and Google as of September 25, 2026, and the order we recommend trying things in.

The short answer

• Prompt engineering changes the instructions and examples the model sees. Always start here: it's the cheapest thing to change and the fastest to test.
• RAG changes the *information* the model sees at answer time, by searching your documents and putting the relevant passages in the prompt. Use it when answers depend on private, fresh or large amounts of content, or when you need citations.
• Fine-tuning changes the model's *behavior*: format, tone, classification labels, how terse it is. Google's Vertex AI docs describe supervised fine-tuning as a way to adapt "model behavior with a labeled dataset." Use it when well-built prompts still aren't consistent enough and you have clean examples of the output you want.

Most production systems combine them: a good prompt, plus retrieval, and occasionally a tuned model underneath.

What each technique changes

Decision table by problem type

How a RAG system is put together

A RAG pipeline has two halves. Google's RAG Engine docs describe the same stages most teams build: ingestion, transformation (chunking), embedding, indexing, retrieval and generation.

Indexing (offline):

1. Ingest and clean. Pull documents from your CMS, help center, PDFs or database and strip boilerplate. Claude can't cite scanned PDFs that have no extractable text, so run OCR on those first.
2. Chunk. Split documents into passages. OpenAI's file search defaults to 800-token chunks with 400 tokens of overlap, which is a sensible starting point to test against. Keep headings with their text so a chunk still makes sense alone.
3. Add context to chunks. Anthropic's contextual retrieval approach prepends a short explanation of where each chunk sits in its document before embedding it. In Anthropic's own tests, contextual embeddings plus contextual BM25 cut failed retrievals by 49%, and 67% with reranking added. Test it on your data rather than assuming the same gain.
4. Embed and store. Turn each chunk into a vector with an embedding model and store it with metadata (source URL, document title, customer or tenant ID, last updated). Postgres with pgvector is a practical default if you already run Postgres.

Answering (per request):

1. Embed the question with the same embedding model you indexed with.
2. Hybrid search. Combine vector similarity with keyword search (BM25 or Postgres full-text). Vectors find paraphrases; keywords find SKUs, error codes and product names. The pgvector docs mention merging the two with Reciprocal Rank Fusion. OpenAI's file search exposes weights for embedding versus keyword matches.
3. Rerank. Pass the top results through a reranker model and keep the best few. Anthropic notes that reranking adds latency at runtime, so measure it.
4. Generate with citations. Put the passages in the prompt with IDs and ask the model to answer only from them. Claude's citations and search result blocks, and OpenAI's file_citation annotations, do this in the API rather than by prompt alone.

Managed options exist if you don't want to run your own index: OpenAI's file search (vector stores on the Responses API), Google's RAG Engine and Claude's document citations. Check their regions and data controls before you commit. For example, Google's RAG Engine page lists Frankfurt as generally available and London in preview, and says data residency controls aren't supported.

A minimal retrieval example in Node.js

This uses the official openai npm package, keeps vectors in memory and uses cosine similarity. It's enough to test retrieval quality on a few hundred chunks before you choose a database. Save it as an .mjs file (it uses top-level await). In production, store vectors in pgvector or another vector store and add keyword search.

import OpenAI from "openai";
const openai = new OpenAI(); // reads OPENAI_API_KEY
const EMBED_MODEL = "text-embedding-3-small";
// Sample chunks; in practice these come from your chunking step
const chunks = [
  { id: "returns-1", text: "Unworn items can be returned within the return window shown on the order page." },
  { id: "shipping-2", text: "International orders are shipped with tracking. Duties are shown at checkout." },
  { id: "warranty-1", text: "Electronics are covered by the manufacturer warranty listed on the product page." },
];
async function embed(texts) {
  const res = await openai.embeddings.create({ model: EMBED_MODEL, input: texts });
  return res.data.map((d) => d.embedding);
}
function cosine(a, b) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
// 1. Index once
const vectors = await embed(chunks.map((c) => c.text));
const index = chunks.map((c, i) => ({ ...c, vector: vectors[i] }));
// 2. Retrieve the top k chunks for a question
async function retrieve(question, k = 2) {
  const [q] = await embed([question]);
  return index
    .map((c) => ({ id: c.id, text: c.text, score: cosine(q, c.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}
// 3. Answer only from the retrieved sources, with citations
const question = "Can I send back shoes I wore once?";
const hits = await retrieve(question);
const sources = hits.map((h) => `[${h.id}] ${h.text}`).join("\n");
const response = await openai.responses.create({
  model: process.env.OPENAI_MODEL, // choose a current model from OpenAI's models page
  instructions: "Answer only from the sources. Cite source ids in square brackets. If the sources don't answer the question, say so.",
  input: `Sources:\n${sources}\nQuestion: ${question}`,
});
console.log(response.output_text);
console.log(hits.map((h) => `${h.id}: ${h.score.toFixed(3)}`));

OpenAI's embeddings are normalized to length 1, so a plain dot product gives the same ranking. The full cosine function keeps the code correct if you switch to a provider whose vectors aren't normalized. Log the scores: when the right chunk isn't in the top results, no prompt will fix the answer.

Fine-tuning in 2026: prerequisites and what's actually available

Before you plan a tuning project, check that your vendor still offers it for the model you want. As of September 25, 2026:

Open-weight models (Bedrock's table also lists several Llama models) are the other route: full control, but you take on hosting and upgrades.

Prerequisites before any tuning run:

• A task you can define. Google describes supervised tuning as a good fit when "you have a well-defined task with available labeled data."
• Enough clean examples. OpenAI's minimum is 10 examples and it suggests "starting with 50 well-crafted demonstrations." Its best-practice guide says a smaller amount of high-quality data generally beats a larger amount of low-quality data. Check that examples written by different people are consistent.
• A held-out test set. Split your data so the test portion is only used for evals, never for training.
• A prompt baseline. If you haven't measured your best prompt on the same test set, you can't tell whether tuning helped.
• A re-tuning plan. A tuned model is tied to its base model. When the base is deprecated, you retrain.

Google also documents two practical details: set thinking to the minimum on tuned Gemini models, and skip controlled generation (structured output mode) at inference, because tuning without it and serving with it can lower quality.

How to evaluate, whichever technique you pick

1. Write success criteria first. Anthropic's guidance is that criteria should be specific, measurable, achievable and relevant. "Answers returns questions correctly and cites the policy" is testable. "Works well" isn't.
2. Build the eval set from real traffic. Take questions from support tickets, search logs or chat transcripts, with the expected answer and source document. Include questions your content can't answer, to test that the system says so.
3. Score retrieval separately from generation. For RAG, check whether the right chunk was retrieved (recall) and how much noise came with it (precision) before judging the final answer. OpenAI's eval guide uses context recall and precision for RAG systems.
4. Automate grading, then calibrate it. Use exact match or code checks where you can and an LLM grader where you can't. OpenAI's guide warns against "vibe-based evals" and recommends calibrating automated scores against human feedback.
5. Re-run on every change: prompt edits, chunk sizes, embedding model, reranker and model version.

Our AI MVP guide includes a minimal eval harness in Node.js you can adapt for this.

Cost and latency: what moves the bill

We don't quote prices here because they change often; check each vendor's pricing page on the day you budget. What drives cost and latency:

• Prompt engineering: longer prompts mean more input tokens on every call. Prompt caching on Claude and OpenAI cuts the cost and latency of a repeated prefix, which makes "put the whole document in the prompt" cheaper than it looks.
• RAG: embedding calls at index and query time, vector storage (OpenAI bills vector store storage above a free allowance), the extra retrieval step, and reranking. OpenAI's max_num_results setting exists to reduce "both token usage and latency."
• Fine-tuning: training runs, data preparation time, and retraining whenever the base model changes. The payoff is shorter prompts on high-volume, narrow tasks.
• Model choice often beats all three. Try a smaller or faster model against your eval set before you build anything.

Common mistakes

• Fine-tuning to teach facts. Tuning shapes behavior. For facts that change or need a source, use retrieval.
• No eval set. Without one, you can't tell whether RAG, a new prompt or a tuned model is actually better.
• Vector search only. Pure embeddings often miss exact identifiers like SKUs, part numbers and error codes. Add keyword search.
• Chunks that lose their context. "It must be returned within the window" means nothing without its document. Keep headings, or add chunk context.
• Ignoring permissions. Filter retrieval by user, customer or tenant before the model sees anything. The model can't enforce access rules on text it has already been given. Our guide to integrating AI into a SaaS product shows where the tenant filter sits.
• Combining Claude citations with structured outputs. Anthropic documents these as incompatible: the API returns a 400 error.
• Planning around a tuning platform that's closing. Check the vendor's deprecations page before you collect training data.

Start here: a practical path

1. Collect a few dozen real questions with expected answers and sources. This is your eval set.
2. Write a solid prompt on a current model and measure it. Many products stop here.
3. If answers lack knowledge: if the content is small and stable, put it in the prompt with caching. If it's large, changing or permissioned, add RAG, starting with a managed option or the in-memory example above.
4. If retrieval misses: add keyword search, chunk context and a reranker, one at a time, measuring each.
5. If format or behavior still fails after good prompts, and you have at least 50 clean examples plus a vendor path that will still exist next year, try fine-tuning and compare it against the prompt baseline on the same test set.
6. Log production traffic and add every failure to the eval set.

For more on where this fits in a wider AI roadmap, see our AI development services. If your content lives in a Shopify store, our AI for Shopify page covers catalog and support assistants, and our guide to AI for Shopify stores compares built-in, app and custom options.

Want help choosing and building it?

We build prompt pipelines, RAG systems on pgvector or managed vector stores, and eval harnesses that tell you whether a change helped. Tell us what your product needs to answer, and we'll tell you honestly whether you need RAG, fine-tuning or just a better prompt. Start on our AI integration page.

Sources

• OpenAI: Model optimization (fine-tuning overview and platform wind-down notice)
• OpenAI: Supervised fine-tuning
• OpenAI: Deprecations
• OpenAI: Retrieval (vector stores, chunking, hybrid search)
• OpenAI: Vector embeddings
• OpenAI: Evaluation best practices
• Anthropic: Introducing Contextual Retrieval
• Anthropic: Citations
• Anthropic: Define success criteria and build evaluations
• Google Cloud: About supervised fine-tuning for Gemini models
• Google Cloud: RAG Engine overview
• AWS: Customize a model with fine-tuning in Amazon Bedrock