AI Learner #9: RAG — How Models Look Things Up

A model’s knowledge freezes the day its training ends. Ask about something newer — or about your own private documents — and it has two options: admit it doesn’t know, or confidently invent something.

It almost always picks “invent something.” That’s a hallucination, and it’s baked right into how these models work.

A model's baked-in knowledge goes stale and fills gaps by making things up

The RAG Pipeline: Fetch, Augment, Generate

RAG — retrieval-augmented generation — fixes this with a beautifully obvious idea: before answering, go look it up.

The RAG pipeline: retrieve, augment, generate

Step 1: Retrieve

Turn your question into a vector (an embedding), then search a store of your own documents for the chunks most similar to it. That “store” is usually a vector database.

Retrieve: embed the question and search the vector store for the closest chunks

Steps 2 & 3: Augment and Generate

Paste those retrieved chunks into the prompt, and let the model answer using them — often with citations you can actually check. The payoff is huge: answers grounded in real, current, private data, and far fewer confident fabrications. Update the documents and the model’s knowledge updates instantly.

Augment & generate: stuff the docs into the prompt for a grounded, cited answer

The Tradeoffs

It isn’t free. Retrieve the wrong chunks and you’ve just handed the model wrong context — garbage in, very confident garbage out. Retrieval quality is the whole ballgame.

RAG vs. Fine-Tuning

Everyone asks: RAG or fine-tuning? They’re different jobs. RAG gives a model knowledge it can swap anytime. Fine-tuning changes how it behaves. Need fresh facts and citations? RAG. Need a consistent voice or skill? Fine-tuning. Often you want both.

RAG adds swappable knowledge; fine-tuning changes baked-in behavior

Where You’ve Seen It

RAG is why a chatbot can cite your company wiki, today’s headlines, or a PDF you uploaded 30 seconds ago. It’s the backbone of “chat with your documents,” Perplexity-style search, and most serious enterprise AI.

What Comes Next

We can give a model knowledge. But a raw, pretrained model still has no manners — it doesn’t even know it’s supposed to be helpful. Time for the finale.

Coming up: fine-tuning and alignment — how a raw model learns to behave.

Quick Quiz 🧠

1. What are the three steps of RAG?

Answer: Retrieve (find relevant documents via embedding similarity), Augment (insert them into the prompt), and Generate (have the model answer using that context).

2. Why does RAG reduce hallucinations?

Answer: It grounds the answer in retrieved, real documents instead of relying solely on the model’s frozen parametric memory — and it can cite sources.

3. When would you choose fine-tuning over RAG?

Answer: When you need to change the model’s behavior, style, or skills — not just its knowledge. RAG supplies swappable facts; fine-tuning bakes in behavior.

Source: What is Retrieval-Augmented Generation? (IBM), RAG Explained (Pinecone), RAG vs Fine-Tuning (Red Hat)