Retrieval-Augmented Generation

Definition

How can a chatbot answer questions on your company's private documents without being trained on them?

Retrieval-Augmented Generation (RAG) is a technique in which a system first searches a knowledge source for passages relevant to a question, then gives them to an LLM along with the question, so the answer is based on that retrieved material.

In simple words

It's an open-book exam. The LLM is a smart student with a fuzzy memory. Without the book, the student may guess. With RAG, a helper finds the right pages first and says, "Answer from these." The answers get more accurate, and can say where they came from.

Why it matters

LLMs don't know your private data or recent events, and retraining them is expensive. RAG adds both cheaply, and updating it is as simple as updating the documents. It's one of the most common ways that companies put LLMs to work today.

How it works

Step 1: index, once ahead of time

Split documents into small chunks.
Turn each chunk into an embedding vector.
Store the vectors in a vector database.

Step 2: answer, on every question

Turn the question into a vector too.
Find the chunks with the closest vectors.
Put those chunks into the prompt with the question.
The LLM writes an answer, ideally with citations.

The maths

Finding the closest chunk: cosine similarity

score (q, d) = \frac{q \cdot d}{∥ q ∥ ∥ d ∥}

$q$: The question's embedding
$d$: A document chunk's embedding

In code

The retrieval half of RAG, using TF-IDF for simplicity. Real systems use neural embeddings and a vector database, but the idea is identical.

tiny_rag.py

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

chunks = [
    "Refunds are allowed within 30 days of purchase with a receipt.",
    "Our office is open Monday to Friday, 9 am to 6 pm.",
    "Shipping to Mumbai takes 3 to 5 working days.",
]
question = "Are refunds allowed after purchase?"

vectorizer = TfidfVectorizer(stop_words="english")
vectors = vectorizer.fit_transform(chunks + [question])
scores = cosine_similarity(vectors[-1], vectors[:-1])[0]

best = scores.argmax()
print("Scores:", scores.round(2))
print("Prompt for the LLM:")
print(f"Answer only from this text: {chunks[best]}")
print(f"Question: {question}")

Output

Scores: [0.64 0.   0.  ]
Prompt for the LLM:
Answer only from this text: Refunds are allowed within 30 days of purchase with a receipt.
Question: Are refunds allowed after purchase?

Common mistakes

Making chunks too big or too small. Too big dilutes the match, and too small loses the context.
Skipping evaluation. Test with real questions and check that it finds the right passages.
Ignoring access control. Users must only get answers from documents they're allowed to see.

Check yourself

1. What does the R in RAG do?

2. A RAG bot gives wrong answers. What should you check first?

Summary

RAG first retrieves relevant text, then asks the LLM to answer from it.
It adds private and fresh knowledge without retraining.
Retrieval quality decides answer quality.

Finished reading?

Sources: Lewis et al., RAG (2020)