Chapter 7: RAG — Giving the Agent Knowledge

Rosa's bookshop agent is doing well. Customers ask about books, it searches the inventory, everyone's happy. Then Rosa uploads the shop's full policy document — 30 pages of returns, shipping, membership rules, and special-order procedures — and asks Tomás: "Can the agent answer questions from this?" Tomás tries the obvious thing. He pastes the whole document into the system prompt. It's 30 pages. The context window groans. The agent answers the first question fine, but every call now sends 30 pages of policy text whether the question is about returns or about book recommendations. It's slow, it's expensive, and it's unsustainable. "There's a better way," Priya says. "Don't give it the whole document every time. Give it a search tool that finds the right paragraph and brings just that back. It's like a librarian — you don't read the whole library to answer one question. You find the right book."

Why This Matters

By the end of this chapter, you'll understand RAG (Retrieval-Augmented Generation): the technique that lets an agent answer questions from your own documents without stuffing them all into the prompt. You'll know what embeddings are, how a vector store works, and how to build a retrieval tool that finds the right passage by meaning, not by keyword. This is how agents get knowledge they weren't trained on.

The Problem: The Model Doesn't Know Your Stuff

The LLM was trained on the public internet up to some date. It knows general knowledge — Lisbon's population, how photosynthesis works, what a Python function is. But it doesn't know your stuff: your company's policies, your codebase, your customer data, your product catalogue. And even for things it once knew, it can be out of date or just wrong.

The LLM is smart about the world, but blind to your stuff. RAG fixes that.

You could paste your documents into the prompt — but that doesn't scale. A 30-page policy doc is maybe 15,000 tokens. A whole knowledge base could be millions. The context window can't hold it all, and even if it could, you'd pay for every token on every call. You need a way to bring in just the relevant part.

RAG: Retrieve, Then Generate

RAG is two steps:

1. Retrieve. When the user asks a question, search your documents for the most relevant passages. Not the whole document — just the paragraphs that are likely to contain the answer.

2. Generate. Give those passages to the LLM along with the question. The LLM reads the passages and answers — grounded in your actual data, not making things up.

Don't stuff the whole doc in. Find the right passage, then let the LLM answer from it.

Note RAG is just another tool in the agent loop. The agent calls a search_documents tool, gets back relevant passages, and uses them to answer. Everything you learned about tools in Chapter 4 applies. The only new thing is how the search works — by meaning, not by keyword.

Embeddings: Meaning as Numbers

Here's the clever bit. How do you search for "relevant passages" when the user's question uses different words than the document? The user asks "what's your return policy?" but the document says "items may be refunded within 30 days." Keyword search fails — no shared words. You need to search by meaning.

Embeddings solve this. An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meanings get similar numbers. "Return policy" and "refund within 30 days" might have close embeddings, even though the words are different. "Return policy" and "pizza recipe" would have very different embeddings.

Embeddings turn meaning into numbers. Similar meanings cluster together.

You create embeddings using an embedding model — a different model than the chat LLM. OpenAI's text-embedding-3-small is a common one. You send it text, it gives you back a vector (a list of ~1,500 numbers). You don't need to understand the numbers — you just need to know that similar meanings produce similar vectors.

from openai import OpenAI
client = OpenAI()

# Create an embedding for a piece of text
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Items may be refunded within 30 days of purchase."
)
embedding = response.data[0].embedding
print(len(embedding))  # 1536 — a list of 1536 numbers

// An embedding is just a list of floats. You don't read the numbers — you compare them.

The Vector Store: Finding Relevant Passages

Here's the full RAG pipeline. You do this once to set up, then the agent uses it at runtime:

Setup (once): Chunk and embed your documents

# Step 1: Split your document into chunks (paragraphs or sections)
chunks = [
    "Returns: Items may be refunded within 30 days of purchase with receipt.",
    "Shipping: Orders ship within 2 business days via standard post.",
    "Membership: Members get 10% off all purchases and early access to sales.",
    "Special orders: Out-of-stock books can be ordered on request, 2-3 week delivery.",
]

# Step 2: Create an embedding for each chunk
embeddings = []
for chunk in chunks:
    resp = client.embeddings.create(model="text-embedding-3-small", input=chunk)
    embeddings.append(resp.data[0].embedding)

# Step 3: Store the chunks and their embeddings together
# (In production, use a vector database like Chroma, Pinecone, or pgvector)
vector_store = list(zip(chunks, embeddings))

// Split into chunks, embed each one, store them. This is the "indexing" phase — done once.

Runtime (per question): Embed the question, find matching chunks

import numpy as np

def search_documents(query: str, top_k: int = 2) -> str:
    """Search the document store for passages relevant to the query."""
# Embed the question
    resp = client.embeddings.create(model="text-embedding-3-small", input=query)
    query_embedding = resp.data[0].embedding

    # Compare the question's embedding to each chunk's embedding
    scores = []
    for chunk, emb in vector_store:
        # Cosine similarity: how close are the two vectors?
        score = np.dot(query_embedding, emb) / (
            np.linalg.norm(query_embedding) * np.linalg.norm(emb)
        )
        scores.append((chunk, score))

    # Return the top_k most similar chunks
    scores.sort(key=lambda x: x[1], reverse=True)
    results = [chunk for chunk, _ in scores[:top_k]]
    return "\n\n".join(results)

// Embed the question, compare it to all stored chunks, return the closest matches. That's retrieval.

Geek Bits The comparison uses cosine similarity — a measure of how aligned two vectors are, regardless of length. It's the dot product divided by the product of their magnitudes. A score of 1 means identical direction, 0 means unrelated, -1 means opposite. In practice, embeddings of related text score above 0.7; unrelated text scores below 0.4. You don't need to memorise the math — vector databases do this for you. But it helps to know what it's doing: measuring meaning-distance.

Plugging RAG Into the Agent Loop

Now the magic moment. search_documents is just a tool. We add it to the agent's tool list, describe it clearly, and the agent calls it when a question needs document knowledge:

# Add the RAG tool to the agent's tool list
{
    "type": "function",
    "function": {
        "name": "search_documents",
        "description": (
            "Search the shop's policy documents for relevant information. "
"Use when a customer asks about returns, shipping, membership, "
"or special orders. Returns the most relevant passages."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "The question or topic to search for."
                }
            },
            "required": ["query"]
        }
    }
}

// Same pattern as every other tool. The description tells the LLM when to use it.

Now watch the agent handle a policy question:

User: "Can I return a book I bought last month?"
THINK "This is a policy question about returns. I should search the documents."
ACT    search_documents(query="return policy time limit")
OBSERVE "Returns: Items may be refunded within 30 days of purchase with receipt."
THINK "The policy says 30 days with receipt. The user said 'last month'
which is about 30 days. I should mention the receipt requirement."
Agent: "Yes, you can return items within 30 days of purchase, as long as you have the receipt. If it's been about a month, you should be within the window — bring it in with your receipt!"

// The agent retrieved the relevant passage, then answered grounded in it. No hallucination.

Note The agent didn't make up a return policy. It retrieved the actual passage and answered from it. This is the whole point of RAG: the agent's answer is grounded in your real documents. If the policy changes, you update the document store — no retraining, no prompt editing. The agent just retrieves the new version.

Why RAG Beats Stuffing the Prompt

RAG brings in only what's relevant. Stuffing brings everything, every time.

There Are No Dumb Questions

Q: How big should each chunk be?

A: Typically 200–500 words, or a few paragraphs. Too small and you lose context; too big and you retrieve irrelevant text alongside the good stuff. There's no perfect number — experiment. Many tools split by paragraphs or sentences with some overlap between chunks, so a passage that spans a boundary isn't lost.

Q: Do I need a vector database, or can I just use a list like in the example?

A: For a few dozen chunks, a list with numpy works fine — that's what we did. For thousands or millions, you need a real vector database (Chroma, Pinecone, pgvector, Weaviate) that can search efficiently. The concept is the same; the scale changes. Start simple, add a vector DB when you outgrow the list.

Q: What if the retrieval returns the wrong passage?

A: This is the main failure mode of RAG. The agent answers from a passage that isn't actually relevant. Fixes include: better chunking, retrieving more passages (top 5 instead of top 2), and telling the LLM "if the retrieved passage doesn't answer the question, say you don't know." We'll cover this more in Chapter 10 (Guardrails) — RAG quality is a tuning problem.

Where People Come Unstuck

Mistake #1: Chunks that are too big or too small

Too small (a single sentence) and the chunk loses context — "30 days" without "of purchase with receipt." Too big (a whole chapter) and you retrieve a lot of irrelevant text. Aim for a paragraph or two — enough to be self-contained, not so much that it dilutes the signal.

Mistake #2: Not telling the LLM to use the retrieved text

The agent retrieves a passage but then answers from its general knowledge instead. Fix it in the system prompt: "When you use search_documents, base your answer on the retrieved passages. If they don't contain the answer, say you don't know." Grounding is a prompt instruction, not an automatic behaviour.

Mistake #3: Treating RAG as a search engine

RAG isn't just "find a passage and show it." The LLM reads the passage and answers the question in its own words, synthesising if needed. The retrieval gives the facts; the LLM provides the understanding. Don't just dump raw chunks at the user — let the agent turn them into a real answer.

Brain Power

You're building an agent for a law firm. They have 10,000 contracts stored as PDFs. Lawyers want to ask: "Does this contract have a non-compete clause?" and "What's the termination notice period?"

Think through the RAG pipeline: How would you chunk contracts? What embedding model would you use? What would the tool description say? What could go wrong — and how would you detect it? Would you retrieve per-contract or across all contracts?

The point isn't to have perfect answers — it's to start thinking in the RAG pipeline: chunk → embed → store → retrieve → generate.

Chapter Summary

The LLM doesn't know your data. It was trained on the public internet, not your documents. RAG fixes this.
RAG (Retrieval-Augmented Generation) is two steps: retrieve relevant passages from your documents, then generate an answer from those passages.
Embeddings turn text meaning into numbers. Similar meanings produce similar vectors, so you can find relevant passages even when the words differ.
The pipeline: chunk your documents → embed each chunk → store them in a vector store → at runtime, embed the question and find the closest chunks.
RAG is just another tool in the agent loop. search_documents retrieves passages; the LLM reads them and answers.
RAG beats stuffing the prompt because it brings in only relevant text — cheaper, scalable, and updateable without retraining.
Grounding is a prompt instruction: tell the LLM to base answers on retrieved passages and to say "I don't know" when they don't contain the answer.

Chapter Challenge

The Policy Agent. Take Rosa's bookshop agent and add RAG. Create 4–5 policy chunks (returns, shipping, membership, special orders, events). Embed them, store them in a list, and add the search_documents tool to the agent.

Test it with: "Can I return a book without a receipt?" and "Do members get a discount?" — questions that use different words than the policy text. Watch the agent retrieve the right passage by meaning, not by keyword, and answer grounded in it.

Then try a question the policy doesn't cover — like "Do you sell gift cards?" — and see if the agent says "I don't know" (if you told it to) or hallucinates. That's the RAG quality problem, and you've just taken your first step toward solving it.