Evals first: what I learned teaching my agent RAG
Thomas Maximini · September 5, 2026
13 min read
Why this project exists
LLMs can answer most questions directly from their training data, but some knowledge — internal company documents, policies, work instructions — never makes it in there. RAG (Retrieval Augmented Generation) closes that gap by letting the model look up facts in an outside knowledge base before it answers.
I wanted to learn more about RAG, and the perfect use case was sitting right in front of me. The support agent I built recently had a decent sized knowledge base — in RAG terms, the corpus: the body of text the system retrieves its answers from. In order to get that into the agent's context, we already compressed it significantly, but I wanted to see if I could improve the agent's performance by implementing RAG.
I didn't really know where to start learning it, so I asked Claude to slowly walk me through the entire process step by step as a learning exercise: chunking, embeddings, retrieval, evals — each piece explained before it got built, and each decision written down in a notes file so I could re-explain it later.
My main takeaways? RAG is not a silver bullet: whether it's the right tool for content retrieval at all depends on the use case, the context, the size of your corpus. And where it is the right tool, its quality hangs on unglamorous details — chunking strategy above all — that you can only get right by measuring.
Luckily, the previous project had already left me with an eval pipeline and the habit of measuring every change. That's what made this a safe playground. I could experiment with retrieval approaches and chunking strategies knowing the impact would show up in numbers — and that regressions wouldn't slip through unnoticed.
Eval before any change
So before making any significant changes to the retrieval logic, I built a golden set: twenty questions with span-level ground truth — for each question, the heading and line ranges in the corpus that answer it, pinned to a content hash so the annotations can't silently drift. A retrieved chunk counts as a hit if it overlaps any gold span, which makes scoring chunking-independent.
One rule I'm glad I followed: questions phrased in customer voice, paraphrased away from the corpus vocabulary — otherwise the corpus writes its own exam.
Annotating meant going through the corpus section by section, and that alone produced findings. The corpus contradicts itself: one section offers a service another says was discontinued, and there are two different refund windows. I kept the contradictions in the golden set as flagged cases: every eval run now shows which of the two contradicting sections retrieval picks, and whether a content fix actually resolved it.
Then, before changing anything, I scored the current architecture — a hand-maintained "compact" knowledge base stuffed into every prompt. The initial human rating over the twenty questions: 14 pass, 4 borderline, 2 fail. 70%. I sorted the failures into classes and wrote down predictions before making any change. The coverage gaps — details that existed in the full corpus but had been lost in the compression — should be fixed by retrieval, because retrieval reads the full corpus. The agent's habit of demanding identity verification for harmless policy questions on the other hand should not be fixed by it, because that's a prompt problem, not a knowledge problem.
Chunking strategies
I measured two chunking strategies instead of assuming one: fixed-size with overlap (deliberately dumb, 109 chunks) against markdown-heading sections (171 chunks). Each run costs about $0.0005.
The metric is called recall@k: for what share of the questions does a correct chunk show up in the top k search results? recall@1 means the right chunk ranked first; recall@5 means it was somewhere in the top five. The k matters because it tells us how many chunks we end up putting into the prompt.
| fixed-size | by heading | |
|---|---|---|
| recall@1 | 45% | 75% |
| recall@3 | 80% | 85% |
| recall@5 | 85% | 85% |
| recall@10 | 95% | 100% |
The interesting row is recall@5: identical. If I'd only looked at recall@5 — the number most tutorials report — I'd have concluded chunking doesn't matter here. The entire win lives at the top of the ranking: one topic per chunk gives a crisper vector (the numerical fingerprint of the chunk's meaning that the search compares), so the right chunk lands on rank one far more often.
Top-rank quality also has a direct practical payoff: it lets k shrink. Heading chunks at k=3 deliver what fixed-size needs k=5 for, which means smaller prompts on every request. Heading chunks are cheaper to index too — no overlap duplication.
For orientation, here is a list of common chunking strategies — the first three are the ones this project measured:
- Fixed-size with overlap — cut every ~N characters, overlap so ideas straddling a cut survive. Structure-blind, the tutorial default. Our baseline: recall@1 45%.
- Structure-aware (by heading) — one markdown section, one chunk, one topic. Crisper vectors, and the right chunk ranks first far more often: 45% → 75%.
- …plus breadcrumbs and thin-merge — each chunk embeds its document title and heading path, and tiny sections merge into their neighbor. What that means and why it helps comes up in the second corpus below.
- Q&A-pair chunking — one question-answer pair per vector; question-to-question matching often beats question-to-paragraph. Right shape for FAQ-style content.
- Semantic chunking — split where the embedding similarity between adjacent passages drops. The tool for text with no usable structure; our documents had structure, so we used theirs.
- Contextual retrieval — an LLM writes a one-line context per chunk before embedding. Breadcrumbs are the cheap, deterministic version of the same idea.
Observed regressions
With the baseline frozen, I wired retrieval into the draft path — top-3 chunks, behind a feature flag, falling back to the old stuffing on any failure so the worst case equals the baseline. Same twenty questions, same blind test.
The predictions held. Both coverage-gap questions flipped to pass — that's what the change was for. The verification-reflex failures stayed, exactly as predicted.
Two questions got worse — one pass→borderline, one pass→fail. Both looked like retrieval regressions. But on further inspection it showed the right chunks had been retrieved (ranks 1–3) and the missing facts were verbatim in the injected prompt. The model simply failed to render them in that one generation.
Why does a model drop a fact that is sitting right there in its prompt? Temperature. The agent generates drafts at temperature 0.5 — a deliberate choice, so replies don't all sound like the same template. But at that setting every generation is a slightly different roll of the dice, and once in a while a detail just doesn't make it into the sampled text. Regenerating the same question brought the missing facts back. The knob to fix it would be lowering the temperature for policy answers, trading some natural variation for consistency.
With retrieval, "why did the answer miss X" splits into three separately checkable questions: Was the right section retrieved? If not, tune retrieval. Was the fact literally in the prompt? If not, fix the corpus. Did the model use it? If not, it's a prompt rule — or sampling noise. Stuffing never allowed that cut: everything was always "in the prompt", so every failure looked the same.
Problems with the grader
In parallel I built an eval for the tool-using path: given a known order state, does the model describe it truthfully? Record real outputs, have a human rate them, then automate a grader against those verdicts.
The very first calibration run disagreed with the human — and the human was right. The test case was an order that hadn't shipped yet, so the draft was not allowed to claim a tracking number exists. My grader enforced that with a blunt rule: fail any draft containing the words "tracking number". The draft it failed had made no such claim — it correctly promised one for later: "as soon as it ships, you'll receive the tracking number". Mentioning a fact and asserting it are different things, and a substring check can't tell them apart. The grader was wrong, not the bot; I narrowed the rule to concrete claims and actual number patterns.
A few days earlier I'd encountered the opposite case. A test order had status completed on a pickup order — meaning the customer already collected it. The draft told them it was "ready for pickup" instead of "already picked up". I rated it pass: friendly wording, clean structure. I'd judged the draft inside its own framing instead of checking its claims against the data — anchoring bias, caught only on a later cross-check.
Both incidents are the same fork. When grader and human disagree, either the system is wrong or the grader is; the actions are opposite; and only a human looking at the case can tell which. That's why a grader has to be checked against human ratings before its numbers mean anything.
The second corpus
With those learnings I realized the system had a second corpus, and this one might be a great fit for RAG: internal work instructions that the support team writes and uploads themselves. Until recently, the agent fetched all of them and stuffed the entire thing into every internal prompt — tens of kilobytes per message, most of it irrelevant to whatever was asked. Same shape as the problem above, but structurally worse: this corpus grows with every upload, so the stuffing cost grew with every document the team added.
A lot of the work from the first corpus transferred directly. Embeddings are cached, so we don't pay to recompute them on every question — and the cache is stored under a fingerprint of the corpus content itself, the same trick as content hashes in JS bundle filenames. When someone uploads or edits a document, the fingerprint changes, the old cache no longer matches, and the index rebuilds itself on the next question. Nobody has to remember to update anything, and serving outdated vectors is impossible by design. The heading-based chunking worked as-is, because the work instructions are structured markdown too. And the discipline carried over: I wrote golden questions and rated a baseline before touching anything.
Some things were genuinely new, though. This corpus is a collection of separate documents, and it changes — people who are not engineers upload and edit them whenever they like. So every chunk now remembers which document it came from, and the document's title gets prepended to the chunk text before embedding. That sounds like a small detail, but it mattered twice: two documents in the collection literally share the same title, and a section deep inside a long manual usually never mentions the tool's name again.
The build itself took a day, because the method was already in place. I created a 19-question golden set, including two questions designed to test exactly that title collision. I measured retrieval quality, then recorded a baseline of answers through the real chat path.
The first measurement came in at 68% recall@1 — mediocre. Looking at what outranked the correct chunks showed two patterns, and each got its own fix. Breadcrumbs: every chunk embeds its heading path, so a section deep inside a manual knows which manual it belongs to. Thin-merge: sections under ~150 characters get folded into the next chunk, because a bare parent heading with two intro lines otherwise wins searches it can't answer. The results were eye-opening:
| by heading | + breadcrumbs | + thin-merge | + both | |
|---|---|---|---|---|
| recall@1 | 68% | 84% | 79% | 89% |
| recall@3 | 84% | 95% | 89% | 95% |
| recall@5 | 95% | 95% | 95% | 95% |
| recall@10 | 100% | 100% | 100% | 100% |
The same pattern as the first corpus, again: recall@5 didn't move at all, the entire improvement happened in the top ranks.
Then I made the switch, behind a flag, with the old stuffing as the fallback. Result: ~20,600 knowledge tokens per request down to ~680 — minus 97% — with zero errors. That switch is now live in production: internal questions get the five most relevant excerpts, each labeled with its source document, instead of the whole pile. Five rather than three, because I picked k while the recall@3/@5 gap was still wide — the chunking fixes closed it, so trying three is the next cheap experiment.
Key takeaways
For content that users upload themselves — documents, PDFs, markdown files, whole knowledge bases — retrieval belongs in the design from day one, with re-embedding triggered automatically by every upload and every edit. Nobody knows in advance how big user-uploaded content gets, so stuffing is not a plan there.
Once you understand the moving parts, a framework is a reasonable default. LangChain and LlamaIndex are the two standard choices in Python; in the TypeScript world the Vercel AI SDK covers similar ground. Hand-building everything was the right call for a learning project — and the lasting payoff is that these frameworks stop being magic. Now I know what they do under the hood, and which defaults deserve a second look.
The same logic applies to the vector database. numpy in RAM was the right size for a few hundred chunks, but any bigger project — and anything built on user uploads — is better off starting with a real one: pgvector when Postgres is already in the stack, a managed store like Pinecone when it isn't. Migrating an index later is more work than starting with one.
And measurement belongs at the beginning, not the end: a golden set of questions with marked source passages, recall@k for the retrieval side, a blind-rated answer baseline before switching anything on, token cost per request — and a re-run after every change, so regressions show up in numbers instead of in production.
For this system, the next steps are already known. The regex graders will become an LLM judge, because every new correct phrasing breaks another regex. Retrieval will become hybrid, combining vector search with proper German full-text search, because some questions hang on a single rare keyword that embeddings blur. And the golden set will grow with questions from real tickets, instead of questions derived from the corpus itself.
The bottom line
Where RAG shines: When the corpus is large, growing, or out of your control. The work-instruction corpus is the archetype. The team uploads documents whenever they like, so nobody knows in advance how big the collection will get, and with stuffing the prompt cost grows with every upload. You can't budget "just put everything in the context" when you don't control how big "everything" gets. Retrieval turned ~20,600 knowledge tokens per request into ~680 there, a 97% reduction.
Where it's overkill: a small, stable FAQ. If the whole thing fits comfortably in the prompt and rarely changes, stuffing is simpler, and there is no retrieval step that could miss. Our hand-maintained compact KB was exactly that, and it scored 70% with zero moving parts. We still switched — but we switched because the measurements said quality wouldn't drop while prompt cost fell 83%, not because RAG is what one does.