Productionising RAG, honestly.

An eval discipline that killed BM25+RRF and parked a Cohere reranker. What survived, what didn't, and the unglamorous engineering between a demo and a product. That is the honest one-line summary of taking retrieval-augmented generation to production, and I am writing it down because almost everything published about RAG is written before this part happens, by people who have not yet had to delete their favorite component.

The system in question is a family-memory archive. Users upload the raw material of a family's past and then converse with it, in three languages, because real families are not monolingual and ours certainly is not. A question can arrive in Belarusian about a memory recorded in Polish and expect an answer in English. That detail is not a footnote. It shaped every decision that follows, and it is the kind of detail a generic RAG tutorial never has, because generic tutorials retrieve English documents for English questions and declare victory.

The demo phase went the way demo phases go. Chunk the documents, embed the chunks, cosine similarity, top-k into the context, and the first twenty questions you try produce answers that feel like magic. Everyone who saw it was impressed, including me, and that is exactly the trap. A RAG demo is a machine for making you trust it before it has earned it, because you instinctively ask it the questions it is good at, and it answers those beautifully, and the failure cases sit quietly outside your imagination until a real user walks into them.

So before touching the architecture again, we built the thing that makes the rest of this essay possible: the eval set. A fixed collection of real questions with known right answers, drawn from real archive material, across all three languages, deliberately salted with the ugly cases. Questions whose answer lives in one sentence of one document. Questions that need two documents stitched together. Questions asked in a different language than the source. Questions with no answer in the archive at all, where the only correct behavior is to say so. Every retrieval change, from that day on, ran the whole set, and the rule was simple: numbers move or the change dies. Not "it feels better on the queries I tried this morning." Numbers, on the fixed set, or it does not ship. When I show people the system I ask them not to tell me it is great, because flattery teaches me nothing. The eval set is that request turned into infrastructure. It is a colleague who is incapable of politeness.

Two words on how the scoring works, because "run the evals" hides the actual labor. Retrieval and generation get judged separately, always. First question: did the right chunks come back at all, because if retrieval missed, nothing downstream can be trusted and no amount of prompt polish will fix it. Second question: given the right chunks, did the answer use them faithfully, in the right language, without decorating the truth. Grading the second part involves a model, because grading thousands of free-text answers by hand does not scale, but the grader itself had to be checked against human judgment on a sample before we trusted it, which is its own small eval project. None of this is glamorous. All of it is the actual work, and I have never seen it in a pitch deck.

Now the graveyard, because the graveyard is the point.

The first serious casualty was hybrid retrieval: BM25 keyword search fused with vector search through reciprocal rank fusion. Every RAG best-practices post recommends it, and the reasoning is sound. Vectors catch meaning, keywords catch exact names and rare terms, fuse the two rankings and you should get the best of both. We built it properly and ran the set. The evals killed it: on our corpus, the fusion was consistently worse than the vector side alone. The autopsy made sense of it. BM25 lives on vocabulary overlap, and a trilingual archive where the question and the answer routinely do not share a language starves it of exactly that. Morphology-heavy languages fragmented the term statistics further. So the keyword ranking was mostly noise for us, and RRF, faithfully democratic, blended that noise into a signal that had been fine on its own. Fusion does not average your components' strengths. It also averages their weaknesses, and it does so silently. On a different corpus, monolingual, terminology-dense, contracts or code or medicine, I would expect the opposite verdict. That is precisely why you run your own evals instead of inheriting conclusions from someone else's corpus.

Killing it still took a week of arguing with myself, which embarrassed me at the time and does not anymore. The component was built, tested, recommended by every reference we respected, and emotionally it felt like rigor. The eval set did not care how it felt. The discipline is not building the fancy thing. The discipline is deleting it when the numbers say delete, and I have come to believe the second skill is rarer than the first.

The second verdict was subtler: the Cohere reranker. A reranker takes your top candidates and reorders them with a heavier model, and unlike hybrid search, it worked. On the hard slice of the eval set, the stitched-together questions, the cross-lingual ones, it produced a real, visible improvement. We parked it anyway, and parked is the precise word, not killed. It stays in the codebase behind a flag, benched. The lift was real but concentrated in a minority of queries, and the price arrived on every query: another network hop in the latency budget, another per-call cost, another external dependency that can have an outage on a Sunday, another key to rotate, another version to pin. The bar for a new moving part in production is not "does it help." It is "does it help enough to pay for itself in latency, money, and 3 a.m. failure modes," and on our traffic the honest answer was not yet. The eval set holds its number; when the retrieval core stops being good enough without it, the flag flips and we will know exactly what we are buying.

There was a third temptation worth recording, even though it never made it far enough to be killed formally: skipping retrieval quality altogether and stuffing ever more context into the model. Windows are enormous now, the reasoning goes, so retrieve generously, pour it all in, let the model sort it out. The eval set punished this faster than anything else we tried. Answers got slower and more expensive with every extra chunk, and past a point they got worse, because the relevant sentence was drowning in plausible neighbors and the model's attention smeared across all of them. Precision beat volume every single time. The cheapest, most accurate token is the one you never send, which is the same conclusion I keep arriving at from every direction: the model is the last resort of the architecture, not the first.

What survived is almost insultingly boring, which I have learned to read as a compliment.

• Chunking shaped by how people ask. Not fixed token windows, but splits that respect the natural units of the material, so that the thing retrieved is the thing a question is actually about. More eval movement came from chunking than from any clever component we added downstream.
• Metadata as a first-class filter. Who, when, which part of the family. Narrowing the candidate pool with plain structured filters before any vector math is cheap, deterministic, and it never hallucinates. The rule-based reflex applies inside retrieval too: use rules for what rules can do, spend the model on what is left.
• The graph. Family questions are relationship questions. Who was at the wedding, which stories mention both grandmothers, what happened around the year the house was sold. Flat chunk retrieval answers those badly no matter how good the embeddings are, because the answer lives in the connections, not in any single chunk. The entity graph over the archive, GraphRAG in current jargon, survived every round of evals for the simple reason that it matches the shape of what users ask.
• Honest refusal. The no-answer questions in the eval set exist because the worst thing a family archive can do is invent a memory. Retrieval confidence low, say so, show what was found, let the human decide. Scoring refusal as a first-class outcome, rather than a failure, changed how we tuned everything upstream of it.

And around all of this sits the layer nobody writes conference talks about, the part between a working pipeline and a product strangers can trust. Ingestion that survives broken encodings, duplicate uploads, and files that lie about their format. Per-family isolation, enforced at the retrieval layer, because crossing that boundary even once is not a bug, it is a betrayal. Latency budgets, cost ceilings per conversation, graceful degradation when a provider stutters. And tests, currently 2,714 of them passing, most of which verify profoundly unexciting truths: this filter filters, this boundary holds, this refusal refuses. The pipeline from upload to answer is maybe a tenth of the code. The rest is the difference between a system that answers questions and a system a family can leave alone with its memories.

Which brings me to the word in the title. Honestly, productionising RAG meant that most of our engineering effort went into things that made no single answer smarter: eval sets, deletion decisions, isolation, refusal, tests. It meant the architecture got simpler over time, not more sophisticated, because every component had to keep re-earning its seat. And it meant accepting that the published state of the art is a menu, not a spec. BM25+RRF and rerankers are on every menu. On our corpus, one was poison and the other was a luxury. You do not know which menu items are yours until your own evals tell you, and any consultant who skips that step is selling you someone else's benchmark. The demo took days and made people say wow. The product took months and makes people say nothing at all, because it just works, quietly, in three languages, on the most unforgiving corpus I know. That silence was the expensive part, and it is the only part I am proud of.