My RAG pipeline lied to me (three times)

In March I rebuilt my document assistant. I had hacked together versions of it before, quick weekend builds that worked well enough to be interesting and badly enough to be abandoned. This time I wanted to do it properly, with the standard recipe the whole industry uses.

The recipe is called RAG, retrieval-augmented generation, and the idea is simple. Language models do not know your private documents. So before asking the model anything, you search your documents, take the best matches, and paste them into the question. The model then answers from what you gave it. To make that search work, you cut every document into small pieces and store them in a way that lets you find pieces by meaning, not just by exact words.

My documents were a mixed bag: datasheets, wiki pages, old reports, some German, some English, some a PDF of a scan of a fax. The demo worked in an afternoon, which is exactly the trap. These demos always work. You ask a question you already know is in the documents, the right piece comes back, the model answers beautifully. Then real questions arrive, and the education begins.

Lie one: a confident answer from half a table. My splitter cut documents into equal-sized pieces, blind to meaning, and it cut an important table in half. The column labels landed in one piece, the numbers in the next. Asked about an allowed input range, the assistant found the piece with the labels, could not see the numbers, and answered with numbers from a neighboring product whose little table had survived intact in the same piece. Confident, plausible, wrong. A person looking at that piece would have said “the table is cut off here”. The assistant said no such thing.

Lie two: similar is not the same as relevant. Searching by meaning finds text that talks about the same topic, not text that answers the question. “How do I reset the controller” is, by meaning, very close to five documents that explain what the controller is. None of them says how to reset it. The search did its job; its job was just not what I thought it was. The standard fix is a second, slower step that actually reads the found pieces and re-sorts them by whether they answer the question. It fixed about half of these cases and made everything a bit slower. Fair deal.

Lie three: German documents, English questions. My pages say “Störung”, “Betriebsanleitung” and “Kap. 4.2” where the questions say “fault”, “manual” and “chapter 4”. Searching across languages mostly works, and “mostly” is exactly the problem. The fix was not clever: clean the documents, write out the abbreviations, and accept that this kind of project is one third AI and two thirds text janitor work.

All three lies have the same cause, and it took me the whole month to say it plainly: the model was never the weak point. The search was. Every bad answer traced back to the wrong text being pasted into the question. Not once did the model fail me when the right piece was actually there. So the effort belongs on the boring side: how you cut, what you clean, what you fetch.

One question I could not answer in March: why is the search strategy fixed at all? My pipeline always fetches five pieces, always the same way, whether the question needs one table or a whole manual. A person does not research like that. They look, judge, and look again somewhere else. I wrote “retrieval should be a decision, not a setting” into my notes. Maybe that is a real thought, maybe it is just frustration written at 22:30. I cannot tell yet.

Meanwhile the news kept happening: DeepSeek shipped a giant new model, and Apple of all companies started putting Gemini into iPhones. I noted it and went back to cleaning umlauts. There is a version of this hobby that is all headlines, and there is the version where you fix a text-cutting bug at 22:30. Only one of them produced answers anyone but me could use.

If you build one of these, and you should, because the failures teach more than any tutorial: budget one afternoon for the pipeline and one month for the humility.