How I know my AI thing actually works
Forty real questions in a plain text file taught me more than every benchmark I read this year.
June began with an uncomfortable question: how do I know that the little document assistant I built actually works? I had three good stories and a warm feeling. If a bridge engineer said that, you would take a different bridge.
What pushed me was a number from a report going around this month: 72 percent of AI agent projects are now in real use, not just demos. Real use means someone depends on it. And depending on a warm feeling is how you end up explaining a breakdown with the word “vibes”.
So I built the least impressive thing I have made all year, and the most useful: a list of forty questions in a plain text file.
They are real questions people actually asked the assistant, plus the mistakes I already knew about. For each one I wrote down two things: what a correct answer must contain, and what it must not contain. That second part matters most. A dangerous wrong answer is never nonsense. It is a believable number from the wrong table. If you only check that the right thing is present, the believable nonsense slips through. And believable nonsense is the only failure that actually scares me.
A small script asks the assistant all forty questions and reports pass, fail, or “needs a human look”. There is no AI magic in any of this. It is a checklist, pointed at a system that answers a little differently every time. Build time: one evening, plus one lunch break of arguing with myself about edge cases.
It paid off in the first week. The company behind the model I use updated it, silently, as they do. Suddenly the assistant handled German abbreviations differently, and two of my forty questions flipped from pass to fail. Without the list I would never have noticed. Or worse: someone else would have noticed, quietly trusted the tool a little less, and never told me. Trust erodes silently. Checklists are loud. I now run the forty questions after every change I make, and every Monday, because the model underneath me changes whether I am watching or not.
Can another AI do the grading? I tried it, with honest results. For questions that have an exact right answer, like “does it name the correct document”, simple text comparison wins and costs nothing. For soft questions, like “is this explanation clear”, the judging model agreed with me about as often as a coin flip would, and it consistently preferred longer answers. Longer, not better. My rule now: automatic checks where an exact answer exists, my own eyes for anything important, and an AI judge only for rough sorting when the pile is big.
The nicest side effect showed up this morning. Claude Sonnet 5 came out today and GPT-5.6 is mid-rollout, and instead of spending a week on comparison threads, I ran my forty questions against the newcomers over coffee. One of them failed five, including both German cases. Debate over. The little file turned “which model is best” from an opinion into forty rows.
And the uncomfortable lesson underneath: writing those forty questions forced me to say, precisely, what “working” means for my assistant. I had been avoiding that for months, because showing a cool demo is easier than making a promise. Nobody warned me that the hardest part of building with AI is admitting what you actually expect from it.