[ WRITING ]
Notes from building AI that has to be trusted.
Short, honest write-ups from real builds: what broke, what I measured, and what I would tell a client. No hype, and no number I can't point at in a repo.
- § 01
My eval failed at 36.7%, and that was the best thing that happened to my copilot
A tool-calling support copilot scored 36.7% on its eval. Every mutating command failed with no tool call. The bug was one line in the system prompt, and the eval caught it before any user did.
- § 02
27/27 retrieval hit@1 without a vector database
A grounded "chat with your docs" RAG with no vector DB and no Docker: a build-time JSON index and a cosine scan in one route handler. 27/27 hit@1 on a golden set, 3.05s median, $0.0035 a query.
- § 03
Why I publish 77.8% instead of rounding up
An invoice extractor measured on a third-party benchmark: 77.8% overall, totals 93.3%, dates 66.7%. The product answer is not a bigger number. It is confidence gating that flags every weak field for a human.