Why I publish 77.8% instead of rounding up

The problem

Extracting data from an invoice with an LLM is easy to demo and hard to trust. You upload a PDF, the model returns a clean JSON object (vendor, date, total, line items) and it looks perfect. The problem is the word "looks." A model that is confidently wrong will happily hand you a total that does not add up, formatted just as neatly as a correct one. If a human pays that invoice, the tidy formatting was worse than useless. It hid the error.

So the real product problem is not "can the model read an invoice." It is "when the model is wrong, does anyone find out before it matters."

What I did

Extraction goes through a single tool call with a strict schema, so the shape of the output is guaranteed by the API and re-checked with Zod at the boundary. That means the structure is never the failure mode. Only the values can be wrong, which is exactly what I built the rest of the product around.

Every field gets a confidence badge, and the badge is the worst of two independent signals. One is the model's own report: high, medium, or low. The other is deterministic code: line items must sum to the total; subtotal plus tax must reconcile; dates must parse. If the arithmetic does not hold, the total is flagged red no matter how sure the model said it was. A model can be talked into agreeing with a wrong number. A sum cannot.

The two signals catch different things. Arithmetic catches a total that does not reconcile. The model catches a mis-read vendor name that no formula would notice. Taking the pessimistic side of both gives a reviewer one honest "look here first" signal. The whole UI is built for review, not for magic: the field table is editable, the checks re-run live as you fix values, and then you export the corrected result to CSV.

The measured result

I graded it on SROIE: 30 real scanned receipts from the ICDAR-2019 dataset, with ground truth I did not write. Ninety fields in total.

Overall field accuracy was 77.8%. Broken down: totals 93.3%, company names 73.3%, dates 66.7%.

I could have made those numbers say anything. If I had authored my own invoices and my own "correct" answers, I could have scored 100%, and measured nothing except my own consistency. A third-party benchmark I do not control is a far more honest signal, even though it is not a perfect match: SROIE receipts are shorter than invoices and only label three fields, so the eval does not test line-item extraction. I say that out loud rather than hide it.

77.8% is not a number you round up to "over 90% accurate" on a landing page, and that is the point. Dates at 66.7% is a real weakness. But because the product flags low-confidence fields for review, a wrong date shows up as something to check, not as a silent error in an exported file.

What I'd tell a client

Do not ask an extraction tool how accurate it is. Ask what happens when it is wrong. A tool that is 95% accurate and shows every field as confidently green is more dangerous than one that is 78% accurate and flags the other 22% for a human. The second one you can trust with real money, because it never pretends.

And be suspicious of anyone (including me) who grades their own AI on data they made up. The number will always be great. Insist on a third-party benchmark, or at least a test set the builder did not author. Imperfect but measured beats perfect but unfalsifiable, every time.

That is why I publish 77.8%. It is the honest number, it tells me exactly where to improve, and the product is designed so that the gap it exposes is caught by a human instead of paid by one.