My eval failed at 36.7%, and that was the best thing that happened to my copilot

The problem

I built an AI support copilot for Tessera, a demo admin app. It can look up a customer, list open items, issue a refund, extend a trial, send an email. The read-only tools run on their own. The mutating ones (anything that touches money or customer state) pause and wait for a human. The operator sees a card with the exact action and its arguments, then clicks Approve or Reject. Nothing writes to the database until they do.

I wrote an eval for it. About thirty commands: read-only ones, mutating ones, and ones the copilot should refuse or ask to clarify. It scores whether the copilot picked the right tool and passed the right arguments. It exits non-zero below 80%, so a bad change cannot merge.

I ran it. It scored 36.7%.

What I did

36.7% is bad. But a number is only useful when you know why. So I read the failures, and there was a clear pattern. Every read-only command passed. Every mutating command failed. And they all failed the same way: no tool call at all. The copilot answered in plain chat instead of emitting the approval-gated tool call the eval was waiting for.

My first guess was the harness. The mutating tools need a human approve step, and an eval does not click buttons, so maybe I was driving the approval flow wrong. I checked. It was fine. My next guess was the provider. The copilot can run on OpenAI directly or on OpenRouter serving the same model, so maybe the fallback path dropped the tools. I checked that too. The tools were being sent. The model was choosing not to call them.

So it was not the harness and not the provider. It was the prompt. I read the system prompt line by line, and there it was. For mutating actions it told the model to "propose the action" to the operator. I meant that as: emit the tool call, and the UI turns it into an approval card. The model read it literally. It proposed the action, in words. "I can refund $40 on invoice INV-1004, shall I go ahead?" That is a reasonable thing to do when someone tells you to propose something. It is just not a tool call, so nothing ever reached the approval card, and the eval saw nothing.

The fix was one idea, written into the system prompt: emitting the tool call is the proposal. The approval card is the proposal. Do not ask in chat. Call the tool, and the human approves it. The gate is the UI, not your words.

The measured result

Here is the honest part: the re-run is still pending. I ran out of API credits for the day, and I am not going to invent the after number. That would defeat the whole point of having an eval. So right now I have a real 36.7% before, a root cause I understand, a one-line fix, and a re-run I will post once the credits reset.

And that is fine. It is the best thing that happened to this build. The eval caught a real product bug before any user did. If I had shipped on the demo (where I click the happy paths and everything looks great), the copilot would have quietly refused to do the one thing it exists to do, every single time an operator asked for a refund. The eval turned a silent, embarrassing failure into a number and a stack of failing cases I could actually read.

What I'd tell a client

A demo tells you the software can work once. An eval tells you how often it works, and it hands you the failures when it does not. The failures are the valuable part. 36.7% was not a disaster; it was information I could act on in an afternoon. The disaster is the build with no eval that feels fine in the demo and breaks in front of a real user.

And read your system prompt the way the model reads it. Literally. "Propose the action" and "emit the approval-gated tool call" mean the same thing to me and completely different things to the model. Most of my hardest AI bugs are not model failures. They are me being unclear, and the model doing exactly what I said.