[ HOW I WORK ]
Every claim is measured, linked, or labeled.
AI features are easy to demo and hard to trust. My work runs on one rule: a number is measured, a source is linked, and anything I can't yet prove is labeled as such, never asserted. Here is the method, and a build where you can check every figure yourself.
The principle
Most "AI chat" features hallucinate, cite nothing, and are graded on a good first impression. I treat an AI feature like any other engineering system: it has a spec, a test suite, and a regression gate. If a claim can't be measured or traced to a source, it doesn't go on the page. It gets marked pending until it can.
The method
Baseline → change → what moved
Nothing ships on a hunch. I measure the current behaviour, make one change, and re-measure. The delta is the result, not the demo that happened to work once.
Golden sets before features
Before building the retrieval or the agent, I write the questions it has to get right: a small, honest golden set drawn from real usage. The target exists before the code does.
Evals as regression gates
The eval isn't a one-off score; it runs before merge and exits non-zero below threshold. A change that quietly degrades quality can't land, so accuracy doesn't rot as features pile on.
A worked example: docs-chat
A grounded "chat with your docs" RAG: a single React component you drop into an existing Next.js app. Ask a question over a help centre and get a streamed, cited answer, with an honest "that's not in these docs" when it's out of scope. Every number below comes from a repeatable eval in the repo, not a screenshot.
- 27/27hit@1 on a 27-question golden set
- 27/27hit@5 (same set)
- ~3.05smedian answer, end-to-end
- $0.0035per query (measured)
- Golden set
- 27 realistic questions over a 22-document help centre. The eval embeds each question, runs the same cosine top-k as the app, and scores retrieval hit@1 / hit@5 per question.
- Regression gate
npm run evalexits non-zero if hit@5 drops below 80%, so a retrieval regression blocks the change before it merges. The eval is CI, not a vanity metric.- Cost & latency
- ~$0.0035 per query and a ~3.05s median answer on Claude Haiku 4.5 via OpenRouter, measured on real runs, not a list price.
- Abuse guards
- Per-IP rate limit (6 req/min sliding window), a global daily cap (200 req/UTC-day), and env-gated Cloudflare Turnstile, backstopped by the provider key's hard credit cap.
- Runtime
- Provider-agnostic: both generation and embeddings speak the OpenAI-compatible wire format, so switching model or provider is a one-line change. No vector DB, no Docker. One route handler, deployed serverless.
What an engagement looks like
Fit call
15 minutes on your process, where it leaks time, and whether AI is actually the fix.
Fixed-price proposal
Scope, timeline, and price in writing. No hourly meter running.
Build, in the open
Weekly demos and updates. You watch it start working, not wait for a reveal.
Handover & support
Docs, training, and an optional retainer for monitoring and improvements.
Want this level of proof on your build?
Book a 15-minute fit call →