27/27 retrieval hit@1 without a vector database

The problem

Teams want "chat with your docs" inside their own app: a box where someone asks a question in plain language and gets an answer from the company help centre, handbook, or API docs. Most versions of this are thin wrappers around a chat model. They hallucinate. They cite nothing. And the standard advice for building one is to stand up a vector database, run it in Docker, and wire up another service that can be down at 3am.

For a lot of products that is too much infrastructure for one feature. I wanted to know if I could build a grounded, cited, honest "chat with your docs" with none of it.

What I did

The corpus was a 22-document help centre. I split it into chunks of about 600 tokens with 80 tokens of overlap, keeping each chunk's title and heading path. Then I embedded every chunk once, at build time, into a small JSON file that gets committed to the repo. There is no database.

At runtime the whole thing is one Next.js route handler. A question comes in. I embed it. I score it against the index with a plain cosine top-k scan: a linear pass over a few hundred chunks, sub-millisecond. I pass the top chunks to the model as numbered sources and ask it to cite each claim inline with [n] markers. The route maps every [n] back to its source chunk and streams the answer to the browser, where each citation becomes a chip you can click to jump to the exact passage.

Two rules make it trustworthy. Every claim is grounded in a retrieved chunk. And when the answer is not in the docs, the model says so ("that's not in these docs") instead of guessing. The model is Claude Haiku 4.5 through OpenRouter; the embeddings are Gemini.

The measured result

I wrote a golden set: 27 real questions the system has to get right. The eval embeds each question, runs the exact same cosine top-k the app runs, and checks whether the right source comes back.

It scored 27/27 on hit@1 (the correct source was the top result every time) and 27/27 on hit@5. The median answer, end to end, was 3.05 seconds. Each query cost $0.0035, measured on real runs, not a list price.

Then I turned the eval into a gate. npm run eval exits non-zero if hit@5 drops below 80%, so it does not just produce a number once. It runs before a change merges, and a retrieval regression blocks the change. The eval is CI, not a screenshot in a slide. You can try the build yourself and read the source at the links below.

What I'd tell a client

First, you probably do not need a vector database to start. For a bounded corpus (a help centre, a handbook, a product's docs), a JSON index and a cosine scan are enough, and they remove a whole class of infrastructure you would otherwise run, pay for, and monitor. My retrieval function has one signature, so the day the corpus outgrows memory I swap a hosted vector store in behind it and nothing else changes. That day comes later than most people think, and often never.

Second, 27/27 is only meaningful because the questions came first. I wrote the golden set before tuning anything. If I had tuned the retrieval and then picked questions it happened to answer, the number would be marketing. Writing the questions first, from real usage, makes the number a test the build has to pass, every time, or it does not ship.

And honesty is a feature. A tool that says "I don't know" when the answer is not in the docs is worth more than one that always sounds confident. The confident one loses trust the first time it is confidently wrong, and you do not get it back.