Measuring answer quality without a labelled dataset

Every guide to evaluating a retrieval system assumes a labelled dataset: questions paired with correct answers, ideally hundreds of them. Almost nobody has this when they need it, and building one is exactly the kind of work that gets deferred forever.

Two proxies are available immediately and cost nothing to collect.

Proxy one: unanswered rate

The share of questions where the assistant said it could not answer. This is a directly observable number and it moves for readable reasons.

It is a proxy for coverage rather than accuracy, and the relationship to quality is not monotonic. A very low unanswered rate is not good news; it usually means the refusal threshold is too permissive and the system is answering things it should not.

What you want is for the number to be stable and for changes to be explainable. A jump after a content change means something broke in indexing. A jump with no change on your side means traffic composition shifted.

Proxy two: human correction rate

Among conversations that reached a human, how often did the human have to correct something the assistant said.

This is the strongest available signal for accuracy because it is a real judgement by someone who knows the answer, generated as a byproduct of work that was happening anyway. It requires one small process change: the person handling the escalation marks whether a correction was needed.

One checkbox. That is the entire instrumentation cost, and it produces a labelled dataset over time as a side effect.

Building the golden set incrementally

Every corrected answer is a labelled example. Question, wrong answer, correct answer. After a few months of collecting these you have the dataset you could not justify building upfront, and it is drawn from real traffic rather than from someone's imagination of real traffic.

This is the practical path: do not build the eval set first, build the mechanism that generates it.

What thumbs up and down does not tell you

Inline feedback widgets have a low response rate and a strong selection bias. People rate when they are annoyed. A thumbs down rate is closer to a frustration measure than an accuracy measure.

It is not useless. Individual thumbs down conversations are worth reading because they point at specific failures. But the rate should not be presented as an accuracy metric, and comparing it across time is unsound because the population rating it changes.

Separating retrieval failure from generation failure

When you review a bad answer, the diagnostic question is which step failed. Look at what was retrieved.

• Correct chunk retrieved, wrong answer produced: a generation or prompt problem.
• Wrong chunk retrieved: a retrieval problem, usually chunking or source selection.
• Nothing relevant existed to retrieve: a content problem, and the right fix is writing the missing page.

These have completely different fixes and the third is the most common on a new deployment. Logging the retrieved chunk IDs alongside each answer costs almost nothing and makes this diagnosis take seconds instead of guesswork.