Retrieval augmented generation, a practical explanation

Retrieval augmented generation is a plain idea wrapped in an intimidating name. Instead of expecting a language model to know your content, you look up the relevant passages at question time and hand them to the model along with the question.

The model's job becomes reading comprehension rather than recall. That is a much easier job, and it is why this approach works at all for content the model has never seen.

The pipeline, concretely

1. Split your content into chunks, typically a few hundred words each.
2. Convert each chunk into a vector, a list of numbers representing its meaning.
3. Store the vectors in an index.
4. At question time, convert the question into a vector the same way.
5. Find the chunks whose vectors are closest to the question vector.
6. Put those chunks and the question into a prompt.
7. The model answers using the supplied text.

Where quality actually comes from

Step five is where most answer quality is won or lost, and it is the step teams spend least time on.

If retrieval returns the right passage, a mid tier model will answer correctly. If retrieval returns the wrong passage, the best model available will answer confidently and wrongly, because from its perspective it was given the relevant context and asked to use it. It has no way to know that the retrieval step failed.

This is the practical consequence: when answers are bad, check what was retrieved before changing the model. Most of the time the model did a reasonable job with bad input.

Why similarity is not relevance

Vector search finds text that is semantically similar to the question. Similar is not the same as containing the answer.

Ask "how much does the Pro plan cost". A chunk that discusses pricing philosophy at length will score highly on similarity because it is full of pricing vocabulary. The pricing table, which is mostly numbers and short labels, may score lower despite containing the actual answer.

This is a known and structural weakness. Sparse or keyword based retrieval handles the table well and handles paraphrased questions poorly. Combining both, sometimes called hybrid retrieval, covers more cases than either alone.

The refusal path is part of the design

If nothing relevant is found, the correct behaviour is to say so. This has to be built in deliberately, because the default behaviour of a model handed weakly relevant context is to use it anyway.

Two mechanisms help. A similarity threshold below which you do not pass context at all, and an explicit instruction that answering from outside the supplied context is not permitted. Neither is perfect. Together they turn most would-be fabrications into an honest refusal, which is the outcome you want on a website where a wrong answer about pricing or policy is worse than no answer.

What to measure

Two numbers, measured separately, because they have different fixes:

• Retrieval hit rate: for a set of questions whose answers you know exist, how often does the correct chunk appear in the retrieved set. This isolates step five.
• Answer correctness: given correct retrieval, how often is the final answer right. This isolates the model and the prompt.

Measuring only the second conflates the two and sends you shopping for a better model when the fix was chunking. Build a set of twenty real questions with known answers and check both. It is an afternoon of work and it will change where you spend the next month.