RAG in production
Document search with embeddings and a reranker, and an evaluation with fixed scores instead of testing by feel.
MRR 0.950on a fixed set of evaluation questions
This case is described anonymously: no client name, no code and no screenshots.
Problem
A search feature that answers from your own documents is quick to build but hard to judge. Without a fixed measurement you do not know after each change whether the answers got better or worse. You end up testing by feel.
What I did
- Split documents into passages and stored an embedding per passage, so a question finds the passages closest to it in meaning.
- Added a reranker that scores the retrieved passages against the question again.
- Built an evaluation with a fixed set of questions for which the right passage is known. Every change now gets a score.
- Tested a summary index as an alternative and threw it away again. That story is in a separate article.
What I measured
The evaluation uses MRR (mean reciprocal rank). If the right passage is in position 1, that counts as 1. Position 2 counts as 0.5, position 3 as 0.33. The average over all questions is the score. The score came out at 0.950.
To be honest about it: that score was the same with and without the reranker. The measurement looked at the position of the passage after the passages had been sorted again, so it never saw the reranker's order. The number therefore says something about finding passages, not about the reranker.
The summary index cost 7.7 seconds per file, against 0.25 seconds for embedding. That is roughly 30 times slower.
What I learned
- Measure first, build second. A fixed score ends the discussion about what "feels better".
- A measurement can be blind to exactly the part you want to judge. Check what the number does and does not see.
- Throwing something away after measuring it is a result too.