Wine search is either useless filters or a chatbot inventing vintages.
"A bold red under $25 for steak night," answered with real bottles.
Ask in plain English and get two or three genuine recommendations, each
cited to a professional review from an index of ~130,000 Wine
Enthusiast tastings. The sommelier can only recommend from what was
retrieved, and if nothing matches it says so. Retrieval is local and free.
And because "better retrieval" is a testable claim, it was tested — with a
result that went against the fashionable answer.
nDCG@10 from adding a cross-encoder reranker. The single change that
mattered.
−0.12
MRR from adding BM25 fusion. The hybrid step that everyone recommends
made this system worse.
130k
real reviews behind every recommendation, each with a [n]
citation you can check.
$0
for retrieval. Local embeddings, local index, nothing leaves the
machine until generation.
↓ the measurement
Three retrievers, one query set
src/eval.py scores three retrieval configurations against twelve
rubric-labelled queries, where relevance is defined by explicit grape, price
and style rules. That rubric is a transparent proxy for human judgement
rather than a perfect one, and it is stated as such.
System
P@10
MRR
nDCG@10
Dense (vector only)
0.68
0.88
0.69
Dense + BM25 (RRF fusion)
0.67
0.76
0.68
+ cross-encoder rerank
0.82
0.92
0.83
The three configurations across all three metrics.
↓ the finding worth keeping
Bolting on BM25 made it worse
Hybrid retrieval — dense vectors fused with a keyword index — is close to
received wisdom. On this corpus it did not work. Precision fell slightly and
MRR dropped from 0.88 to 0.76, which is a meaningful
regression in how high the first good result lands.
The cause is visible in the results: BM25 drags in reviews that share
vocabulary with the query while being about the wrong wine. Reciprocal Rank
Fusion then rewards those lexical matches for appearing in one list, and they
push genuinely relevant bottles down.
The cross-encoder reranker is what actually helps, because
it reads the query and the review together rather than comparing two
independently-produced vectors. It lifts precision and nDCG about 21% over
the dense baseline.
Adding the component everyone recommends made this system worse. The
result is in the repository because a benchmark you only publish when it
agrees with you is not a benchmark.
On why the BM25 row stayed in the table
↓ the grounding
It cannot recommend a bottle it did not retrieve
Generation is instructed to recommend only from the retrieved reviews and to
cite each pick with a [n] that points at a real review. When
nothing in the index matches the request, the correct output is to
say nothing matched rather than to widen the search quietly
and present the result as if it fit.
The pipeline as benchmarked, fusion step included.
↓ what this cannot tell you
Twelve queries is twelve queries
The evaluation set is small, and the relevance labels come from a rule-based
rubric rather than from people. Both facts limit how far the numbers travel.
A 21% nDCG lift on twelve rubric-scored queries is a real signal and not a
published benchmark result.
The rubric is also the sort of thing that can quietly encode its own
preferences — a grape-and-price rule will favour retrievers that match on
grape and price. It is written down in the repository so that bias is
inspectable rather than implicit.
Finish · what it is built on
Where the numbers come from
Corpus
~130,000 Wine Enthusiast reviews
Dense retrieval
sentence-transformers + Chroma, entirely local
Sparse retrieval
BM25, fused with Reciprocal Rank Fusion
Reranking
A cross-encoder reading query and review together
Generation
Claude CLI by default, so no per-token cost; ANTHROPIC_API_KEY switches to the API
