The reranker held up on 30 requests it had never seen

One recorded run

A bold red under $25

With a $25 price cap passed as a filter, the hybrid retriever and reranker returned four bottles, and Claude led with a Chilean Carmenère and explained why it passed over one of the four. Every bottle, score and price in the answer comes from a retrieved review.

Query by query

The reranker lost 6 of 42 queries to plain dense search

An average can hide a system that wins big on a few queries and loses on the rest. Each row is one query: the hollow square is dense search, the filled square the reranked pipeline, on nDCG@10. Rows are sorted by the change. Two queries score zero for both, and the error analysis below says why.

Dense Hybrid + rerank

Hover or focus a row for its numbers.

BM25 fusion

Fusion hurt on the first 12 queries and helped on the next 30

On the July set, fusing a BM25 keyword index into the dense ranking left precision flat and dropped MRR from 0.82 to 0.77: the first relevant wine landed lower. On the 30 new queries the same fusion raised MRR from 0.73 to 0.82 and nDCG@10 from 0.59 to 0.68. The new queries name grapes and places such as Grüner Veltliner, Mosel and Mendoza, exact words BM25 matches; the July queries mostly describe a style, where shared words point to the wrong kind of wine as often as the right one. So the July finding was a property of those queries, and the honest reading of both sets is that fusion depends on how people phrase requests, while the reranker helped on both.

Fusion can still promote a weak match. Reciprocal Rank Fusion rewards any review that shows up in both lists, even far down them: a review ranked 61st in both outscores the one dense search put first. The calculator uses the formula from src/hybrid.py (k = 60, 80 candidates from each list in the benchmark).

The cross-encoder (ms-marco-MiniLM-L-6-v2) reads the request and each review together and re-scores the top fused candidates. It is also the slow stage: its median of 145 ms is most of the 196 ms median for a whole query.

Error analysis

76 misses in 300 top-ten slots, and what kind they are

A miss is a wine in the reranked top ten that fails its query's rubric. Every top ten is saved in eval/top10.json, and eval/error_analysis.py sorts each miss by the first rubric clause it breaks. On the 30-query set there are 76; on the July set, 22 of 120.

The price group is a design gap: price, country and grape filters exist, but only as CLI flags. Nothing reads "under $15" out of the sentence, and the benchmark runs with no flags, so the retriever ranks a $29 Syrah as the best smooth red. The fourth kind is the rubric's own error: 16 of 76 misses are wines a sommelier would accept, so all the 30-query scores are a little low.

The answers stayed inside the retrieved reviews

eval/generation.py ran the full chain on 20 of the queries and checked every answer in code: no [n] citation outside the six sources, no score that isn't in them, and no dollar amount that isn't a source price, the guest's budget, or the difference between the two (six amounts such as "$10 over your limit"). In the two runs where retrieval found nothing that met the request, the answer said so in its first line: "None of these six wines is under $15. The cheapest is $19." Twenty answers is a small sample, and these checks catch invented numbers and say nothing about whether a pick is well argued.

Cost and latency

Retrieval takes a fifth of a second; the answer takes eight

Retrieval was timed on an Apple M4 (embedding and reranker on the MPS GPU), replaying the default path (60 fused candidates, top 6) for all 42 queries three times after a warm-up pass: 126 timings. Generation is 20 runs through the Claude CLI on 4 November, the same way src/llm.py calls it, with --output-format json added so the CLI reports time, tokens and cost. Cost is the API-equivalent figure the CLI reports; on a Claude subscription nothing is billed per call.

The slowest generation took 26.6 seconds, on the under-$15 request where nothing fit. The CLI's default model on that day was claude-opus-5-5. Each call carried about 28,000 input tokens, while the request, six reviews and instructions come to 450 to 620 words: nearly all the cost is the CLI's own context, which a direct API call with ANTHROPIC_API_KEY would not send. That path was not measured. Raw numbers: eval/latency_retrieval.csv, eval/generation_runs.jsonl.

The grounding

It cannot recommend a bottle it did not retrieve

The system prompt tells Claude to recommend only from the retrieved reviews, to cite each pick by its [n] index, and to say so if none of them fit. Price, country, grape and minimum score can go into the vector query as metadata filters, so a $25 cap never spends a retrieval slot on an $80 bottle. The answer prints with the reviews it cites underneath.