Local OCR, grounded retrieval and a fine-tuned grape classifier

A 1.5B model tuned on a laptop names the grape from a blind note

Cellar Scanner reads a wine label with local OCR and Claude, retrieves similar wines from 30,000 Wine Enthusiast reviews, and keeps only the sentences it can ground in those reviews. Its Blind Taste Mode runs a Qwen2.5-1.5B model, LoRA fine-tuned with MLX on Apple Silicon, that guesses the grape from a tasting note with the grape name masked out: 27.7% Top-1 across 40 varieties, against 8.7% for the untuned base model and 23.0% for Claude prompted zero-shot.

Read the writeup Code on GitHub

27.7%

Top-1 accuracy of the LoRA model on 40 grape varieties, 3.2 times the base model's 8.7%.

0.253

macro-F1 for the LoRA model, against 0.119 for Claude zero-shot and 0.021 for the base model.

10.6 s

per note on the laptop, against 69.6 s for Claude through the CLI.

20.1%

of notes named their own grape and were masked before training and scoring.

At a glance

Task: Name the grape, one of 40 varieties, from a tasting note with the grape's name masked
Eval set: The first 300 notes of a 2,024-note held-out split, split by wine with a leakage check and shuffled with a fixed seed; Claude ran on the first 200 of them
Baseline: Claude zero-shot 23.0% Top-1; the untuned base model 8.7%
Result: LoRA 27.7% Top-1, macro-F1 0.253. On the 200 notes both saw the lead is within noise (sign test p = 0.33)
Cost per note: $0 per call on the laptop; Claude $0.054 API-equivalent ($53.64 per 1,000), from 25 notes rerun on 2026-11-05 with the benchmark's prompt and CLI flags (eval/claude_cost/). Most of it is the Claude CLI's own context, about 13,500 tokens written to cache per call
Latency: LoRA 4.0 s median, 4.8 s p95 (n = 300, model reloaded every call); Claude 13.5 s median, 40.4 s p95 (n = 200). A few slow calls pull the means to 10.6 s and 69.6 s
Main failure: It falls back on a few default answers: of the 75 errors in its 15 most common confusions, 72 answer Viognier (36), Cabernet Franc (22) or Sangiovese (14)

The benchmark

Base model, LoRA, and Claude on the same task

training/bench_classifier.py scores three systems on the held-out test split (2,024 rows), shuffled with a fixed seed. The base and LoRA models were run on the first 300 rows, because mlx_lm.generate reloads the model on every call; Claude was run on the first 200 of those same rows.

System	N	Top-1	Top-3	Macro-F1	Latency	Cost/1k
Base Qwen2.5-1.5B	300	8.7%	14.3%	0.021	4.3 s	$0
+ LoRA (this project)	300	27.7%	27.7%	0.253	10.6 s	$0
Claude, zero-shot	200	23.0%	30.5%	0.119	69.6 s	Subscription

The LoRA model is ahead of Claude on Top-1 by 4.7 points. Because Claude's 200 notes are a subset of the LoRA model's 300, the two can be compared note by note: on those 200, the LoRA model was right on 55 (27.5%) and Claude on 46 (23.0%). Of the 67 notes only one of them got right, the LoRA model took 38 and Claude 29, and an exact sign test puts that split at p = 0.33. It is a lead that sampling noise could erase. The macro-F1 gap (0.253 against 0.119) is wider, which says the LoRA model's hits are spread across more varieties. Claude is ahead on Top-3 (30.5% against 27.7%).

The LoRA model's Top-1 and Top-3 are identical because it never fills ranks 2 and 3. Its training target was the single line "1. {variety}", so it learned to give one answer and stop. A retrain with three-item targets would let it use the extra guesses.

Your turn

Taste blind against the model

These are the 200 held-out notes that all three systems were scored on, with the grape masked exactly as the models saw it. Pick a grape, then see what the LoRA model and Claude answered. The models chose from all 40 varieties; you get four options, which always include the right answer and both models' picks, so your job is easier than theirs.

Note 1 of 200

Play a few notes to start a tally.

Notes, true varieties and answers from eval/blind_samples.json, which training/paired_check.py writes from the cached benchmark outputs. Notes come up in a random order, with the reviews' dashes set as commas. On all 200, the LoRA model was right on 55 and Claude on 46.

Where it goes wrong

The mistakes are the ones a taster would make

The most frequent confusions in the LoRA model's test predictions pair grapes that share a style: Cabernet Sauvignon read as Cabernet Franc, Nebbiolo read as Sangiovese, and a group of aromatic or fresh whites read as Viognier.

Blind tasting, marked

Most common confusions, LoRA model

PouredCalledTimes

Cabernet SauvignontoCabernet Franc10

Pinot GristoViognier8

SyrahtoCabernet Franc6

NebbiolotoSangiovese6

Red BlendtoSangiovese5

ChardonnaytoViognier5

Portuguese WhitetoViognier5

Grüner VeltlinertoViognier5

RieslingtoViognier5

Sauvignon BlanctoViognier5

From eval/benchmark.md, 300-row subsample.

Confusion matrix for the twelve most frequent test varieties. The largest off-diagonal cell is Cabernet Sauvignon predicted as Cabernet Franc, ten times; Rosé and Sparkling Blend are mostly correct. — Confusion matrix for the twelve most frequent test varieties.

Making the task blind

Masking the grape name, and three bugs found by running it

Many reviews name the grape outright ("this Cabernet shows..."), which would turn blind tasting into string matching. A regex masks each variety name and its common synonyms before training and scoring; it fired on 20.1% of the 41,389 rows, and the rate is written to eval/masking_report.json.

The dataset keeps the 40 varieties with at least 500 reviews, caps each at 1,200, and splits by wine title so no wine appears in both train and test (37,289 train, 2,076 validation, 2,024 test; the leakage check in eval/dataset_card.json passed).

Three numbers looked wrong during the build, and each was a bug. The first index build reached a third of its 30,000 target until a water-filling allocation replaced a flat per-variety quota. Claude first scored below the untrained base model because markdown-wrapped answers failed an exact string match; answers are now matched against the class list. And the first benchmark read the test file in order and covered only 6 of 40 varieties; a seeded shuffle fixed it (40 of 40 in the 300-row subsample) and changed the headline result.

The training run

800 iterations on a laptop

The adapter was trained with mlx_lm.lora on the 4-bit Qwen2.5-1.5B-Instruct: 12 layers, batch 4, 800 iterations, learning rate 1.5e-4, sequence length 768. That is 3.96M trainable parameters (0.256% of the model), 3,200 training examples seen, and a peak of 4.4 GB of memory. Validation loss fell from 3.895 at the start to 1.154 at iteration 600 and finished at 1.196.

Training and validation loss for the LoRA fine-tune over 800 iterations, falling steeply in the first hundred iterations and then flattening near 1.2.

Loss over the run, parsed from training/train_run.log: training loss every 20 iterations, validation loss every 100. Hover to read a checkpoint.

The scanner

From a label photo to a cited profile

tesseract reads the label locally. One claude -p call checks that the photo is a wine label and extracts a structured card (producer, grape, vintage, region). The card queries a Chroma index of 30,000 reviews, embedded with sentence-transformers and sampled across every represented variety. A second Claude call writes a profile and pairings from the retrieved reviews, and a token budgeter records what went into that prompt.

Domain gateA photo that is not a wine label gets a refusal, and no card is made up for it.
Field validationVintage is range-checked; the grape is validated against the winemag vocabulary.
Claim groundingEach sentence of the profile and each pairing reason must match a retrieved review; the rest are dropped and the answer regenerates. Every citation is the id of a retrieved review.
Confidence gateA low-confidence read asks for a retake.
Semantic cacheRe-scanning the same bottle returns the cached answer; the hit rate shows in the dev panel.

The scan pipeline, with Blind Taste Mode as a separate branch.

Limitations

What the numbers do not show

The LoRA-versus-Claude gap on Top-1 is within sampling noise (sign test p = 0.33 on the 200 notes both saw). The LoRA model cannot rank a second or third guess. Training saw 3,200 of the 37,289 training rows; whether a longer run would help is untested here. The labels are the dataset's variety field, which mixes grapes with blends and styles (Red Blend, Rosé, Port), so some "errors" are arguable.

There is no hosted demo: the app is a Gradio interface run locally with python app.py. Label reading depends on OCR quality; when the read is weak, the confidence gate asks for a retake.

How it was built

Stack

Fine-tune: Qwen2.5-1.5B-Instruct (MLX 4-bit), LoRA via mlx_lm.lora on Apple Silicon
Training data: 41,389 masked Wine Enthusiast notes, 40 varieties, split by wine title
Index: 30,000 reviews, water-filling stratified by variety; sentence-transformers + Chroma
Vision: tesseract OCR, then claude -p for the domain check and extraction
Generation: claude -p on a Claude subscription, so no per-token cost
App: Gradio (app.py): Scan and Blind Taste Mode
Benchmark: training/bench_classifier.py; results in eval/. training/paired_check.py re-scores the cached outputs on the 200 shared notes
Related: The same review corpus is analyzed in Wine Score Inflation and searched in Wine Sommelier RAG