No green checkmark for allergens, ever

Menu Decoder

Photograph any foreign-language menu, say your dietary constraints out loud — get every dish translated, allergen-flagged with a deliberately fail-safe design, and pronounced, plus an honest "what I'd order for you."

By Jeremy Lee

✗ contains ⚠ may — ask the staff ○ not indicated

Why

Ask a chatbot to read a menu you can't decipher and tell you what's safe, and you're trusting it to translate correctly and never quietly clear an allergen it wasn't sure about. Menu Decoder is built around one asymmetric rule: a missed allergen is dangerous, a false warning is just a shrug — so the system is biased toward warning everywhere it can be. There is no code path anywhere in this app that renders a green checkmark for an allergen.

📷

One photo, whole menu

Vision extraction reads every dish in one call — unreadable lines are flagged, never silently dropped, with a completeness % shown.

🧠

Locally fine-tuned tagger

A LoRA-tuned Qwen2.5-1.5B model tags cuisine + all 14 EU allergens + spice + vegetarian/vegan per dish — runs on a laptop, $0 marginal cost.

🛡️

3-layer allergen fail-safe

Cautious prompt rubric · unoverridable badge rendering · deterministic safe-list filter before the LLM ever picks.

🔊

Dish-name pronunciation

Tap a dish to hear its name spoken in the menu's own language — unsupported locales hide the button rather than faking it.

The tagger, honestly benchmarked

Given a dish name (+ optional description), the model emits the full allergen/dietary JSON tagging — a generative structured-output task, not a single-label guess. Trained on 6,750 synthetic dish rows across 15 cuisines, split by dish family so a dish's 3 prompt variants never straddle train/test. The metric that matters is allergen recall on "contains" and the false-safe rate — reported per allergen, not averaged away.

System	Allergen recall	False-safe rate	Cost
Base Qwen2.5-1.5B (bare prompt)	0.0%	100.0%	$0
+ LoRA fine-tune	69.3%	7.3%	$0
Claude teacher (bare prompt, same as LoRA sees)	0.0%	100.0%	Max subscription
Claude teacher (schema-primed)	79.2%	1.7%	Max subscription

🔎 Claude appears twice on purpose. Given the identical bare prompt the LoRA model trained on (no schema described anywhere), Claude answers a different, reasonable question and its own JSON shape — hence the ~0% row, which is a real measured result, not a bug. Told the exact output contract (no answer key), it jumps to 79.2% recall. That gap is the finding: the LoRA model needs zero schema in its prompt because the schema is baked into its weights — a real token-optimization result, detailed in writeup.md alongside two real CLI bugs found while building this benchmark.

📋 Human slice not yet available. The spec's mandatory ~60-dish stratified spot-check (Jeremy hand-verifying the labels) hasn't run yet — it requires him personally. Reported as an open gap, not hidden or faked.

Full per-allergen benchmark →

How it works

Photograph the menu — one vision call extracts every dish, its translation, price, and an explicit unreadable-line flag where needed.
Speak or type your diets and allergens — parsed into a structured preference brief.
Each dish gets tagged locally (LoRA) or via a batched Gemini call on the hosted Space — badges render with no green checkmark ever possible for an allergen.
The safe-list filter runs in code first; the model only picks 2–3 recommendations from what already passed.
Tap any dish to hear it pronounced in the menu's own language.

🔒 Everything runs locally except one claude -p call per menu for extraction — no metered API, no vision API key required. See the full writeup for the honest engineering story, including the real bugs found and fixed along the way.

View on GitHub Read the writeup