Local LoRA fine-tune, benchmarked against Claude
Menu Decoder reads a photographed menu, translates each dish, and tags it
for cuisine, 14 EU allergens, spice and diet. The tagger is Qwen2.5-1.5B
with a LoRA adapter that writes the whole tag set as JSON from a prompt as
short as Dish: Pad Thai. Claude given that prompt describes
the dish instead. Claude given the full schema catches more allergens
than the fine-tune does, and misses fewer outright.
Dish: X. prompt, write JSON tags for cuisine (15), 14 EU allergens, spice and dieteval/claude_cost/)contains. False-safe is the share of contains dishes tagged not_indicated, the dangerous miss; a may still warns, so it does not count. Claude ran on a 100-row subsample.Benchmark
Given only Dish: X., Claude answers a different question,
"tell me about this dish". It returns valid JSON 94% of the time, but
under its own keys (region, main_ingredients,
flavor_profile), none of which match the scored fields, so
its allergen scores are zero by construction. The second row appends
src/decoder.py's tagging rubric and a description of the
output format, with no examples and no answers. With that, Claude beats
the fine-tune on allergens.
The fine-tune's case is narrower. The 14-allergen contract lives in its weights, so the prompt can be one line, and it runs locally with no per-call cost. It also gets cuisine right 99.3% of the time against Claude's 81.0%.
Per allergen
An average hides the dangerous cases, so the benchmark scores each allergen on its own. Switch to false-safe to see where the fine-tune tells a diner a dish is clear when it is not.
Within 11 points on the eight common allergens, then the fine-tune falls behind
Higher is better. Rows run from most to fewest test cases.
contains for that allergen: the first figure for LoRA on all 303 rows, the second for Claude on its 100-row subsample. Faded rows have fewer than 5 Claude cases. Lupin had no contains cases and is left out.
Molluscs is the worst result: the fine-tune tagged 39.1% of mollusc
dishes "not indicated", and peanuts follow at 18.2%. Sulphites and celery
show 0% recall with no false-safes, which means the fine-tune never said
contains for either; it said may, a warning
without confidence.
Fail-safe design
The tagger misses allergens, so the app around it assumes it will. Three layers push every decision toward a warning, and two of them are plain code the model cannot override.
may whenever an allergen is plausible in a typical preparation: sesame oil in many Asian dishes, gluten in most sauces and breading. "When uncertain, prefer may over not_indicated."ALLERGEN_BADGE in src/safelist.py maps contains to a cross, may to "ask the staff" and not_indicated to an open circle. It has no entry for a check mark.filter_safe_dishes removes every dish marked contains or may for any of the diner's allergens, and any dish that fails a stated diet or spice limit, before the recommendation call. The model picks from what is left.The cost of layer 03 shows up when you run it over the fine-tune's own tags for the 303 test dishes. Pick allergens below and see how much of the menu survives.
A gluten allergy clears none of the 303 test dishes
The fine-tune tagged gluten contains on 137 dishes and may on the other 166.
is_safe_for_brief in src/safelist.py, applied in your browser to the fine-tune's stored output for each test dish. A dish is counted under the first rule that removes it. The fine-tune called 11 of the 303 test dishes vegetarian and none vegan.The filter is doing what it was built to do, and this is what it costs. For gluten the filter leaves nothing, for sulphites 31 dishes, and for soy or sesame fewer than half the menu. A diner with one of those allergies gets few or no recommendations and a menu of "ask the staff" badges. Lines the vision step can't read stay on the menu, flagged as unreadable, rather than disappearing.
Training
scripts/gen_dishes.py generated 6,750 rows with
claude -p: 15 cuisines, 150 dishes each, three prompt
variants per dish (name only, name with description, original script
with translation, such as ผัดไทย (Pad Thai)). The three
variants of a dish always share a split, so the model can't memorize a
dish from one variant and get credit on another. Row IDs are hashes, so
training/prep_tagger.py rebuilds the generator's item list
to recover each row's family: 2,250 families, 6,096 train, 351
validation and 303 test rows, leakage check passing.
Training used MLX on Apple Silicon: batch 4, 12 LoRA layers, 800 iterations, learning rate 1.5e-4, sequence length 768, set after measuring a median target of about 204 tokens.
Limitations
Every label in the benchmark was generated by Claude, and Claude is also one of the systems being scored. The spec requires a human spot-check of about 60 dishes stratified across cuisines; it has not been done. Until it is, the Claude rows measure agreement with Claude's own labels, and every figure here is about synthetic data.
The first Claude run was contaminated. claude -p ran inside
the repo, and 8 of the first 100 answers read the training code to find
the schema. Turning tools off made it worse: answers turned into
narrated tool calls. Running the CLI from a scratch directory outside
the repo fixed it, and the rerun showed no contaminated answers.
The LoRA path in the app was broken at first too. It parsed
mlx_lm.generate's raw output, which wraps the JSON in a
banner, and expected a dish_id the model never produces,
so every call fell back to the default. Both are fixed and the 10 tests
pass. This is not a medical device; confirm allergens with restaurant
staff.
Build
mlx_lm.generate; a batched Gemini call is the fallback when no adapter is presentMenu: language, sections, dishes, translations, prices(original_text, language)