The risk with receipts is not bad OCR. It is a confident wrong total.

So the model is never allowed to produce a number.

Photograph a stack of receipts and get reconciled, categorized spending you can query out loud. Totals are checked in plain code against the printed total. Spending answers are computed by pandas and only phrased by the model, with a guard that rejects the phrasing if a number changes. And the categorizer is a 1.5B model fine-tuned on this laptop that beats Claude's zero-shot accuracy by 34.5 points.

Read the writeup or the code

90.5%

Top-1 category accuracy from the fine-tuned model, against Claude's zero-shot 56.0% on the identical blind prompt.

0.893

macro-F1, against 0.506. It holds up across all twelve categories, not just the common ones.

4 layers

of PII defence, starting with a schema that has nowhere to put a card number in the first place.

numbers originating from the model in a spend answer. The guard makes a hallucinated figure structurally impossible.

↓ the headline result

34.5 points over the frontier model

90.5%

against Claude zero-shot at 56.0%, on the same twelve-category blind prompt.

All three systems saw the identical prompt with no category list supplied, matching the training format exactly. The fine-tuned model was not handed a hint the others lacked — this is a fair comparison of what each brings on its own.

System	Top-1 acc	Macro-F1	Latency	Cost/1k
Base Qwen2.5-1.5B (no fine-tune)	27.7%	0.333	5.0s	$0
+ LoRA (this project)	90.5%	0.893	4.5s	$0
Claude (teacher, zero-shot)	56.0%	0.506	17.7s	Max sub

It is also four times faster and costs nothing per call. The training set is 4,799 synthetic line items written in realistic point-of-sale styles — SBUX #4821, WM SUPERCENTER #2847, TST* JOE'S PIZZA — and split by merchant, so no merchant appears on both sides of the split.

Confusion matrix across the twelve spend categories showing a strong diagonal with limited confusion between neighbouring categories such as dining and coffee. — Twelve categories. The residual confusion is mostly dining against coffee, which is fair.

↓ the design idea

The number never originates from the model

Ask "how much did I spend on dining in March" and the arithmetic is done by analysis.answer_query in pandas. The LLM receives the computed figures and is asked only to turn them into a sentence. Then reconcile.assert_numbers_present checks that every number it was given still appears in what it wrote — and if one was dropped or altered, the caller falls back to a template sentence.

This is the difference between asking a model not to hallucinate and building a path where it cannot. There is no code route by which a number the model invented reaches the screen.

A hallucinated number is structurally impossible here, because the number never originated from the model.

From the repository's own README

↓ reconciliation

A mismatch is flagged, never quietly fixed

For each receipt, computed = Σ(line items) + tax is compared against the printed total within a small tolerance. When they disagree the receipt is rendered flagged for one-tap human review rather than silently adjusted to agree.

Silently correcting a mismatch is the tempting behaviour and the wrong one: it destroys exactly the signal that tells you the OCR misread a line.

Receipt Auditor pipeline diagram: receipt photos into local OCR, structured extraction, PII sweep, code-level reconciliation, fine-tuned categorisation, then a pandas analysis layer with an LLM used only for phrasing. — The pipeline. Note where the LLM sits, and where it does not.

↓ privacy

Four layers, the first of which is doing most of the work

1 · The schema has no room

The Receipt and LineItem models simply have no fields for cardholder name, card number, address or loyalty id. The vision model is never asked for them.

2 · Raw OCR is discarded

Only the validated Receipt object survives extraction. The full text, with whatever was on it, does not persist.

3 · A regex sweep anyway

privacy.sweep_receipt redacts anything PII-shaped that got into a merchant or item string, and reports how many redactions it made.

4 · Session-only when hosted

On a hosted Space everything stays in memory. privacy.is_space_environment() is the single source of truth both paths check before persisting anything.

↓ the gap, reported not hidden

These numbers are synthetic-only

There is currently no real-receipt held-out column in the benchmark. The specification calls for 10–20 real receipts, photographed and hand-labelled, as the honest synthetic-versus-real tiebreaker, and that has not been supplied yet.

So 90.5% is a real number on a real held-out split with no merchant leakage, and it is a number about synthetic line items. A model trained and tested on the same generator can share that generator's blind spots. Until the real slice runs, that caveat travels with the headline.

Training loss curve for the categorizer LoRA fine-tune, falling steeply and then flattening. — The training run for the categorizer.

Finish · what it is built on

Where the numbers come from

Base model: Qwen2.5-1.5B, LoRA fine-tuned locally, called through mlx_lm.generate
Fallback: A plain "other" category if the adapter is absent — inspectable, not a black box
Training data: 4,799 synthetic line items, split by merchant so none straddles train/test
Categories: Twelve, from groceries and dining through to subscriptions and travel
OCR: tesseract locally, then claude -p for extraction from the OCR'd text
Analysis: pandas. The LLM phrases; it does not compute
Benchmark: training/bench_categorizer.py