Local vision, reconciled, private by construction

Receipt Auditor

Photograph a stack of receipts — get reconciled, categorized spending data with PII redaction built into the schema, plus a locally fine-tuned classifier that beats Claude's cold guess on spend categorization.

By Jeremy Lee

Why

Ask a chatbot to tally your spending from receipt photos and you're trusting it to read every number right and report the sum honestly. Receipt Auditor never lets the model do arithmetic: totals are reconciled in plain code, spend answers are computed by pandas and only phrased by the model — with a guard that catches a dropped or altered number before it ships.

📷

No vision API key

Local OCR (tesseract) + Claude reads each receipt — refuses non-receipt photos, flags low-confidence reads.

🧮

Reconciled, in code

Computed total vs. printed total, checked in plain Python. Mismatches flag for human review, never a silent "fix."

🔒

PII, four layers deep

Schema has no PII fields · raw OCR discarded · regex sweep · session-only on the Space.

🧠

Locally fine-tuned

A LoRA-tuned classifier categorizes spending — runs on a laptop, $0 marginal cost.

The categorizer, honestly benchmarked

Every line item gets a spend category from a Qwen2.5-1.5B model, LoRA fine-tuned locally on 4,799 synthetic receipts across realistic POS merchant styles. Benchmarked against the untrained base model and Claude — all three given the identical blind prompt, no category list in-context:

System	Top-1 acc	Macro-F1	Cost
Base model (no fine-tune)	27.7%	0.333	$0
+ LoRA fine-tune	90.5%	0.893	$0
Claude (teacher, zero-shot)	56.0%	0.506	Max subscription

🏆 The fine-tuned local model's Top-1 accuracy beats Claude's zero-shot Top-1 by 34.5 points — same blind prompt for all three systems, so this is a fair fight, not LoRA getting a hint Claude didn't see.

📋 No real-receipt column yet: the honest synthetic-vs-real tiebreaker needs 10-20 of Jeremy's real receipts, hand-labeled — reported as an open gap, not hidden.

How it works

Upload a batch of receipt photos — local OCR extracts the text, Claude structures it into a receipt and flags anything illegible for you to correct.
Each receipt's total is reconciled against its line items + tax in plain code — mismatches are flagged, never silently fixed.
Each line item is categorized by the locally fine-tuned classifier; PII is swept from every field before anything is cached.
Ask a spending question by voice or text — pandas computes the answer, the model only phrases it, and a guard verifies the number survived phrasing.

🔒 Everything runs locally except one claude -p call per receipt — no metered API, no vision API key. See the full writeup for the honest engineering story, including two real bugs found and fixed by the test suite.

View on GitHub Read the writeup