A pacing plan that has to pass six checks in code before you see it

The pipeline

Photo to verified plan

1. Read the map. src/vision.py runs Tesseract OCR on the photo, then claude -p parses the text into a CourseProfile: distance, hill segments with trend and severity, and a confidence. A photo that isn't a course map gets a refusal.
2. Retrieve. 14 knowledge cards on pacing research (negative-split rarity, hill effort, the heat tax, the wall) plus findings digests, embedded locally with sentence-transformers.
3. Pack the prompt. A priority-based ContextBudgeter fits the course, the runner brief and the retrieved cards into a token budget and reports each section's share. Teacher and student get the same assembled prompt.
4. Generate. Claude, or the local LoRA model, writes a structured PacingPlan: per-km target paces, predicted finish, fade allowance, heat adjustment, cautions, citations.
5. Verify and retry. verify() returns a list of violation sentences. An empty list passes. Otherwise the sentences go into the next prompt, up to two retries.

The benchmark

Base against LoRA against teacher

All three systems ran zero-shot, with no retries and identical prompts, on 70 held-out scenarios split by scenario id (seed 42, 90/5/5). The numbers come from eval/benchmark.md.

The pass rate alone hides where the student fails.

The LoRA student is close to the teacher on structure, coverage, hill plausibility, heat and citations; where it falls down is the two checks that need exact arithmetic across 42 numbers. A typical student plan misses the goal by 430 to 730 seconds, and its fade_allowance_pct often disagrees with its own splits in sign, for example reporting 4.5 while the splits compute −5.1%. In a blind, position-swapped LLM-judge comparison on 60 pairs, the teacher was preferred 93.3% of the time, the student never, with 6.7% ties.

Two plans, one scenario

Where the student's arithmetic goes wrong

The archived outputs keep each plan's cumulative time per km, so the teacher's and the student's plans for the same held-out scenario can be laid over each other. A plan on target reaches the goal time at the finish. Pick a scenario; the default is one where the student is far off.

The fine-tune

Four training attempts on a 16 GB Mac

Training data: 1,458 accepted scenarios across 6 course profiles, 14 goal times, 6 temperatures and 3 experience levels, split into 1,321 train, 67 validation and 70 test, with a leakage check. Training used mlx_lm LoRA on mlx-community/Qwen2.5-3B-Instruct-4bit, 16 layers, batch size 1, gradient accumulation 4, learning rate 5e-5, gradient checkpointing.

Sequence length

The planned 1,536 tokens was too short. The longest training example measured 4,197 tokens with the Qwen tokenizer, so the budget became 4,608, asserted before training.

Silent failure

Without gradient checkpointing, memory overflowed and training printed "Train loss 0.000, Trained Tokens 0" every report instead of crashing. A 10-iteration repro confirmed it; checkpointing brought peak memory to about 13 GB.

Divergence

Loss fell from 1.666 to 0.256 over 280 iterations, then climbed to 1.651 within three reports. Halving the learning rate and adding accumulation delayed it but didn't stop it.

Clipping

train_clipped.py reuses mlx_lm.lora's loop and adds clip_grad_norm, since the CLI has no clipping flag. It diverged again; gradient norms stayed at 0.18 to 0.35, under the 1.0 threshold, so clipping never engaged.

Decision

Ship the best checkpoint, about 400 clean iterations at validation loss 0.226, and leave the next hypothesis (a constant learning rate with no decay) for later.