A marathon coach that runs the numbers before it answers

Tool-call trace

One question, call by call

A recorded run of question n01 from 4 November 2026, saved as eval/trace_n01.json by scripts/measure_cost.py. Each model call returns one JSON action; the loop runs the tool and feeds the result back as [T1]. Times and costs are per call, as claude -p reported them.

“I went through halfway in 1:42 and 10 km in 47:30. What finish should I expect, and how likely am I to hit the wall?”

1. Call 12.9 s
   $0.0007

   Model reply (46 tokens out)

   {"action": "tool", "tool": "predict_finish", "args": {"splits": {"10": 47.5, "21": 102}}}

   Loop runs predict_finish as [T1] in 53 ms, returns

   {
    "median_min": 213.54,
    "lo80_min": 207.48,
    "hi80_min": 226.75,
    "lo95_min": 204.1,
    "hi95_min": 239.48,
    "wall_risk": 0.1645,
    "pace_type": "Steady fade",
    "n_train": 176059,
    "model_version": "2026-10-28.1",
    "caveat": "Fitted on Boston splits; has no weather input. On the 2024 holdout its 80% interval at 35 km covered 74% of runners, so late-race intervals run narrow."
   }

2. Call 27.0 s
   $0.0072

   Model reply (693 tokens out)

   {"action": "final", "answer": "Expect to finish in about 213.5 minutes (median), with a 80% range of 207.5 to 226.8 minutes and a 95% range of 204.1 to 239.5 minutes [T1]. The model ..."}

3. Answer10.0 s total

   Final answer, as the runner sees it

   Expect to finish in about 213.5 minutes (median), with a 80% range of 207.5 to 226.8 minutes and a 95% range of 204.1 to 239.5 minutes [T1]. The model reads your pacing as a "Steady fade" [T1]. Your chance of hitting the wall, meaning a 35-40 km pace more than 25% slower than your 10-20 km pace, is about 16% (0.1645) [T1].
   
   How far to trust this: the model was fitted on Boston splits and has no weather input. On the 2024 holdout, its 80% interval at 35 km covered only 74% of runners, so the late-race ranges are narrower than they should be [T1]. Treat the upper end (226.8 minutes and beyond) as quite possible, especially if the day is warm or windy.
   
   What to do: (1) Hold your current effort through 30 km and don't chase time in the middle miles, since that is what drives wall risk. (2) If it is warm or windy, check the weather effect and slow your target pace to match, because this prediction ignores it.
   
   FINAL: finish_min=213.54; wall_risk_pct=16.45

Expected values for n01 from src/gold.py: finish 213.54 min (tolerance 4.8), wall risk 16.45% (tolerance 5). The answer matches both. The second call cost more because it writes the whole answer; the first is mostly a cached system prompt.

More recorded runs from the eval

Three of the coach's answers from the 28 October evaluation run, exactly as recorded: each tool it called, with the arguments it chose, what came back, and the final answer. Long passages are cut short here; the full records are in the repo.

Cost and latency

Under two cents a question

The October run timed every answer but did not record cost, so on 4 November scripts/measure_cost.py reran a fixed sample, every fourth question of the gold set (11 questions, 7 numeric and 4 strategy), through all three systems one answer at a time, without the judge. Cost is the total_cost_usd that claude -p --output-format json reports: the API-equivalent price, since the runs used a Claude subscription. It includes the CLI's automatic prompt caching, which is why the coach's repeated system prompt is cheap. Retrieval runs locally on an Apple M4.

Calls are model calls per answer. Seq. is one answer at a time; Run is the October eval run. Cost and Seq. columns: n = 11 per system, eval/cost_latency.json, per call in eval/cost_latency_calls.csv. Eval-run columns: n = 43 per system with four answers running at once, 28 October, from eval/runs/run_2026-10.jsonl via eval/error_analysis.json. Judge calls are excluded throughout.

The coach makes more calls but each is short: a JSON action, then an answer written from compact tool results, where the RAG call reads five passages. So it costs about what RAG costs and finishes in about the same time. Its p95 is below RAG's in both runs. The RAG coach's slowest answer in the eval run took 128 s, which sets its p95 there.

Error analysis

Where the baselines miss

The tool coach got every scored value, so the failures worth reading are the other two systems'. Grouped by the kind of question, with one recorded answer from each group. Counts are from eval/error_analysis.json, which scripts/error_analysis.py builds from the saved run without calling a model.

1. Heat: the wrong rule of thumb. The bare model came in low on all nine heat values. For a 2:55 runner at 80°F it used a temperature-plus-dew-point table and said 5.3 minutes against the model's 19.2: Combined temp + dew point: 80 + 65 = 145 ... about a 3% slowdown. The RAG coach applied the July card's minute per degree from a 50°F base and mostly came in high: 31 minutes on the same question.
2. Wall risk: a population rate, nudged. Neither baseline can compute a per-runner probability, so both started from a card figure or a hunch. At 35 km in 3:05 the runner had been speeding up and the model put wall risk at 0%; the RAG coach said 30%: In the Boston data, 27% of finishers slowed by more than 25% in the closing kilometers [1]. That is my baseline. The bare model's guesses ran from 25% to 65%.
3. Heat plus pace: the first error carries through. Both c-questions chain a heat or finish estimate into a pace, so a miss on the first value costs the second. For 3:20 at 76°F the RAG coach reset the goal to 226 minutes by the minute-per-degree rule; the model gives 216.1, and the pace follows.
4. Finish times: mostly close. Doubling the half split is a fair guess, so both baselines landed seven of ten. The RAG coach missed n04 by 17 minutes: the runner had been speeding up, and it applied the card's median 15% late slowdown anyway: Slowing 15% from there gives about 6:05/km, which puts you at about 229 min.
5. Pace targets: the fade left out. Asked for a first-half pace for 4:30 with the typical fade, the bare model assumed a 3% fade and said 6:18/km; the RAG coach advised against planning a fade at all and gave 6:24/km. The pace table, using the fade band from the negative-split card, says 5:46/km.
6. Card arithmetic: retrieval fixes it. The RAG coach got all eight. The bare model used numbers from memory instead of the cards: for carbs over four hours it used 60 to 90 g an hour and said 240 to 360 g, where the card's 30 to 60 g gives 120 to 240.

The tool coach's own failures

Outside the 42 values, three things went wrong. Asked "how much slower will race-day heat around 80°F make me?" (s02), it called no tool and asked for the runner's finish time instead. The judge scored that 0.30 for relevance and 1.00 for faithfulness, since an answer with no claims has nothing unsupported in it. On c01 it gave an adjusted finish range of "205 to 219 minutes" that no tool returned; it added the heat model's slowdown range (4.53 to 19.33) to 200 in its head, and the number check's 2.5% rounding allowance let it through. And 3 of 43 answers carried a number with no source: once it turned 17.7% into "a 1-in-6 chance", and twice it gave "30 km" advice without citing the card that says so.

How the loop works

Plan, call, read, repeat

The Claude CLI takes text in and gives text out, with no function-calling interface, so src/agent.py runs the loop itself.

1. Ask for one action. The model sees the question, the tool list and every earlier step, and must reply with one JSON object: a tool call or a final answer.
2. Run the tool. predict_finish and heat_effect are the models from the pacing and heat projects, vendored with their source commits. pace_table does even or fade splits, with fade sizes read from the negative-split card. retrieve_cards searches the cards. A calculator covers sums so the model does not do arithmetic in its head.
3. Feed the result back. Passages are numbered [1], [2] and tool results [T1], [T2], so the answer can cite both.
4. Repair or stop. A reply that is not valid JSON gets one retry with the parse error quoted back. After six tool calls the model must answer with what it has.
5. Check the numbers. A deterministic post-check pulls every number out of the answer and looks for it in the question, the tool results and the passages the answer cites, allowing for rounding and for clock times written as minutes or seconds. Anything unmatched is flagged.

The model results carry their own caveats, copied from the holdout checks in the source projects, and the coach is told to pass them on. The pace model has no weather input, and on its 2024 holdout its 80% interval at 35 km covered 74% of runners. The heat model did not beat a flat no-weather guess on its 2015–2019 holdout (2.26% error against 2.15%), and it ignores dew point, which did not beat dry-bulb temperature in cross-validation.

Retrieval

Hybrid search and reranking, measured: no gain on 14 cards

The July page guessed that BM25 plus a cross-encoder would help, as it did for the wine sommelier RAG. On the frozen questions, scored by which cards each question needs, it did not. Dense search already finds a relevant card in its top three for nearly every question, and no mode's mean reciprocal rank differs from dense by more than its bootstrap noise. The coach keeps dense search, which is also the fastest of the embedding-based modes.

From eval/retrieval_compare.json (src/retrieval_eval.py): 43 questions, 46 chunks, card-level relevance labels written with the gold set. Reranking uses cross-encoder/ms-marco-MiniLM-L-6-v2 on a fused shortlist of 20.

MCP server

The same tools, from Claude Desktop or Claude Code

src/mcp_server.py exposes retrieve_cards, predict_finish, heat_effect and pace_table over stdio with the official mcp Python SDK. In a recorded round trip (eval/mcp_roundtrip.json), Claude Code was asked for a 3:40 even pace, called pace_table and quoted 8:23 per mile and a 1:50:00 half from the result.

claude mcp add marathon-coach -- \
  ~/venvs/marathon-strategy-rag/bin/python \
  /path/to/marathon-strategy-rag/src/mcp_server.py

Claude Code. Use the Python that has the repo's requirements installed.

{
  "mcpServers": {
    "marathon-coach": {
      "command": "/Users/you/venvs/marathon-strategy-rag/bin/python",
      "args": ["/path/to/marathon-strategy-rag/src/mcp_server.py"]
    }
  }
}

Claude Desktop, in claude_desktop_config.json.

The knowledge base

Fourteen cards, six from my own studies

The corpus ships in the repo as markdown, one card per topic, each with a > Source: line. Eight cover established training principles: carbohydrate fueling, carb-loading, tapering, hydration and sodium, even pacing, the long run, race week, and hills. Six are distilled from analyses I ran and published:

Negative splits

2.5% of 31,912 Boston 2014 finishers ran a negative split; about 94% slowed, median +14%.

The wall

Pace inflects at about 30 km; 27% of finishers slowed by more than 25% late in the race.

Heat

About one minute slower per °F of race-day heat at Boston (the July card; the heat tool is the newer model).

Courses

Boston +101 s and New York +78 s against Berlin, from a simulated field with planted penalties. The card labels them as simulated; they are not measurements of the courses.

Super-shoes

About 67 seconds of the modern elite gain is footwear.

BQ standards

Across 22 age and sex brackets there is no average gender gap, but women's brackets vary three to four times more in difficulty.

July 2026, superseded

The first evaluation: grounded against bare

In July the coach was scored on seven strategy questions: faithfulness 0.98 grounded against 0.76 bare, answer relevance 0.95, context precision 0.78, citation validity 1.00. Those seven questions are s01 to s07 in the new set.

2026-10-28: superseded by the three-system run above, which uses a different judge prompt (one answer at a time, judged against that system's own context) and a pinned model, so the two sets of scores are not directly comparable. The July figures stay in eval/results.csv. Also 2026-10-28: the simulated-data label was added to the course card after the July evaluation; the October run used the labeled card.