Tool calls · retrieval · MCP · Python · Claude

A marathon coach that runs the numbers

Ask a race-strategy question in plain English. The coach searches 14 knowledge cards and calls tools: a finish-time and wall-risk model fitted on Boston splits, a heat model fitted on Boston results against race-day weather, and a pace table. It cites each card as [n] and each tool result as [T1], and a check in code flags any number with no source. Updated October 2026.

By Jeremy Lee · 14 marathons, 2:48 PB

Numeric answers within tolerance, 42 frozen targets

50% to 100%

RAG coach to tool coach, same model (claude-sonnet-5-5). The bare model got 33%. Expected values come from the same models the coach calls, so this measures faithful tool use.

Bare / RAG / tool coach, 43 questions

Answers with an unsourced number100% / 65% / 7%
Faithfulness (LLM judge)0.33 / 0.88 / 0.90
Median seconds per answer6.6 / 7.4 / 7.2

What an answer looks like

RunnerI went through halfway in 1:42 and 10 km in 47:30. What finish should I expect, and how likely am I to hit the wall?

Toolpredict_finish({"10": 47.5, "21": 102}) returned a median of 213.54 min, an 80% range of 207.48 to 226.75, wall risk 0.1645.

CoachExpect to finish around 213.5 minutes (about 3:33-3:34) [T1]. The 80% range is 207.48 to 226.75 minutes [T1]. Your chance of hitting the wall (35-40 km pace more than 25% slower than your 10-20 km pace) is about 16% [T1]. How far to trust this: the model was fitted on Boston splits and has no weather input. On the 2024 holdout, its 80% interval at 35 km covered only 74% of runners, so the ranges are somewhat too narrow [T1].

Excerpt from a recorded run, question n01 of the evaluation, 28 October 2026 (claude-sonnet-5-5). Full trace on the write-up page and in eval/runs/run_2026-10.jsonl.

How it works

The model replies with one JSON action per turn: call a tool, or answer.
The loop runs the tool (finish-time model, heat model, pace table, card search or calculator) and feeds the result back as [T1], [T2].
Bad JSON gets one repair retry; after six tool calls the model must answer.
A check in code looks for every number in the answer among the question, the tool results and the cited cards.

The same four domain tools run as an MCP server for Claude Desktop and Claude Code. Hybrid BM25 search and a cross-encoder reranker were tested and made no measurable difference on 14 cards, so retrieval stays dense. The full writeup has the comparison, three recorded runs step by step, and the limitations, including that the heat model did not beat a flat no-weather guess on its own holdout.

Code on GitHub Setup and commands