Generic running advice is everywhere, and a bare LLM will invent numbers.
Every figure this coach quotes traces back to a study you can open.
Ask a race-strategy question in plain English — "should I try to
negative-split my first marathon?", "how much slower will 80°F make
me?" — and get practical, cited advice from a curated knowledge base.
Several of its cards are distilled from my own published analyses
of 30,000+ Boston finishers, so when it says only 2.5% of runners
negative-split, that figure has a repository behind it.
faithfulness for the grounded coach, against 0.76 for
the same model with retrieval switched off.
1.00
citation validity. Every [n] points at a card that exists
and says what the answer claims it says.
0.95
answer relevance. Grounding did not come at the cost of answering the
question asked.
0.78
context precision — the weakest number, and the one that says where the
work is.
↓ the measurement
Grounding is a claim, so it gets scored
It is easy to build a RAG system and assert that retrieval makes it more
trustworthy. src/eval.py measures it RAGAS-style with an LLM
judge over a gold question set, comparing the grounded coach against the
identical model with no retrieval at all.
The gap is the whole argument: 0.98 against 0.76. The bare
model is not catastrophically wrong, which is exactly what makes it
dangerous — it is plausible three-quarters of the time and confidently
fabricating the rest.
Grounded against bare, on the same questions.
Context precision at 0.78 is the honest weak spot. Retrieval
is pulling in cards that are not always the most relevant available, and the
generation step is compensating. It is reported alongside the good numbers
rather than omitted from the summary.
↓ the knowledge base
Cards with a repository behind them
Most of what makes this coach worth asking is that a handful of its cards are
not summarised from the internet. They are distilled from analyses I ran and
published, each of which you can open and check:
About 67 seconds of the modern elite gain is footwear, not 3–5× the
frequency the press reports.
Plus the general cards
Fuelling, tapering, carb-loading, hydration, pacing and hills, from the
standard literature.
Retrieval is local and free; only generation calls out.
↓ what this cannot tell you
An LLM judge is still an LLM
Faithfulness and relevance here are scored by a model, not by people. LLM
judges are known to favour fluent, well-structured answers and to be
inconsistent near the boundary. The 0.98 should be read as "the grounded
system is substantially more faithful than the bare one," not as a precise
quantity.
The knowledge base is also deliberately small and curated. That is what makes
citation validity reach 1.00, and it also means the coach has a narrow
competence. Ask it something outside the cards and the right behaviour is to
decline, not to improvise.
Several cards are distilled from single-race analyses, so their caveats
travel with them. Boston is a qualifying field on a distinctive course, and
any advice derived from it inherits that.
Finish · what it is built on
Where the numbers come from
Retrieval
sentence-transformers + Chroma, fully local and free
Generation
Claude CLI by default; ANTHROPIC_API_KEY switches to the API
Evaluation
src/eval.py, RAGAS-style with an LLM judge over a gold question set
