A Python command-line tool, rebuilt in October 2026
On four sources, the LLM episodes covered 7.4 of 12 frozen key points on average, against 4.4 for rule mode reading the same number of words. A second checker kept out of the pipeline (Opus) found 8 of 172 claim lines not fully supported before the repair step and 8 of 165 after it. The repair did not lower the rate.
LLM mode covers more of every source
Key points covered, of 12
The repair step left the unsupported share flat
Claim lines not fully supported, by Opus
From eval/results/summary.json (eval run of 2026-10-29, one run per source and
mode, six-minute target). Coverage was graded by Opus against 12 key points per source that were written
and committed before any episode existed.
Focus or hover a row for its counts.
db27eca before any generation.claude -p calls (Sonnet). One fresh Great Stink episode, n = 1.Podcastify reads an EPUB or article end to end, pulls out key points that cite their source sentences, plans an episode of a set length, and has Claude write a conversation between a host and a co-host in which every factual line carries the ids of the sentences it rests on. A second model call then checks each line against those sentences; lines that fail are rewritten once or dropped. Free edge-tts voices render the result as an MP3 with a chapter per section.
Select a line to see the source it cites and how it was checked.Select a time to play from there.
The tool's real output for the whole of Alice's Adventures in Wonderland
(Project Gutenberg eBook #11, public domain; 143,485 characters, 1,431 sentences), LLM mode with a
six-minute target, Claude Sonnet through claude -p, default voices, rendered on
2026-10-29 with edge-tts 7.3.1. The file is 5 minutes 55 seconds, re-encoded at 40 kbps for this
page; the chapter markers are the outline's four sections. Struck-out lines were cut by the check
and are not in the audio. Character offsets refer to the cleaned text that
eval/fetch_sources.py writes.
frozen key points covered, on average over four sources, by the LLM episodes. Rule mode, reading the same number of words from the start, covered 4.4.
claim lines failed the pipeline's own check (6.0%), all as "partly". Three were rewritten and kept; six were dropped.
claim lines that a second checker (Opus) marked not fully supported, before the repair step (of 172) and after it (of 165). The repair fixed what the first checker flagged, and the two checkers mostly flag different lines.
randomly drawn claim lines where my own reading against the source agreed with the pipeline's checker on pass or fail. Only one of the 20 was a fail by my reading.
How an episode is made
The old LLM mode sent the first 12,000 characters of a source to the Anthropic API and asked for a conversation. For Alice that is the first chapter and a half. The new pipeline reads everything, in four steps, each a separate model call:
-o book.m4b.
All LLM calls go through the local claude -p CLI on a Claude subscription, with built-in tools
off and no user settings loaded (--tools "" --setting-sources ""), so the model sees the
prompt and nothing on disk. With ANTHROPIC_API_KEY set, the same prompts go to the Messages
API instead. Rule mode, which needs no model at all, is still the default.
The grounding check
The local check embeds the line and its cited sentences with sentence-transformers all-MiniLM-L6-v2 and flags a cosine similarity below 0.40 (a threshold set before any eval run), or any figure written in digits that does not appear in the cited text. It flagged 3 of 151 claim lines and agreed with the LLM verdict on 82% to 100% of claim lines per source, mostly because both call nearly everything fine. It is a cheap screen for a line that cites the wrong place, which the script writer rarely did.
The LLM check is a separate claude -p call that reads each line with its cited sentences
(and the line before it, since a short answer leans on its question) and returns supported, partly,
unsupported or no claim. Lines the writer marked as non-claims are checked too, against their two
nearest outline sentences, so a factual line cannot skip the check by being labeled a reaction. Seven
lines across the four episodes were claims the writer had not marked.
What failed was rarely an invented fact. The nine failures were a question that presupposed something from outside the cited sentences ("It sits right on the river"), a quote attributed to a source the cited sentence does not name, or an inference stated as fact ("the consequence is prison"). The check also made one plain mistake, visible in the episode above and set out under "Where it goes wrong" below.
The eval
Each source was run three ways at a six-minute target: rule mode reading from the start until it had
used as many words as the LLM episode, the LLM pipeline with the grounding pass, and the same LLM
draft before the grounding pass. Before any episode existed, Opus wrote 12 key points for each source
from the full text; I checked the figures in them against the sources and committed them
(eval/keypoints/). Opus then marked each point as covered (1), partly covered (0.5) or
missing in each script.
| Source | Mode | Key points | Not fully supported | Words | Length | Wpm |
|---|
"Not fully supported" is the share of claim lines that Opus, as a second checker kept out of the pipeline, rated partly or unsupported against their cited sentences. Rule-mode narration is the source read verbatim, so it is supported by construction and was not sent. Length and words per minute come from rendering every script with edge-tts and measuring it with ffprobe.
Opus compared each pair of scripts for each source, labeled only "Script 1" and "Script 2", once in each order. A mode wins a source only if it is preferred in both orders.
| Pair | First wins | Second wins | No consistent winner |
|---|
Counts are sources, out of four. The judge preferred the unchecked draft over the checked episode in three of four sources, in both orders; on the fourth the two scripts were identical because nothing failed. Dropping a line can leave a question without its answer, and the judge noticed.
Checking the checker
eval/hand_check.py drew 20 claim lines at random (seed 19) from the four drafts, before any
repair, and printed each with the sentences it cites and no verdict. I read all 20 and recorded my own
verdict in eval/hand_check.json before scoring. My reading agreed with the pipeline's
checker on pass or fail for 18 of 20 lines, and with Opus on 19. The two
disagreements with the pipeline went both ways: it called a correct paraphrase of the end of
Alice "partly", and passed "That points at the water, not the air", an inference the cited
sentences about John Snow's pump do not state.
Twenty lines with one real failure among them says the checker passes good lines; it says little about how many bad lines it misses. That would need a sample built around failures.
Error analysis
Four kinds of failure show up in the saved eval outputs. Each example below is copied from
eval/results/; eval/error_examples.py pulls them into
error_examples.json without calling a model.
1 correct line dropped, 1 rewritten, 1 question rated unsupported, all in the last paragraph
Carroll's last paragraph has Alice's sister picture "this same little sister of hers" grown up. The draft line got it right:
She sits with closed eyes and half believes herself in Wonderland. She pictures Alice grown, keeping the simple and loving heart of her childhood.Draft line, cites S1430 and S1431
The pipeline's checker (Sonnet) rated it "partly": "the sister pictures her own self grown, not Alice." The rewrite swapped in "her little sister", the check failed it again, and the line was dropped. It made the same misreading of the next line, which was rewritten. In the checked episode Opus then rated the bridging question, "What would grown-up Alice do?", unsupported.
6 of 12 frozen key points missed in Origin, ch. III; the outline kept 13 of 53 mapped points
Every one of the six missed points rests on a worked example or a count: the elephant calculation, feral cattle in South America and Australia, the Fulmar petrel, the marked seedlings, game and vermin, the Staffordshire heath and the cats and clover. The map step had found all of them, at importance 2 of 3, for example:
Eggs and young generally suffer most, and seedlings are destroyed in vast numbers by crowding and enemies; of 357 marked weed seedlings, 295 were destroyed chiefly by slugs and insects.Map-step key point, S80 to S82, not chosen for the outline
The reduce prompt asks for about two points per minute and the model picked the general claims over the evidence. A coverage-aware reduce prompt is untested.
4 of 4 drafts over the 930-word target; 2 of 4 checked episodes over six minutes
| Source | Draft words | Over target | Checked episode |
|---|---|---|---|
| Great Stink | 1,279 | +38% | 7.9 min |
| Civil Disobedience | 957 | +3% | 5.6 min |
| Origin, ch. III | 1,002 | +8% | 6.3 min |
| Alice | 1,022 | +10% | 5.9 min |
The script prompt asks for six minutes at 155 words a minute, 930 words. The writer goes over every time, and the voices run 154 to 173 words a minute, so the two errors only sometimes cancel. The fresh Great Stink run for the cost measurement below came out at 7.8 minutes again.
Opus rated 22 question lines as claims in the drafts; 4 of its 8 not-fully-supported lines are questions
A question can presuppose a fact, so both checkers grade questions. The pipeline's checker rated 9 of the 88 question lines in the checked episodes, 8 of them "supported", for example:
And then she wakes up?Alice, rated supported, cites S1423
Opus counted 22 questions among its 172 draft claim lines and marked 4 of them partly or unsupported, among them "So Parliament finally acts. How?" and "What did he make of the State after his own night in jail?". Half of the 8 lines behind the headline rate are questions like these, so the rate says more about how a checker reads a question than about invented facts.
Cost and time
The eval did not record cost or time, and its disk cache means a rerun makes no model calls. So
eval/measure_cost.py made one fresh episode from the shortest source, the Great Stink article
(26,640 characters), with an empty cache, on 2026-11-04, on an Apple M4. Cost is what
claude -p --output-format json reports as total_cost_usd: the API-equivalent price, since
the run used a Claude subscription. This is one episode (n = 1), so there is no spread to report; a
whole book maps more chunks and costs more.
| Stage | Model calls | Cost | Wall time |
|---|---|---|---|
| Map (3 chunks) | 3 | $0.084 | 10.6 s |
| Reduce | 1 | $0.017 | 4.2 s |
| Script | 1 | $0.100 | 37.9 s |
| Check, repair, recheck | 6 | $0.145 | 57.9 s |
| Audio (edge-tts, free) | 0 | $0 | 112.1 s |
| Episode | 11 | $0.345 | 222.8 s |
From eval/results/cost_episode.json and cost_calls.csv (one row per call). The
episode had 36 claim lines; the checker rated 4 "partly", and all 4 were rewritten and kept. It ran 7.8 minutes.
Limitations
Four sources, one run each, one six-minute target. The coverage gap between rule mode and LLM mode is large and consistent; the grounding numbers are small counts (8 lines of 172), and a different run would move them by a few lines. The key points, the coverage grading, the second check and the listenability judge are all Claude models, and the scripts are Claude's too, so the judge may favor writing like its own.
The grounding pass, as built, did not lower the second checker's not-supported rate. It removes what one model flags, which is a narrow filter: Sonnet and Opus mostly disagree about which lines are partly supported, and both are sometimes wrong. What the pass does give is a record: every line in the transcript carries its source offsets and verdict, so a reader can check any line in a few seconds.
A "supported" verdict means the cited sentences back the line, not that the line is a fair summary of the work. The outline step decides what an episode leaves out, and nothing checks that. Six minutes of Alice skip the pool of tears, the Caucus-race and the croquet game. Episodes ran 5.6 to 7.9 minutes against the six-minute target. edge-tts is an unofficial Microsoft endpoint; 7.2.8 returned HTTP 403 on 2026-10-26 and 7.3.1 works.
Built with
.epub, .txt, .md or .html, whole books includedclaude -p (Sonnet by default, --model to change), or the Anthropic Messages API over urllib when ANTHROPIC_API_KEY is set. Replies are cached on disk by prompt hash