{ "measurement": "What book-to-skill costs to have installed, which of its own published figures reproduce, and which of this site's two answers about description cost is the right one", "claude_code_version": "2.1.251 (Claude Code)", "models_recorded_on_every_run": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "subject": { "repo": "virgiliojr94/book-to-skill", "commit": "9c207f870adebe20ade4f7d2f11bc3d759c2fd88", "version_note": "pyproject.toml declares 1.4.0, but this commit is unreleased main: the CHANGELOG dates [1.4.0] to 2026-08-10 and this commit is from 2026-08-28, carrying three merged PRs that appear in no release. Read every finding here as being about this sha, not about the tag.", "stars_at_trial": 27490, "stars_source": "the GitHub API repos endpoint on the trial date, read by hand rather than computed by the harness, and the only unmeasured number here", "license": "MIT", "trial_date": "2026-08-31", "claims_adjudicated_from": "docs/performance.md, README.md and CHANGELOG.md at that commit", "extractor": "pdftotext (poppler) 26.04.0" }, "why": "book-to-skill turns a book into an agent skill, and publishes an unusually specific set of first-party numbers for a project of its size: a per-book extraction table, a 24x to 51x context saving, and a per-file token budget. Two of its table rows name books that can be fetched and pinned, so those rows are reproducible end to end. Its own skill description is 398 characters, which is past the 300-character bound on the description rate /cost/ publishes.", "results": { "what_it_costs_to_have_installed": { "tokens_per_session": 159, "rounds": 6, "description_chars": 398, "tokens_per_description_char": 0.3995, "skill_md_bytes": 38205, "files_installed": 104, "bytes_installed": 1703001, "megabytes_installed": 1.7, "modal_floor_context": 17060, "modal_installed_context": 17219 }, "the_noise_in_this_run": { "readings_total": 24, "readings_off_the_modal_value": 0, "excursions_seen": [], "modal_treatment_context": 17219, "modal_floor_context": 17060, "reading": "24 readings across two modal values, 17219 for the three treatment arms and 17060 for the floor. 0 readings sat off their modal value. An earlier run of this same harness on this same fixture produced two, each exactly 897 low, one of them a floor and one a treatment arm, which is why the assembler asserts that any excursion equals that component and why the design is paired: read unpaired, those two rounds report a 738-token saving and a 1,056-token cost. Nothing here is a spread to average over." }, "the_body_and_the_rest_of_the_repository_are_free": { "full_repository_delta": 159, "skill_md_alone_delta": 159, "frontmatter_only_delta": 159, "files_full": 104, "files_skill_md_only": 1, "files_frontmatter_only": 1, "bytes_full": 1703001, "bytes_skill_md_only": 38205, "bytes_frontmatter_only": 442, "reading": "Cloning the repository into the skills directory, which is one of the two installs the README documents, leaves 104 files and 1.7 MB of working tree, including a 38,205 byte SKILL.md. Three arms cost the same to the token: the whole tree, SKILL.md alone, and SKILL.md truncated to its YAML frontmatter (442 bytes). So the body is free and so is everything around it. THE CONDITION IS THAT THE SKILL IS NEVER INVOKED: the prompt is trivial and nothing calls it. Invoking it has to read the body, and this corpus does not price that." }, "which_of_this_sites_own_answers_is_right": { "measured_tokens": 159, "description_chars": 398, "measured_tokens_per_char": 0.3995, "answer_a_the_cost_page_model": { "model": "39 tokens at a 90-character description, plus 0.276 per character", "published_valid_range_chars": [ 30, 300 ], "predicted_on_the_marginal_anchor": 124.0, "predicted_on_the_n1_anchor": 134.0, "n1_anchor_source": "skills-cost-2-1-251 count_curve['1'], mode 49 across 11 rounds with zero spread, same release, same 90-character description, same empty-project design. 39 is the MARGINAL rate below 200 skills and carries no first-skill overhead, so 49 is the right anchor for a trial at n=1.", "shortfall_against_the_n1_anchor": 25.0, "verdict": "UNDER. Extended past its own 300-character bound to 398 characters it predicts 134.0 against a measured 159." }, "answer_b_the_measured_english_band": { "source": "skillpack-language-2-1-247, five real English skill descriptions, each installed ALONE in an empty project against a paired floor, guarded mode over four rounds, the same trivial prompt. That is this trial's design, on a different release.", "per_char_range": [ 0.3629, 0.4819 ], "measured_over_chars": [ 198, 485 ], "predicted_range_at_this_length": [ 144.4, 191.8 ], "measured": 159, "measured_inside": true, "verdict": "HOLDS. 398 characters predicts 144.4 to 191.8 tokens and the measurement is 159. The per-character rate, 0.3995, sits inside the published 0.3629 to 0.4819 band." }, "reading": "This site publishes two answers to 'what does a long description cost' and only one of them is right out here. The /cost/ model is printed with a 30-to-300-character bound and a paragraph beneath it that invites extending the rate anyway; extended, it is 25.0 tokens low. The band measured on five real English descriptions at n=1 brackets this measurement without adjustment. THE POINT IS NOT THAT A NEW RATE WAS FOUND HERE, because one point cannot fit a rate. It is that the answer already existed in a sister corpus, agrees with this measurement, and is not on the page where a reader would look for it.", "the_cost_page_prediction_mixes_two_arms": "The 0.276 rate was fitted on the DESC_LEN arm at 40 skills while the 49-token anchor comes from the COUNT arm at n=1, so combining them assumes the per-character rate does not depend on how many skills are listed. That assumption is untested. The English band needs no such splice, which is a second reason to prefer it." }, "extraction_figures_checked_against_the_doc": { "per_book": [ { "book": "think-python-2", "source_url": "https://greenteapress.com/thinkpython2/thinkpython2.pdf", "pdf_md5": "d4acd947dd464f477e332eac4bd000cc", "pages_measured": 244, "tokens_claimed_in_doc": "119K", "raw_pdftotext_cl100k": 119237, "raw_rounds_to": "119K", "raw_matches_doc": true, "after_the_projects_cleanup_cl100k": 118489, "cleanup_removes_tokens": 748, "cleanup_removes_pct": 0.63 }, { "book": "pro-git", "source_url": "https://github.com/progit/progit2/releases/latest/download/progit.pdf", "pdf_md5": "3eac9b78618d8c3a9aa115480d5ccd33", "pages_measured": 501, "tokens_claimed_in_doc": "229K", "raw_pdftotext_cl100k": 229371, "raw_rounds_to": "229K", "raw_matches_doc": true, "after_the_projects_cleanup_cl100k": 227139, "cleanup_removes_tokens": 2232, "cleanup_removes_pct": 0.97 } ], "reading": "THE DOC'S TOKEN FIGURES REPRODUCE. Counting raw pdftotext -layout output gives 119,237 against a claimed 119K, which is 27 tokens from the exact 119,264 the discovery table prints, and 229,371 against a claimed 229K. What the current CLI reports is 0.6 to 1.0 percent lower, and the cause is first-party and documented: clean_pdftotext() dehyphenates and strips running headers and page numbers, and the CHANGELOG dates that change to [1.3.0] with the note 'Fewer tokens', after the table was measured. An earlier version of this corpus reported both books as failing to reproduce and blamed an uncontrolled pdftotext version. That was wrong and it was the least charitable of the available readings.", "page_counts_are_an_entry_GATE_not_a_result": "Both books' page counts equal the doc's. That is asserted before extraction so that a different edition stops the run, which means no published corpus could ever contain a mismatch. It is a control, not a finding, and it was previously reported as one." }, "the_on_screen_token_figure_is_not_the_doc_figure": { "per_book": [ { "book": "think-python-2", "metadata_estimated_tokens": 109649, "cli_prints": "~109K", "tiktoken_cl100k": 118489, "shortfall_tokens": 8840, "shortfall_pct": 7.5 }, { "book": "pro-git", "metadata_estimated_tokens": 198310, "cli_prints": "~198K", "tiktoken_cl100k": 227139, "shortfall_tokens": 28829, "shortfall_pct": 12.7 } ], "reading": "The CLI prints a rounded figure computed as words divided by 0.75 (config.WORDS_PER_TOKEN), and metadata.json carries the same number unrounded under estimated_tokens. docs/performance.md states its table is tiktoken cl100k_base. Both are true and they are different quantities, so a reader budgeting context from the number on screen under-budgets by 7.5 percent on one book and 12.7 on the other, and the two differ, so this is a gap and not a rate. THE PROJECT DOES NOT CLAIM OTHERWISE: the constant is commented approximate, the field is named estimated_tokens and the CLI prints a tilde. This is a note about reading the output, not a defect." }, "the_chapter_count_on_screen": { "per_book": [ { "book": "think-python-2", "reported_by_tool": 28, "method_reported": "numeric", "claimed_in_doc": 19, "corroborated_any_line_rule": 19, "corroborated_first_line_rule": 18, "uncorroborated_numbers": [ 50, 62, 82, 100, 136, 176, 197, 216, 222 ], "matching_lines": 40, "matching_lines_from_the_plain_branch": 19, "matching_lines_from_other_branches": 21 }, { "book": "pro-git", "reported_by_tool": 5, "method_reported": "numeric", "claimed_in_doc": null, "corroborated_any_line_rule": 0, "corroborated_first_line_rule": 0, "uncorroborated_numbers": [ 2, 22, 78, 79, 80 ], "matching_lines": 12, "matching_lines_from_the_plain_branch": 12, "matching_lines_from_other_branches": 0 } ], "the_two_rules_disagree_and_both_are_published": "A number counts as corroborated under the ANY-LINE rule if any of its matching lines is a bare chapter heading and nothing else, and under the stricter FIRST-LINE rule only if the first one is. On Think Python 2 they give 19 and 18: the number 16 has a running header for page 16 ('16 Chapter 2. Variables, expressions and statements') before its real 'Chapter 16' heading. On Pro Git both give zero. An earlier version of this corpus claimed the partition was invariant under the stricter rule; it is not, and the counter-example is checkable in the hits array below.", "mechanism": "The first branch of _match_chapter_number treats any line matching ^([1-9]\\d{0,2})\\s{2,}\\S as a chapter heading: a one to three digit number, two or more whitespace characters, then one non-whitespace character. It is guarded by an earlier `if len(s) > 80: return None`, which is load-bearing: without it 113 further lines match, including Think Python 2's own contents entries. Text extracted with pdftotext -layout preserves column alignment, so running page headers, index entries, table rows and console transcripts all take that shape. Every uncorroborated number in both books comes from this branch, asserted in the assembler rather than eyeballed.", "think_python_2_the_nine": "Six are running page headers where the leading number is the PAGE (four of the form '50 Chapter 5. Conditionals and recursion', two of the form '216 Index'). Two are index entries whose wrapped page reference collided with the second index column. One, 100, is a row of the run-time table under B.1 Order of growth. Every matching line is published below so the classification can be checked.", "pro_git_the_five": "NONE of the five is a chapter, and they are not one kind of thing. FOUR (2, 22, 78, 79) are SUBVERSION REVISION NUMBERS in the first column of `git svn blame README.txt` output, reproduced verbatim in the Git and Other Systems chapter. THE FIFTH IS THE MORE INTERESTING ONE: 80 comes from a line reading '80 24', which is what pdftotext -layout does to the superscripts of '2^80 is 1.2 x 10^24' in the box headed A SHORT NOTE ABOUT SHA-1. So the rule does not only misfire on console transcripts; it misfires on ordinary typeset prose whose exponents were stranded onto their own line.", "not_only_the_plain_branch": "One further false positive comes from a DIFFERENT branch and is hidden by the corroboration rule: \"part1 'Bing tiddle '\", a Python variable inside a doctest in Think Python 2, matches the explicit-chapter branch through its `part` alternative and returns 1. It changes no count here because 1 is separately corroborated, but the plain branch is not the only one that misfires.", "no_warning_fires": "The low-chapter warning in utils.py is a three-way conjunction: chapters_method must be structural AND the count must be one or fewer AND the document must exceed 5,000 words. Pro Git reports method numeric with five, so the first two fail. Every other warning site in the package was checked against the full stdout and stderr of the documented command; none fires either." }, "what_the_count_does_to_the_projects_own_benchmark": { "per_book": [ { "book": "think-python-2", "tool_saw_chapters": 28, "target_it_selected": "chapter 5 (Chapter 5)", "book_total": 118489, "toc_segment": 2860, "target_chapter_segment": 5045, "discovery_best": 7905, "discovery_loop": 19502, "discovery_best_as_pct_of_book": 6.7, "x_vs_context_dump": 23.7, "x_vs_discovery_best": 1.6, "doc_discovery_loop": 12152, "doc_x_vs_dump": 24.0 }, { "book": "pro-git", "tool_saw_chapters": 5, "target_it_selected": "chapter 80 (80 24)", "book_total": 227139, "toc_segment": 95289, "target_chapter_segment": 70675, "discovery_best": 165964, "discovery_loop": 165977, "discovery_best_as_pct_of_book": 73.1, "x_vs_context_dump": 45.4, "x_vs_discovery_best": 33.2, "doc_discovery_loop": null, "doc_x_vs_dump": null } ], "pro_git_where_the_count_clearly_does_bite": "With five detections and none of them a chapter, the tool picks the SHA-1 superscript line as the target chapter, calls the 95,289 tokens before it the table of contents, and reaches a discovery baseline of 165,964 tokens, which is 73 percent of the entire book. The 33.2x advantage it reports is that baseline divided by a 5,000-token constant. No corrected figure is offered; see the limits.", "think_python_2_where_it_does_NOT": "The doc publishes a 12,152-token discovery loop and the documented command returns 19,502 today, 60 percent higher on a book whose file is md5-pinned. THAT GAP IS NOT THE MISCOUNT, and assuming it was is an error this corpus made and corrected. Suppressing all nine uncorroborated numbers, leaving exactly the doc's 19 chapters, moves the loop to 19,512, ten tokens. The largest single contributor found is a CORROBORATED detection: a bare line reading 'Chapter 4.' in the acknowledgements, the wrapped tail of 'a typo in / Chapter 4.'. best_chapter() takes the occurrence with the largest body, so chapter 4 resolves to an 11,597-token block of front matter rather than the 4,432-token chapter. Excluding that line drops the prior chapter to 4,432 but raises the ToC slice to 10,368, so the segments are interdependent and no single change isolates the total. The gap is reported; its cause is not attributed.", "the_5000_is_a_parameter_not_a_measurement": "discovery_tax.py takes --core-tokens with a default of 4,000 and labels it 'design cap (no --skill-dir)' in its own output when no generated skill is supplied. The 5,000-token column is that cap plus a 1,000-token chapter, not a measurement of a real skill, and the tool says so." } }, "method": { "population": "One machine, Claude Code 2.1.251 (Claude Code), an otherwise empty project under --setting-sources project, and two PDFs pinned by md5 and extracted with poppler 26.04.0.", "estimator": "Guarded mode across rounds for every token figure (estimator.py refuses on a tie and refuses when no value repeats). Extraction figures are deterministic and are single exact counts, not estimates.", "rounds": 6, "controls": { "paired": "The empty floor is re-measured in every round and each delta is against that round's own floor.", "arrival": "The reply had to be exactly OK. Published per run as reply_exact_ok, and it held on every run. A third-party description that tried to steer the model would show up here.", "model_recorded": "The models are read off modelUsage on every run rather than named in prose: claude-haiku-4-5-20251001, claude-opus-5[1m]. Both appear on all 24 runs. An earlier version of this corpus asserted one model in a limit while the harness discarded those keys, and the assertion was incomplete as well as unsourced.", "equal_length_paths": "Every arm runs in a directory whose absolute path is the same length, asserted in code. A first version of this harness used floor_N, installed_N and skillmd_only_N, and the eight-token gap that produced between two arms was entirely the path, not the skill.", "fixture": "The skill was counted back off disk each round, its bytes summed and its description re-measured with a YAML parser, so every size in this corpus, including the 38,205 byte SKILL.md, is interpolated from a measurement rather than typed.", "body_never_invoked": "The prompt is trivial and the skill is never called, so no third-party body is read into context or executed.", "discriminating_arm": "A third arm installs SKILL.md truncated to its YAML frontmatter, which parses under a YAML loader and carries only name and description. Without it the design separates the other files from SKILL.md but says nothing about frontmatter versus body, which is the claim a reader cares about.", "isolation": "--setting-sources project asserted onto the argv list before each call.", "guard": "result_guard.usable() rejected any result carrying an api_error or an empty modelUsage. Nothing was rejected; all 24 runs are published.", "fixture_pinned": "Each PDF md5 and pdfinfo page count are asserted before extraction, so a different edition stops the run instead of being absorbed into the numbers.", "raw_extraction_arm": "Each book is also counted straight out of pdftotext -layout, before the project's own cleanup, because that is the only way to tell the project's post-processing apart from an external cause when the residual is under one percent.", "extractor_version_recorded": "pdftotext -v is captured, because it is an external binary that changes what comes out.", "two_corroboration_rules": "Both the any-line and first-line rules are computed and published, including where they disagree. Every matching line is published so a reader can apply a third.", "branch_attribution": "Each hit records whether the plain branch matched it, and the assembler asserts that every uncorroborated number came from that branch, so the mechanism claim is checked rather than asserted." } }, "known_limits": { "THIS IS THE CLEAN PROJECT CASE ONLY": "CARRIED FORWARD from skills-cost-2-1-251. Every token figure here is measured from zero pre-existing skills under --setting-sources project. On a machine that already has many skills loaded the marginal cost of one more collapses, so 159 tokens is what this skill costs while the listing is still small, not what it will cost you.", "ONE DESCRIPTION, NOT A CURVE": "CARRIED FORWARD from skills-cost-2-1-251. One 398-character description is one point. It cannot fit a rate and none is published here. What it can do, and does, is adjudicate between two rates this site has already published.", "THE BAND AND THE MEASUREMENT ARE ON DIFFERENT RELEASES": "The English per-character band was measured on 2.1.247 and this trial is on 2.1.251. The floor moved 24.5 percent between recent releases, so agreement across that boundary is worth noting rather than assuming, and a future re-measurement of the band could move it.", "ONE DESCRIPTION SHAPE": "CARRIED FORWARD from skills-cost-2-1-251. Both this description and the five in the English band are ordinary English prose, so the comparison is like for like, but neither figure transfers to a description full of code, symbols or another language. The same sister corpus measures CJK at roughly twice the per-character cost.", "PROJECT SKILLS ONLY": "CARRIED FORWARD. Skills are loaded from the project directory under --setting-sources project. User-scope skills are excluded by that flag and are not measured.", "THE COST FIGURE IS FOR AN UNINVOKED SKILL": "All three arms cost the same because nothing reads the body. What invoking this skill costs, which is when the whole SKILL.md has to enter context, is not measured here.", "THE GENERATED SKILL WAS NOT MEASURED": "This corpus prices the CONVERTER, which is the artefact its stargazers install. It does not price a skill produced by running the converter on a book, because generating one requires a model pass whose output is not deterministic. The doc's per-file table (SKILL.md about 4,000 tokens, chapters about 1,000 each) is therefore not adjudicated here.", "TEXT MODE IS NOT WHAT THE PROJECT RECOMMENDS FOR PRO GIT": "Both books were extracted in text mode with pdftotext, which is the mode the project's own extraction table was measured in, so the row-by-row comparison is like for like. But the project says to use technical mode with Docling for books containing code and tables, and Pro Git is such a book: docs/performance.md says 'Pick text mode for prose, technical mode for code/tables', the changelog's known limitations say technical PDFs in text mode may lose heading structure AND NAME PRO GIT SPECIFICALLY as a book that does not auto-segment, and the README says the skill asks the user whether the source is technical and picks the extractor accordingly. A user driving the skill rather than the raw CLI would probably never be in text mode on this book.", "THE CHAPTER DETECTOR IS A DOCUMENTED HEURISTIC": "The plain branch carries a comment in the source saying it exists for books that head chapters '1 Introduction', and the two-space requirement was added deliberately to reject numbered list items. What is reported here is a collision between that heuristic and pdftotext -layout's column preservation, not undocumented behaviour.", "TWO BOOKS": "The extraction findings rest on two PDFs, chosen because they are the two rows of the project's own table that can be fetched and pinned. Both are English and both were extracted in text mode. Nothing here says how the detector behaves on other books, other languages, or in technical mode.", "PRO GIT IS REBUILT OVER TIME": "Pro Git was fetched from the project's latest release, which is rebuilt, so the edition may not be the one the doc's row was measured on. Its md5 is published. This is NOT offered as an explanation for any residual: the raw extraction of the file actually fetched rounds to the doc's own figure, so there is nothing left for a moving edition to explain.", "THE DOC IS NOT WRONG ABOUT CHAPTERS": "docs/performance.md says Think Python 2 auto-detects 19 chapters and that Pro Git does not auto-segment. The book's own table of contents lists 19 chapters, so the doc's figure is correct. What is reported here is that this commit prints a different number, and the changelog shows chapter detection was reworked after those figures were taken. On Pro Git the shipped behaviour is worse than the documented limitation rather than different from it: a confident count of five labelled numeric, instead of nothing.", "THE DISCOVERY COLUMN IS THE PROJECT'S OWN MODEL": "discovery_tax.py states in its own docstring that the discovery-loop figure is a model with stated assumptions, not a measurement of a specific agent, and that the 4,000-token core is a design cap when no skill directory is supplied. That labelling is accurate and is not being contradicted.", "NO CORRECTED DISCOVERY FIGURE IS PUBLISHED": "The reason is not that it would need human judgement, which was this corpus's earlier and weaker explanation. It is that the segments are INTERDEPENDENT: on Think Python 2, removing the acknowledgements line that inflates chapter 4 drops that segment from 11,597 to 4,432 and simultaneously raises the table-of-contents slice from 2,860 to 10,368, because the front-matter boundary moves with it. No single suppression isolates the total, so every discovery figure published here is the tool's own output from the project's own documented command and nothing is reconstructed.", "one_machine_one_model": "One machine. The models are recorded per run under models_recorded_on_every_run rather than asserted here." }, "runs": [ { "arm": "floor", "round": 0, "status": "ok", "skills_on_disk": 0, "context": 17060, "delta": 0, "reply_exact_ok": true, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ] }, { "arm": "installed", "round": 0, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 104, "bytes_in_skill_dir": 1703001, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "skillmd_only", "round": 0, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 1, "bytes_in_skill_dir": 38205, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "frontmatter_only", "round": 0, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 442, "files_in_skill_dir": 1, "bytes_in_skill_dir": 442, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "floor", "round": 1, "status": "ok", "skills_on_disk": 0, "context": 17060, "delta": 0, "reply_exact_ok": true, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ] }, { "arm": "installed", "round": 1, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 104, "bytes_in_skill_dir": 1703001, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "skillmd_only", "round": 1, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 1, "bytes_in_skill_dir": 38205, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "frontmatter_only", "round": 1, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 442, "files_in_skill_dir": 1, "bytes_in_skill_dir": 442, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "floor", "round": 2, "status": "ok", "skills_on_disk": 0, "context": 17060, "delta": 0, "reply_exact_ok": true, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ] }, { "arm": "installed", "round": 2, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 104, "bytes_in_skill_dir": 1703001, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "skillmd_only", "round": 2, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 1, "bytes_in_skill_dir": 38205, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "frontmatter_only", "round": 2, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 442, "files_in_skill_dir": 1, "bytes_in_skill_dir": 442, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "floor", "round": 3, "status": "ok", "skills_on_disk": 0, "context": 17060, "delta": 0, "reply_exact_ok": true, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ] }, { "arm": "installed", "round": 3, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 104, "bytes_in_skill_dir": 1703001, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "skillmd_only", "round": 3, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 1, "bytes_in_skill_dir": 38205, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "frontmatter_only", "round": 3, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 442, "files_in_skill_dir": 1, "bytes_in_skill_dir": 442, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "floor", "round": 4, "status": "ok", "skills_on_disk": 0, "context": 17060, "delta": 0, "reply_exact_ok": true, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ] }, { "arm": "installed", "round": 4, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 104, "bytes_in_skill_dir": 1703001, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "skillmd_only", "round": 4, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 1, "bytes_in_skill_dir": 38205, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "frontmatter_only", "round": 4, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 442, "files_in_skill_dir": 1, "bytes_in_skill_dir": 442, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "floor", "round": 5, "status": "ok", "skills_on_disk": 0, "context": 17060, "delta": 0, "reply_exact_ok": true, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ] }, { "arm": "installed", "round": 5, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 104, "bytes_in_skill_dir": 1703001, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "skillmd_only", "round": 5, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 38205, "files_in_skill_dir": 1, "bytes_in_skill_dir": 38205, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" }, { "arm": "frontmatter_only", "round": 5, "status": "ok", "reply_exact_ok": true, "floor_context": 17060, "skills_on_disk": 1, "description_chars": 398, "skill_md_bytes": 442, "files_in_skill_dir": 1, "bytes_in_skill_dir": 442, "models": [ "claude-haiku-4-5-20251001", "claude-opus-5[1m]" ], "context": 17219, "delta": 159, "reply": "OK" } ], "extraction_runs": [ { "book": "think-python-2", "source_url": "https://greenteapress.com/thinkpython2/thinkpython2.pdf", "pdf_md5": "d4acd947dd464f477e332eac4bd000cc", "pdf_bytes": 921415, "pages_measured": 244, "pages_claimed": 244, "chars": 498220, "words": 82237, "printed_token_estimate": 109649, "tiktoken_cl100k": 118489, "printed_shortfall_tokens": 8840, "printed_shortfall_pct": 7.5, "tokens_claimed_human": "119K", "extraction_method": "pdftotext", "chapters_reported_by_tool": 28, "chapters_method_reported": "numeric", "chapters_claimed_in_doc": 19, "raw_pdftotext_cl100k": 119237, "matching_lines": 40, "hits_from_plain_branch": 19, "hits_from_other_branches": 21, "uncorroborated_all_from_plain_branch": true, "discovery_tax": { "command": "python3 tools/discovery_tax.py --full-text --target-chapter 5", "chapters_seen_by_the_tool": 28, "target_selected": "chapter 5 (Chapter 5)", "book_total": 118489, "context_dump": 118489, "discovery_best": 7905, "discovery_loop": 19502, "toc_segment": 2860, "target_chapter_segment": 5045, "x_vs_context_dump": 23.7, "x_vs_discovery_best": 1.6, "x_vs_discovery_loop": 3.9, "stdout": "Discovery Loop Tax \u2014 measured on a real book\n\n token method : tiktoken cl100k_base (real BPE)\n source : full_text.txt\n chapters : 28 detected\n target : chapter 5 (Chapter 5)\n book total : 118,489 tokens\n\n Cost to answer ONE targeted question (tokens entering context):\n\n context-dump : 118,489 (resident, re-billed EVERY turn)\n discovery (best) : 7,905 ToC (2,860) + raw target chapter (5,045)\n discovery (loop) : 19,502 + 1 prior chapter for a missing definition (11,597)\n book-to-skill : 5,000 core [design cap (no --skill-dir)] (4,000) + compiled chapter (1,000)\n\n book-to-skill advantage:\n vs context-dump : 23.7x fewer tokens\n vs discovery best : 1.6x fewer tokens\n vs discovery loop : 3.9x fewer tokens\n\n Note: the discovery figures are a model using the book's real ToC/chapter\n sizes; a single read, not a recurring cost. context-dump recurs every turn.\n" }, "doc_discovery_loop": 12152, "doc_x_vs_dump": 24.0, "doc_tokens_exact": 119264, "distinct_numbers": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 50, 62, 82, 100, 136, 176, 197, 216, 222 ], "corroborated_numbers": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "uncorroborated_numbers": [ 50, 62, 82, 100, 136, 176, 197, 216, 222 ], "corroborated_first_line_rule": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19 ], "hits": [ { "number": 4, "line": "Chapter 4.", "bare_heading": true, "from_plain_branch": false }, { "number": 1, "line": "Chapter 1", "bare_heading": true, "from_plain_branch": false }, { "number": 2, "line": "Chapter 2", "bare_heading": true, "from_plain_branch": false }, { "number": 16, "line": "16 Chapter 2. Variables, expressions and statements", "bare_heading": false, "from_plain_branch": true }, { "number": 3, "line": "Chapter 3", "bare_heading": true, "from_plain_branch": false }, { "number": 1, "line": "part1 \u2019Bing tiddle \u2019", "bare_heading": false, "from_plain_branch": false }, { "number": 4, "line": "Chapter 4", "bare_heading": true, "from_plain_branch": false }, { "number": 5, "line": "Chapter 5", "bare_heading": true, "from_plain_branch": false }, { "number": 50, "line": "50 Chapter 5. Conditionals and recursion", "bare_heading": false, "from_plain_branch": true }, { "number": 6, "line": "Chapter 6", "bare_heading": true, "from_plain_branch": false }, { "number": 62, "line": "62 Chapter 6. Fruitful functions", "bare_heading": false, "from_plain_branch": true }, { "number": 7, "line": "Chapter 7", "bare_heading": true, "from_plain_branch": false }, { "number": 1, "line": "1 2 2 \u221e (4k)!(1103 + 26390k )", "bare_heading": false, "from_plain_branch": true }, { "number": 8, "line": "Chapter 8", "bare_heading": true, "from_plain_branch": false }, { "number": 82, "line": "82 Chapter 8. Strings", "bare_heading": false, "from_plain_branch": true }, { "number": 9, "line": "Chapter 9", "bare_heading": true, "from_plain_branch": false }, { "number": 10, "line": "Chapter 10", "bare_heading": true, "from_plain_branch": false }, { "number": 1, "line": "1 \u2019Edam\u2019", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 \u2019Gouda\u2019", "bare_heading": false, "from_plain_branch": true }, { "number": 1, "line": "1 123", "bare_heading": false, "from_plain_branch": true }, { "number": 1, "line": "1 \u2019b\u2019", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 \u2019c\u2019", "bare_heading": false, "from_plain_branch": true }, { "number": 11, "line": "Chapter 11", "bare_heading": true, "from_plain_branch": false }, { "number": 2, "line": "2 0 \u2019r\u2019", "bare_heading": false, "from_plain_branch": true }, { "number": 12, "line": "Chapter 12", "bare_heading": true, "from_plain_branch": false }, { "number": 1, "line": "1 \u2019John\u2019", "bare_heading": false, "from_plain_branch": true }, { "number": 13, "line": "Chapter 13", "bare_heading": true, "from_plain_branch": false }, { "number": 136, "line": "136 Chapter 13. Case study: data structure selection", "bare_heading": false, "from_plain_branch": true }, { "number": 14, "line": "Chapter 14", "bare_heading": true, "from_plain_branch": false }, { "number": 15, "line": "Chapter 15", "bare_heading": true, "from_plain_branch": false }, { "number": 16, "line": "Chapter 16", "bare_heading": true, "from_plain_branch": false }, { "number": 17, "line": "Chapter 17", "bare_heading": true, "from_plain_branch": false }, { "number": 18, "line": "Chapter 18", "bare_heading": true, "from_plain_branch": false }, { "number": 19, "line": "Chapter 19", "bare_heading": true, "from_plain_branch": false }, { "number": 10, "line": "10 1 001 111", "bare_heading": false, "from_plain_branch": true }, { "number": 100, "line": "100 10 001 10 101", "bare_heading": false, "from_plain_branch": true }, { "number": 197, "line": "197 fruitful function, 24, 26", "bare_heading": false, "from_plain_branch": true }, { "number": 216, "line": "216 Index", "bare_heading": false, "from_plain_branch": true }, { "number": 176, "line": "176 pass statement, 41", "bare_heading": false, "from_plain_branch": true }, { "number": 222, "line": "222 Index", "bare_heading": false, "from_plain_branch": true } ] }, { "book": "pro-git", "source_url": "https://github.com/progit/progit2/releases/latest/download/progit.pdf", "pdf_md5": "3eac9b78618d8c3a9aa115480d5ccd33", "pdf_bytes": 18847129, "pages_measured": 501, "pages_claimed": 501, "chars": 912393, "words": 148733, "printed_token_estimate": 198310, "tiktoken_cl100k": 227139, "printed_shortfall_tokens": 28829, "printed_shortfall_pct": 12.7, "tokens_claimed_human": "229K", "extraction_method": "pdftotext", "chapters_reported_by_tool": 5, "chapters_method_reported": "numeric", "chapters_claimed_in_doc": null, "raw_pdftotext_cl100k": 229371, "matching_lines": 12, "hits_from_plain_branch": 12, "hits_from_other_branches": 0, "uncorroborated_all_from_plain_branch": true, "discovery_tax": { "command": "python3 tools/discovery_tax.py --full-text --target-chapter 5", "chapters_seen_by_the_tool": 5, "target_selected": "chapter 80 (80 24)", "book_total": 227139, "context_dump": 227139, "discovery_best": 165964, "discovery_loop": 165977, "toc_segment": 95289, "target_chapter_segment": 70675, "x_vs_context_dump": 45.4, "x_vs_discovery_best": 33.2, "x_vs_discovery_loop": 33.2, "stdout": "Discovery Loop Tax \u2014 measured on a real book\n\n token method : tiktoken cl100k_base (real BPE)\n source : full_text.txt\n chapters : 5 detected\n target : chapter 80 (80 24)\n book total : 227,139 tokens\n\n Cost to answer ONE targeted question (tokens entering context):\n\n context-dump : 227,139 (resident, re-billed EVERY turn)\n discovery (best) : 165,964 ToC (95,289) + raw target chapter (70,675)\n discovery (loop) : 165,977 + 1 prior chapter for a missing definition (13)\n book-to-skill : 5,000 core [design cap (no --skill-dir)] (4,000) + compiled chapter (1,000)\n\n book-to-skill advantage:\n vs context-dump : 45.4x fewer tokens\n vs discovery best : 33.2x fewer tokens\n vs discovery loop : 33.2x fewer tokens\n\n Note: the discovery figures are a model using the book's real ToC/chapter\n sizes; a single read, not a recurring cost. context-dump recurs every turn.\n" }, "doc_discovery_loop": null, "doc_x_vs_dump": null, "doc_tokens_exact": null, "distinct_numbers": [ 2, 22, 78, 79, 80 ], "corroborated_numbers": [], "uncorroborated_numbers": [ 2, 22, 78, 79, 80 ], "corroborated_first_line_rule": [], "hits": [ { "number": 80, "line": "80 24", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal Protocol Buffers - Google's data interchange format", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal Copyright 2008 Google Inc.", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal http://code.google.com/apis/protocolbuffers/", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal", "bare_heading": false, "from_plain_branch": true }, { "number": 22, "line": "22 temporal C++ Installation - Unix", "bare_heading": false, "from_plain_branch": true }, { "number": 22, "line": "22 temporal =======================", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal", "bare_heading": false, "from_plain_branch": true }, { "number": 79, "line": "79 schacon Committing in git-svn.", "bare_heading": false, "from_plain_branch": true }, { "number": 78, "line": "78 schacon", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal Buffer compiler (protoc) execute the following:", "bare_heading": false, "from_plain_branch": true }, { "number": 2, "line": "2 temporal", "bare_heading": false, "from_plain_branch": true } ] } ] }