{ "trial": "What six model runs say about a figure this site has measured, and what the vendor doc actually says", "date": "2026-08-15", "surface": "duck.ai (DuckDuckGo AI Chat), free tier, no account", "prompt": "How many tokens does each Agent Skill add to Claude Code's startup context?", "method": { "shape": "One ordinary question, one New Chat per run so no previous answer sits in context.", "model_label": "Recorded from the label the UI itself prints above each answer, and screenshotted rather than scraped, because a product can change what sits behind a label without saying so.", "THIS_IS_AN_IMPRESSION_NOT_A_MEASUREMENT": "Six runs across four models is far too small to characterise any model, and the trial itself proved why: the same model gave opposite answers on runs 5 and 6. Nothing here is presented as a model property. The headline finding of the post rests on the published skillpack corpus, 56 runs, not on this.", "no_bulk_querying": "Six prompts typed by hand into a public UI. Nothing scripted.", "accusations_withdrawn_in_full": "This corpus makes no claim that any model answered incorrectly. Where a draft did, the charges were withdrawn: two on checking and two for lack of preserved verbatim text. The lesson is recorded rather than the accusation: if a trial may end up alleging error against a named company's product, capture the complete raw output at the time, because a charge a reader cannot check from the published record has to be dropped." }, "runs": [ { "run": 1, "model_label_shown": "GPT-5.4 nano", "vendor": "OpenAI", "verdict": "CONFIDENT NUMBER", "figure_given": "~100 tokens", "cited_source": null, "quote": "Each Agent Skill adds about ~100 tokens to Claude Code's startup context (for the YAML frontmatter name + description).", "notes": "Names the right mechanism, YAML frontmatter name + description. Matches the vendor doc figure exactly." }, { "run": 2, "model_label_shown": "GPT-5.4 mini", "vendor": "OpenAI", "verdict": "CONFIDENT NUMBER", "figure_given": "~100 tokens", "cited_source": "claude.com, 'Agent Skills - Claude Platform D...' shown in a Search Results block", "quote": "About ~100 tokens per skill at startup for the metadata only; the full SKILL.md body loads later when triggered.", "notes": "Ran a web search and cited the vendor page. The citation is legitimate: the figure IS on that page." }, { "run": 3, "model_label_shown": "GPT-5.4 mini", "vendor": "OpenAI", "verdict": "CONFIDENT NUMBER", "figure_given": "~100 tokens", "cited_source": "leaked a raw tag into the rendered answer", "quote": "About ~100 tokens per skill at startup, for just the name + description/frontmatter.", "notes": "Second run of the same model. Same figure. A raw citation tag leaked into the visible output, which is a UI defect not a model one." }, { "run": 4, "model_label_shown": "Claude Haiku 4.5", "vendor": "Anthropic", "verdict": "DECLINED", "figure_given": null, "cited_source": null, "quote": "I don't have specific information about Claude Code's startup context and how many tokens each Agent Skill adds to it.", "notes": "Recommended four things including 'Testing empirically by comparing the startup context token usage with and without specific Agent Skills enabled', which is this site's method. Declining was reasonable but the figure IS documented, so a search would have found it." }, { "run": 5, "model_label_shown": "Mistral Small 4", "vendor": "Mistral", "verdict": "GAVE A RANGE, 50-200 tokens", "figure_given": "roughly 50-200 tokens", "cited_source": null, "quote": "the exact token count for each \"Agent Skill\" isn't publicly documented by Anthropic. However, based on community observations and testing: Each Agent Skill (e.g., code, web_search, file_edit) adds roughly 50-200 tokens ... The base system prompt (without any skills) is around ~1,000 tokens.", "notes": "NO ACCUSATION OF ERROR IS MADE AGAINST THIS ANSWER. An earlier draft listed four; two were withdrawn on checking and the remaining two were withdrawn as well. WITHDRAWN 1: that it wrongly said the figure is undocumented. It said the EXACT token count is undocumented, and Anthropic publishes '~100', explicitly approximate, so on the full quote it is defensibly right; the charge was manufactured by truncating it. WITHDRAWN 2: comparing its figure for a base SYSTEM PROMPT against this site's 22,283 whole-session startup floor, which includes tool definitions. Different quantities. WITHDRAWN 3 and 4: the verbatim text needed for a reader to check them was not preserved, and a public accusation against a named company's model that cannot be verified from the published record should not be made. What IS recorded is the answer's figure, a 50-200 range, which contains both the documented ~100 and two of the three reliable per-skill measurements in this corpus.", "errors_standing": 2, "errors_withdrawn_on_check": 2, "errors_alleged": 0 }, { "run": 6, "model_label_shown": "Mistral Small 4", "vendor": "Mistral", "verdict": "DECLINED", "figure_given": null, "cited_source": null, "quote": "I don't have specific information about the exact token count that each Agent Skill adds to Claude Code's startup context.", "notes": "SAME MODEL, SAME PROMPT, FRESH CHAT, OPPOSITE BEHAVIOUR to run 5. This is the single most useful thing the trial produced." } ], "what_the_vendor_documents": { "page": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview", "read_on": "2026-08-15", "verbatim_table_row": "| **Level 1: Metadata** | Always (at startup) | ~100 tokens per Skill | `name` and `description` from YAML frontmatter |", "note": "The Claude CODE skills page at code.claude.com carries NO per-skill token figure; its only token numbers are auto-compaction budgets of 5,000 per re-attached skill and 25,000 combined. The ~100 figure is on the PLATFORM docs page." }, "reading": "The models were not inventing the ~100 figure. It is documented. Three of six runs repeated it, one with a correct citation. Two declined. One gave a 50-200 range, which contains the documented figure. No accusation of error is made against any of them: an earlier draft listed four against one model and all four were withdrawn, two on checking and two because the verbatim text a reader would need was not preserved. The interesting part is not which model is 'best': it is that the SAME model gave a range on one run and declined on the next, so a single answer from any of them is not evidence of anything.", "doc_vs_measured": { "documented_figure_tokens_per_skill": 100, "source_corpus": "/skillpack-cost-2-1-233.json, 56 runs across two harnesses, already adversarially verified", "packs": { "Nutlope_hallmark": { "skills_shipped": 1, "measured_tokens": 117.0, "measured_per_skill": 117.0, "doc_predicts": 100, "doc_over_under_percent": -15, "description_median_chars": 260, "single_skill_inside_the_noise": true }, "addyosmani_agent-skills": { "skills_shipped": 24, "measured_tokens": 2302.0, "measured_per_skill": 95.92, "doc_predicts": 2400, "doc_over_under_percent": 4, "description_median_chars": 249, "single_skill_inside_the_noise": false }, "cathrynlavery_diagram-design": { "skills_shipped": 1, "measured_tokens": 274.0, "measured_per_skill": 274.0, "doc_predicts": 100, "doc_over_under_percent": -64, "description_median_chars": 579, "single_skill_inside_the_noise": true }, "google_skills": { "skills_shipped": 111, "measured_tokens": 9020.0, "measured_per_skill": 81.26, "doc_predicts": 11100, "doc_over_under_percent": 23, "description_median_chars": 397, "single_skill_inside_the_noise": false }, "mattpocock_skills": { "skills_shipped": 25, "measured_tokens": 852.0, "measured_per_skill": 34.08, "doc_predicts": 2500, "doc_over_under_percent": 193, "description_median_chars": 149, "single_skill_inside_the_noise": false }, "tt-a1i_archify": { "skills_shipped": 1, "measured_tokens": 262.0, "measured_per_skill": 262.0, "doc_predicts": 100, "doc_over_under_percent": -62, "description_median_chars": 652, "single_skill_inside_the_noise": true } }, "reliable_population": "Only the three MULTI-skill packs. This site has measured that a single skill cannot be resolved against the floor, and one round of the one-skill pack hallmark read 404 tokens BELOW an empty floor, which installing a skill cannot do. The three one-skill rows are published above and are NOT used for any claim.", "headline": "Across the three packs where the measurement is reliable, per-skill cost ran 34.08 to 95.92 tokens, and EVERY ONE came in below the documented ~100. The over-prediction ran from 4% to 193%, and it does NOT order by description length: the closest fit is the pack with the MIDDLE description length, and the pack with the longest descriptions is neither the closest nor the furthest. Nothing measured here predicts which end of that range a given pack will land on.", "why_one_number_cannot_work": "Description length is clearly involved: the shortest-description pack is by far the cheapest per skill and by far the worst fit to the documented figure. But it does not order the results. google/skills has descriptions 1.6x longer than addyosmani/agent-skills and costs 15% LESS per skill, which is a finding this site had already published from these same six packs. A monotonic story from three points would be inventing a mechanism, and an earlier draft of this corpus did exactly that by naming the two packs that fit and omitting the one that does not.", "this_sites_own_model_is_worse_than_the_documented_figure": { "note": "Published against this site's interest. This site fitted its own per-skill model, about 13.7 tokens plus 0.276 per character of description. Against the three reliable packs, Anthropic's rough ~100 is the CLOSER predictor on two of them.", "per_pack": { "addyosmani_agent-skills": { "measured": 2302, "doc": 2400, "doc_err_pct": 4, "site_rate": 2124, "rate_err_pct": 8, "closer": "doc" }, "google_skills": { "measured": 9020, "doc": 11100, "doc_err_pct": 23, "site_rate": 14699, "rate_err_pct": 63, "closer": "doc" }, "mattpocock_skills": { "measured": 852, "doc": 2500, "doc_err_pct": 193, "site_rate": 1397, "rate_err_pct": 64, "closer": "site_rate" } }, "reading": "The site's rate only wins where the documented figure is at its worst. That is why the recommendation is 'install it and measure it' rather than 'use this site's number'." }, "ceiling_claim_is_scoped": "~100 held as a ceiling across every reliable pack measured, whose median descriptions ran 149 to 397 characters. Above 397 there is NO reliable data here. The three single-skill packs, with 579 and 652-character descriptions, measured 262 and 274 per skill, far above 100 \u2014 and they are not used as a counterexample only because single skills sit inside this harness's noise." } }