{ "measurement": "What JetBrains' go-modern-guidelines skill costs when it fires on a Go task, and whether it changed the code", "claude_code_version": "2.1.247 (Claude Code)", "subject": { "repo": "JetBrains/go-modern-guidelines", "commit": "40781f1", "stars_at_trial": 1916, "license": "Apache-2.0", "plugin": "modern-go-guidelines 1.1.1, one skill: use-modern-go", "skill_description_chars": 151, "skill_body_bytes": 2610, "what_it_does": "The skill body tells the model to shell out to a pinned Go CLI (installed on first use via go install) and apply its guidance before editing Go code.", "security_scan": "PASSED. 29 files, no eval, no exfiltration, hosts limited to github/jetbrains/go.dev. The wrapper pins a version, installs into a cache dir, and REFUSES to run the binary unless it reports the expected version." }, "method": { "population": "one project, one prompt, measured in a constant directory created and removed per run", "estimator": "guarded mode for the listing cell; the go-task arms are reported as ranges because they do not overlap and no point estimate is needed", "rounds": 4, "prompts": { "trivial": "Reply with only: OK", "go_task": "Write a small Go function in a file called sum.go that takes a slice of ints and returns their sum. Just write the file. Do not explain." }, "controls": [ "paired within round", "--setting-sources project asserted onto the command line", "SKILL.md asserted byte-identical to the cloned upstream file every run", "BOTH go-task arms run with the same --permission-mode bypassPermissions", "wrote_sum_go recorded per run: every go-task run in BOTH arms produced the file", "cli_cache_exists_after recorded per run: the CLI genuinely installed and ran", "result_guard refuses an unusable result rather than recording a zero" ] }, "results": { "headline": { "go_task_without_the_skill": { "min": 45582, "max": 45785 }, "go_task_with_the_skill": { "min": 146286, "max": 146872 }, "arms_overlap": false, "gap_tokens": 100501, "multiplier_range": [ 3.2, 3.22 ], "plain_statement": "On this task the same work cost 45,582 to 45,785 tokens without the skill and 146,286 to 146,872 with it, about 3.2x. The arms do not overlap across four rounds. Both arms wrote the file." }, "what_the_code_looked_like": { "identical_except_one_comment": true, "no_skill": "package main\n\n// Sum returns the sum of all ints in the slice.\nfunc Sum(nums []int) int {\n\ttotal := 0\n\tfor _, n := range nums {\n\t\ttotal += n\n\t}\n\treturn total\n}", "with_skill": "package main\n\n// Sum returns the sum of all ints in nums.\nfunc Sum(nums []int) int {\n\ttotal := 0\n\tfor _, n := range nums {\n\t\ttotal += n\n\t}\n\treturn total\n}", "the_only_difference": "the doc comment reads 'in the slice' without the skill and 'in nums' with it. The function body is identical." }, "listing_cost_when_it_does_not_fire": { "mode": 65, "mode_seen_in_rounds": 2, "rounds": 4, "all_rounds": [ -832, -133, 65, 65 ], "caveat": "POORLY RESOLVED. The mode is carried on only 2 of 4 rounds and two rounds are negative, which is this machine's intermittent component swamping a small signal. Treat it as 'small, tens of tokens', not as a measured figure." } }, "known_limits": { "ONE TASK, AND A TASK WITH NO MODERN IDIOM TO APPLY": "Summing a slice of ints with a range loop is already the idiomatic answer; Go has no stdlib Sum. So this task cannot show the skill's BENEFIT and it is not evidence the guidance is worthless. What it shows is that the cost is paid anyway. The skill's own description says to use it 'whenever writing, modifying, fixing, or refactoring Go code', which is unconditional, so a task with nothing to gain still pays. A task with a real modern-idiom opportunity is the obvious next measurement and is NOT done here. RETIRED 2026-08-27 by /gomg-skill-idiom-2-1-247.json, which is that arm: given a task the tool's own ruleset covers (slices_sort_func), the no-skill arm wrote the pre-1.21 sort.Slice in 6 of 6 runs and the skill arm wrote slices.SortFunc with cmp.Compare in 6 of 6, with no run going the other way. The skill DOES deliver, and this corpus alone must not be read as evidence it does nothing. Read the two together.", "THE FIRST PASS WAS INVALID AND IS REPORTED, NOT DISCARDED SILENTLY": "An earlier 16-row pass ran without a permission mode. Claude Code's -p denies Write and Bash by default, so every skill/go_task reply said the CLI and the file write were blocked, and the CLI's cache directory was never created. That pass measured permission-denial retries, not the skill. It is superseded entirely by this one and none of its numbers are used.", "the_multiplier_is_a_round_trip_effect_not_a_payload": "The skill does not inject 100,000 tokens. It adds tool round trips: install and run the CLI, read its output, then edit. This site has measured that each tool call re-sends the whole conversation, so context grows faster than linearly in round trips. The 3.2x is that mechanism, and no per-call breakdown is published here because the harness records only the session total.", "quality_is_not_measured": "Neither file was compiled, tested or reviewed. 'Identical except a comment' is a statement about the bytes produced on this task, not a judgement that the guidance is useless.", "first_run_includes_the_install": "Round 0's skill arm is the round that compiled the CLI via go install. It came in at 146,565, inside the same range as the cached rounds, so the install did not dominate. On a machine without a Go toolchain the skill cannot run at all.", "one_machine_one_model": "Opus on Claude Code 2.1.247, project scope, one machine.", "bypass_permissions_is_not_how_most_people_run_it": "Both go-task arms bypass permission prompts so the task could complete unattended. Interactively a user would be asked to approve the CLI, and might decline, in which case the cost is not paid and the guidance is not applied." }, "runs": [ { "fixture": "no_skill", "prompt_type": "trivial", "round": 0, "status": "ok", "skill_bytes_on_disk": 0, "context": 23174, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": false, "reply_head": "OK" }, { "fixture": "skill", "prompt_type": "trivial", "round": 0, "status": "ok", "skill_bytes_on_disk": 2610, "context": 23041, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": false, "reply_head": "OK" }, { "fixture": "no_skill", "prompt_type": "go_task", "round": 0, "status": "ok", "skill_bytes_on_disk": 0, "context": 45582, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": false, "reply_head": "Written to ``." }, { "fixture": "skill", "prompt_type": "go_task", "round": 0, "status": "ok", "skill_bytes_on_disk": 2610, "context": 146565, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "`` written." }, { "fixture": "no_skill", "prompt_type": "trivial", "round": 1, "status": "ok", "skill_bytes_on_disk": 0, "context": 23174, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": true, "reply_head": "OK" }, { "fixture": "skill", "prompt_type": "trivial", "round": 1, "status": "ok", "skill_bytes_on_disk": 2610, "context": 23239, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": true, "reply_head": "OK" }, { "fixture": "no_skill", "prompt_type": "go_task", "round": 1, "status": "ok", "skill_bytes_on_disk": 0, "context": 45781, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "Written to ``." }, { "fixture": "skill", "prompt_type": "go_task", "round": 1, "status": "ok", "skill_bytes_on_disk": 2610, "context": 146872, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "`sum.go` written." }, { "fixture": "no_skill", "prompt_type": "trivial", "round": 2, "status": "ok", "skill_bytes_on_disk": 0, "context": 23174, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": true, "reply_head": "OK" }, { "fixture": "skill", "prompt_type": "trivial", "round": 2, "status": "ok", "skill_bytes_on_disk": 2610, "context": 23239, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": true, "reply_head": "OK" }, { "fixture": "no_skill", "prompt_type": "go_task", "round": 2, "status": "ok", "skill_bytes_on_disk": 0, "context": 45583, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "Written to ``." }, { "fixture": "skill", "prompt_type": "go_task", "round": 2, "status": "ok", "skill_bytes_on_disk": 2610, "context": 146846, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "`sum.go` written." }, { "fixture": "no_skill", "prompt_type": "trivial", "round": 3, "status": "ok", "skill_bytes_on_disk": 0, "context": 23174, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": true, "reply_head": "OK" }, { "fixture": "skill", "prompt_type": "trivial", "round": 3, "status": "ok", "skill_bytes_on_disk": 2610, "context": 22342, "reply_exact_ok": true, "refused_because": null, "wrote_sum_go": null, "cli_cache_exists_after": true, "reply_head": "OK" }, { "fixture": "no_skill", "prompt_type": "go_task", "round": 3, "status": "ok", "skill_bytes_on_disk": 0, "context": 45785, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "Written to ``." }, { "fixture": "skill", "prompt_type": "go_task", "round": 3, "status": "ok", "skill_bytes_on_disk": 2610, "context": 146286, "reply_exact_ok": null, "refused_because": null, "wrote_sum_go": true, "cli_cache_exists_after": true, "reply_head": "`sum.go` written." } ] }