{ "measurement": "Which AI crawlers actually fetched robots.txt and llms.txt over 14 days on a site that explicitly allows every named AI crawler", "headline": "1 of the 5 AI crawlers this site allows ever appeared. The other 4 fetched neither file.", "window": { "first_hit": "2026-08-10T07:14:00+00:00", "last_hit": "2026-08-25T02:44:56+00:00", "days": 14, "hours": 19 }, "method": { "how": "robots.txt and llms.txt are served through a PHP shim that appends one JSON line per request. Every well-behaved crawler fetches robots.txt, which is why that path was chosen as the observation point.", "why_it_exists": "This hosting plan exposes no web access logs over SSH, so crawler visits were otherwise unobservable.", "allowlist_source": "Parsed from the live robots.txt at assembly time, not hand-typed. Ordinary search crawlers (Bingbot, Googlebot) and the wildcard are excluded as not being the AI question.", "curl_excluded": "632 of 910 lines were this site's own uptime checks and are excluded from every figure.", "aggregates_only": "The raw log carries an ip_hash per visit. None is published here.", "purpose_labels": "training/search labels come from each vendor's published crawler documentation. Nothing here measures what a crawler did with the bytes." }, "instrument_changed_after_this_window": "On 2026-08-25, AFTER the window below closed, the shim was extended to a third path, sitemap.xml, and control-tested both ways: a marked fetch of the sitemap was recorded and a marked fetch of an unrouted page was not. That does not change any figure here, because every count below is from the two-path period. It does mean a LATER corpus can separate 'a crawler checked permissions' from 'a crawler went looking for URLs to fetch', which this one cannot.", "SCOPE_THAT_BOUNDS_EVERY_ZERO": "The shim serves exactly two paths. A crawler recorded with 0 hits never fetched robots.txt or llms.txt in this window. That is NOT the same as never fetching a page, and this corpus does not claim it. A crawler that cached robots.txt earlier, or ignores it, is invisible here.", "results": { "total_non_curl_hits": 278, "by_file": { "robots.txt": 253, "llms.txt": 25 }, "ai_crawlers_that_actually_crawl": { "GPTBot": { "hits": 0, "robots_txt": 0, "llms_txt": 0, "documented_purpose": "training", "vendor_doc": "https://developers.openai.com/api/docs/bots", "vendor_quote": "used to crawl content that may be used in training our generative AI foundation models" }, "OAI-SearchBot": { "hits": 50, "robots_txt": 49, "llms_txt": 1, "first_seen": "2026-08-10T21:46:00+00:00", "last_seen": "2026-08-25T00:49:01+00:00", "documented_purpose": "search", "vendor_doc": "https://developers.openai.com/api/docs/bots", "vendor_quote": "used to surface websites in search results in ChatGPT's search features" }, "ClaudeBot": { "hits": 0, "robots_txt": 0, "llms_txt": 0, "documented_purpose": "training", "vendor_doc": "https://support.claude.com/en/articles/8896518", "vendor_quote": "collecting web content that could potentially contribute to their training" }, "Claude-SearchBot": { "hits": 0, "robots_txt": 0, "llms_txt": 0, "documented_purpose": "search", "vendor_doc": "https://support.claude.com/en/articles/8896518", "vendor_quote": "navigate the web to improve search result quality for users" }, "PerplexityBot": { "hits": 0, "robots_txt": 0, "llms_txt": 0, "documented_purpose": "search", "vendor_doc": "https://docs.perplexity.ai/guides/bots", "vendor_quote": "It is not used to crawl content for AI foundation models" } }, "not_crawlers_at_all": { "Google-Extended": { "documented_purpose": "training-use control token", "governs_the_crawler": "Googlebot", "vendor_doc": "https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers", "vendor_quote": "a standalone product token, controlling whether crawled content may be used for training generative AI models", "hits": 0, "why_the_zero_means_nothing": "This token does not make requests, so it can never appear in a request log. Its zero is definitional and is excluded from the headline count.", "the_crawler_it_governs_did_appear": 26 }, "Applebot-Extended": { "documented_purpose": "training-use control token", "governs_the_crawler": "Applebot", "vendor_doc": "https://support.apple.com/en-us/119829", "vendor_quote": "Applebot-Extended does not crawl webpages", "hits": 0, "why_the_zero_means_nothing": "This token does not make requests, so it can never appear in a request log. Its zero is definitional and is excluded from the headline count.", "the_crawler_it_governs_did_appear": 38 } }, "llms_txt": { "total_fetches": 25, "from_the_ai_crawlers_that_actually_crawl": 1, "every_fetcher": { "unidentified": 10, "Googlebot": 1, "YandexBot": 2, "AhrefsBot": 1, "Applebot": 3, "OAI-SearchBot": 1, "Baiduspider": 3, "PetalBot": 2, "BuiltWith": 1, "DotBot": 1 } } }, "known_limits": { "one_site_one_window": "One domain, 14 days from 2026-08-10. A young low-authority domain is exactly where crawl budget is scarce, so these counts are not a general claim about these crawlers.", "absence_is_inference_not_measurement": "That a well-behaved crawler reads robots.txt before crawling is why a zero here is suggestive. That is an inference about crawler behaviour, not something this log measures.", "user_agent_is_self_reported": "Every identification is the UA string the client sent. Nothing here verifies a request claiming to be a given crawler came from it.", "llms_txt_is_not_a_standard": "llms.txt is a convention, not a specification any crawler is obliged to fetch. A low count is not non-compliance.", "two_of_the_seven_are_not_crawlers": "robots.txt names seven AI agents but only five of them fetch anything. Google-Extended and Applebot-Extended are training-use control tokens read by Googlebot and Applebot. They are counted separately here, and both of the crawlers they govern DID crawl this site.", "a_crawler_is_not_an_answer_engine": "Being fetched is not being cited, and not being fetched here does not mean a model cannot surface this site by other routes." } }