seoWhat AI Crawlers Actually Do With llms.txt: Logs From 19 Sites
llms.txt is a markdown summary at /llms.txt that AI agents can read instead of whole web pages, proposed by Jeremy Howard in September 2024. On 19 sites that publish llms.txt, most of them with a plain-text /llms query endpoint added in August 2026, the server and CDN logs showed plenty of requests to those files, and only about one in eight came from AI crawlers whose IP addresses could be verified.
Which AI Crawlers Read llms.txt
Amazon's and Meta's crawlers did most of the reading among verified AI crawlers, well ahead of OpenAI's GPTBot and OAI-SearchBot, Google's GoogleOther, Anthropic's ClaudeBot, and PerplexityBot, with You.com's YouBot and Common Crawl's CCBot trailing.
Verification used IP addresses, not user agents: the published ranges for OpenAI's agents, Perplexity, Anthropic, and Meta, and reverse DNS lookups for Amazon, Google, Common Crawl, and You.com. The rest of the requests included the sites' own tests, scanners, other bots, and clients that only claimed to be AI crawlers.
How Many "AI Crawlers" Were Fake
Counting AI crawler visits by user agent alone overstates them, because a large share of the requests that use a crawler's name come from IP addresses the crawler's operator doesn't use.
Half of the requests that called themselves ClaudeBot came from outside Anthropic's ranges, and one residential IP address posed as both ClaudeBot and PerplexityBot. A dashboard that counts user agents would have reported AI interest that never existed, which is why the crawler findings in this note come from IP-verified requests only.
What Bots Did With the /llms Query Endpoint
The /llms endpoint on these sites takes a question in a query parameter and answers in plain text, and many of the query calls in the logs used the literal placeholder your_question, copied word for word from the example in the llms.txt file.
Only one crawler, Baiduspider-render, composed questions of its own, such as a question about a company founder's history, and none of its phrasings appeared on the site or in the file. The placeholder sat in four places: the llms.txt files, the longer llms-full.txt files, a hint in the page footers on thousands of pages, and the endpoint's usage line. Every copy was replaced with a real example question, and a daily monitor now checks that each site's example query gets a real answer.
What the Logs Can't Tell You
A fetch in the logs proves only that a crawler downloaded the file, not that an AI assistant used it in an answer, and Google states that its AI Overviews and AI Mode don't need AI text files at all, so llms.txt is a low-cost convenience rather than a ranking lever.
One site that published llms.txt without linking it from any page or from robots.txt logged only a handful of reads, far fewer than the others. The same site also sat behind a stricter bot-protection setting, so the gap can't be credited to the missing link alone. On several of the sites, a firewall rule also blocked the /llms endpoint for Amazonbot and Meta's crawler until the rule was fixed; the note on how a firewall rule blocked AI crawlers covers that failure.
Should You Publish llms.txt?
Publishing llms.txt takes little effort and does no harm, so it's worth doing for sites that want AI agents to read their facts, provided the file is accurate, free of placeholders, and reachable by the crawlers that ask for it.
- Serve llms.txt with a 200 to crawler user agents through the firewall and CDN, and test it the way crawlers request it.
- If the file documents a query endpoint, give a real example question that returns a real answer, never a placeholder.
- Verify AI crawler traffic by IP range before reporting it to anyone.
- Keep every fact in llms.txt identical to the facts on the pages it summarizes.
- Measure AI visibility where it shows up: AI Overviews and AI Mode in Search Console, and ChatGPT referrals in analytics.
llms.txt setup, crawler verification, and AI visibility tracking are part of AI SEO services at seodima.com.
Need This Checked on Your Website?
Send your website address and the problem you see, and Dmytro Verzhykovskyi replies personally with a written assessment, a recommended scope, and a fixed price, with no obligation.
