seoHow a Firewall Rule Blocked AI Crawlers That robots.txt Allowed
A firewall rule written to stop fake browsers blocked Amazonbot and Meta's AI crawler on every HTML page of a group of sites for more than a week, while robots.txt allowed both crawlers and every llms.txt check came back green. The note covers how that kind of rule goes wrong, why the usual tests miss the damage, and how to check a rule before it blocks anything.
How a Fake-Browser Rule Works
A fake-browser rule compares what a request claims to be with what it actually sends: real browsers send a set of browser-only request headers with every page visit, so a request that claims to be a Mozilla browser but lacks those headers is usually a script.
The logic is sound for scrapers, and the trouble starts with crawlers. Several legitimate crawlers identify themselves with Mozilla-style user agents and do not send browser headers, so to the rule they look exactly like a fake browser. The table below shows how three kinds of visitors look to such a rule.
| Question the rule asks | Real browser | AI crawler with a Mozilla-style user agent | curl test |
|---|---|---|---|
| Sends the headers every real browser sends | Yes | No | No |
| Calls itself a Mozilla browser | Yes | Yes | No |
| What the rule decided | Real visitor, allowed | Fake browser, blocked | Not a browser, ignored |
What the Rule Caught by Mistake
Amazonbot and Meta's meta-externalagent crawler both identify themselves with Mozilla-style user agents, neither sends browser-only headers, and neither was on the rule's list of known crawlers, so both received a 403 on every HTML page.
Common Crawl's CCBot and You.com's YouBot fit the same pattern. The same kind of rule can also catch the validation requests of a certificate authority, since Let's Encrypt's validation server announces itself as a Mozilla-compatible client; a rule like this has to leave the certificate validation path alone.
Why the Usual Checks Missed the Block
Three blind spots hid the block for days: the files people check first end in extensions, test requests don't look like crawlers, and the CDN rewrote each block into a 404 page before it reached the logs.
| Blind spot | Why it hid the block | Check that catches it |
|---|---|---|
| Extensions | llms.txt and sitemap.xml end in extensions the rule skipped, so those checks passed | Test an extensionless page, not only the text files |
| Test clients | curl and uptime monitors do not claim to be a browser, so the rule never fired for them | Test with each crawler's real user agent |
| Error rewrites | The CDN turned each block into a 404 page, so the logs showed 404s | Check the CDN's custom error mapping |
The Fix and the Lesson
The fix added the crawlers that had been caught to the rule's list of known crawlers, kept machine endpoints and certificate validation out of the rule, and confirmed with live requests that each crawler received its pages again.
A list of crawler names is only a first step, because any client can send any user agent: the log study behind the llms.txt note found that about a third of requests carrying AI crawler names were fake. The stronger check verifies crawlers by the IP ranges that OpenAI, Anthropic, and Google publish, or through a managed bot-control service that does the same.
Checklist Before Switching a Bot Rule to Block
A bot rule can be tested for crawler damage in an afternoon, and the checks below would have caught this rule before it reached blocking mode.
- List every named crawler the rule matched in count mode, by user agent and by verified IP range, and decide on each one before switching to block.
- Request extensionless pages with each important crawler's real user agent.
- Keep certificate validation paths and machine endpoints out of browser heuristics.
- Check the CDN's custom error mapping, so a block can't hide behind a 404 page.
- Run a daily canary that requests a page as Googlebot, GPTBot, ClaudeBot, and Amazonbot and alerts on anything but a 200.
The same checks are part of every AI crawler and bot management engagement at seodima.com.
Need This Checked on Your Website?
Send your website address and the problem you see, and Dmytro Verzhykovskyi replies personally with a written assessment, a recommended scope, and a fixed price, with no obligation.
