Investigator finds 639 websites block AI crawlers

hackernoon.com5 min read

An investigation into 640 websites revealed that nearly one in six automatically blocked AI crawlers without the site owners' knowledge or documentation. The author found that blocking often originates from multi-layered vendor defaults, such as Cloudflare's bot security settings or host-level IP reputation firewalls, rather than deliberate robots.txt rules. Testing requires checking behavior from multiple networks, examining both edge and origin logs, and auditing third-party security configurations. Open article →