I came across this feature request about allowing AI bots access despite Cloudflare blocking them by default, and I completely agree this is becoming increasingly important.
I run a content site at https://7brewcoffemenu.com and I’ve noticed that AI systems (like ChatGPT and other assistants) are now a major discovery channel for informational websites. When AI crawlers are blocked at the edge, it can significantly reduce visibility and organic discovery of content.
Having a simple, optional toggle or whitelist for trusted AI bots would be extremely valuable — especially for sites that want to be indexed and surfaced in AI-generated answers. Ideally, this could be something like a “Allow AI crawlers for discovery/SEO” setting, similar to how other bot controls are being added in Cloudflare Enterprise integrations.
This would help site owners like me ensure our content remains accessible to modern AI-driven search and recommendation systems while still keeping security controls in place.
anna fraud
Hi team,
I came across this feature request about allowing AI bots access despite Cloudflare blocking them by default, and I completely agree this is becoming increasingly important.
I run a content site at https://7brewcoffemenu.com and I’ve noticed that AI systems (like ChatGPT and other assistants) are now a major discovery channel for informational websites. When AI crawlers are blocked at the edge, it can significantly reduce visibility and organic discovery of content.
Having a simple, optional toggle or whitelist for trusted AI bots would be extremely valuable — especially for sites that want to be indexed and surfaced in AI-generated answers. Ideally, this could be something like a “Allow AI crawlers for discovery/SEO” setting, similar to how other bot controls are being added in Cloudflare Enterprise integrations.
This would help site owners like me ensure our content remains accessible to modern AI-driven search and recommendation systems while still keeping security controls in place.
Thanks for considering this improvement.