-
Cloudflare has been quietly blocking GPTBot and PerplexityBot on my site for months. Here’s how to check yours.
Posted by on April 21, 2026 at 12:07 pmSpent the morning debugging why my site wasn’t showing up in any AI search tools like ChatGPT, Perplexity, or AI Overviews. Everything looked fine on my end. My robots.txt in Next.js explicitly allowed every AI crawler.
Then I ran a curl on /robots.txt and saw this block that I definitely didn’t write:
# BEGIN Cloudflare Managed content User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / ...Turns out Cloudflare has a toggle called "AI Scrapers and Crawlers" that, when enabled, prepends its own rules to your robots.txt, effectively overriding whatever your app serves. For anyone doing GEO, meaning trying to get cited by ChatGPT, Perplexity, or AI Overviews, this can silently block you.
How to check in 10 seconds:
curl https://yoursite.com/robots.txt | grep "Cloudflare Managed"If you see it and you want AI crawlers indexing your site, go to the Cloudflare dashboard → Security → Bots → disable "AI Scrapers and Crawlers."
Or via API, set
is_robots_txt_managed: falseon the zone’sbot_managementendpoint.If you don’t want to appear in AI results, leave it on. It’s actually a useful default for sites that haven’t thought about this. Just be aware that it exists.
-
1 Reply
-
It wasn’t quiet. Cloudflare announced they were doing this months ago.
-
[removed]
-
We just finished checking, just in case, and no site on Cloudflare had this set as default. They were all off. I double-checked with the sysadmin, and he didn’t do anything, so it was off by default, and we’re talking about a bit over 1,000 websites.
That said, I remember this happened with a small batch at the very beginning, and they changed it because people complained. Maybe there were some leftovers?
-
Yes I’ve noticed this too and make sure to disable their filtering. Their on-page explanation for what this does is pretty ambiguous. I’m personally not concerned about AI scraping my website, and I definitely don’t want to NOT appear in citations.
I haven’t had any time to dig deeper into what this actually does. Would love if anyone with real knowledge explains.
-
[removed]
-
The toggle is by default off when you add your domain to cloudflare. so surely you enabled this setting yourself?
Log in to reply.