Is Cloudflare Bad for SEO? How Bot Rules Can Deindex Your Site

We see this same case at least once a month now.

A business owner calls us because their organic traffic dropped off a cliff. No Google penalty, no algorithm update, no lost backlinks. Google Search Console shows Google has dropped a large share of pages from the index, with the error “Blocked due to access forbidden (403)” in the page indexing report. Crawl logs show 4xx errors everywhere. The site is telling Google to go away.

Fig 1 – Screenshot from Google Search Console showing ‘Blocked due to access forbidden (403)’ error in page indexing report

The cause? A single toggle in their Cloudflare dashboard.

If your website sits behind Cloudflare, and about 24% of all websites do, you need to read this. A security setting you turned on to protect your content from AI scrapers might be the reason Google can no longer crawl your pages.

With a major Cloudflare policy change taking effect on September 15, 2026, the problem gets worse for anyone who does not act.

Cloudflare Is Not Bad for SEO. Bad Configuration Is.

Cloudflare, when set up correctly, is good for SEO. The CDN speeds up delivery, SSL certificates come free on every plan, and DDoS mitigation keeps your site reachable during attacks. Each of these helps rankings.

The Cloudflare SEO problems people run into come almost entirely from their own configuration. Settings designed to block bad bots often end up blocking search engine crawlers as well.

We have worked with hundreds of websites over the past decade, and the sequence plays out the same way each time. Someone on the team enables a security feature, nobody checks whether Googlebot can still access the site, and weeks later the traffic chart looks like it fell off a table.

The difference between a site that ranks well on Cloudflare and one that drops out of Google entirely comes down to 3 specific features. Bot Fight Mode, the Block AI Bots toggle, and WAF rules. Get those wrong and you will have Cloudflare SEO issues that no amount of content or link building can fix.

3 Cloudflare Features That Cause SEO Problems

1. Bot Fight Mode and Super Bot Fight Mode

Bot Fight Mode is a free Cloudflare feature that automatically challenges traffic it thinks is automated. Super Bot Fight Mode, available on Pro plans and above, gives you more control over what happens to suspected bots.

The problem? These features are blunt instruments. Cloudflare built them to stop credential stuffing, inventory hoarding, and scraping attacks. They do not always tell the difference between a malicious bot and Googlebot.

Multiple site owners have noticed that Bot Fight Mode intercepted Googlebot with zero warning in Search Console. The site owner sees no errors anywhere. Pages simply stop getting indexed.

As Search Engine Roundtable reported, one site owner recently found that their managed IT provider had toggled on crawl controls to stop all bots. The result? Google could not reach the site for 2 weeks, and traffic cratered.

Fig 2 – Screenshot from Cloudflare showing Bot Fight Mode toggle that is turned on to block bot traffic on the website.

2. The “Block AI Bots” Toggle

In July 2025, Cloudflare launched a single toggle to block AI crawlers. Cloudflare positioned it as a way for publishers to protect their content from training large language models. The intent was sound. The execution was blunt. The toggle blocked every AI crawler, including the ones that power AI search features like ChatGPT Search, Google AI Overviews, and Perplexity.

If you turned this on and forgot about it, you may have been invisible in AI search results for months without realizing it.

That toggle is about to cause an even bigger problem. More on that in the next section.

Fig 3 – Screenshot from Cloudflare showing Configuration block where you configure AI bot policies for your website.

3. WAF Rules That Fire Too Broadly

Cloudflare’s Web Application Firewall lets you create custom rules based on user agents, IP ranges, ASNs, and threat scores. This works well for blocking real attacks. We have also seen site owners write rules that accidentally block every request with a suspicious user agent, catch search engine crawlers by IP range, or set threat score thresholds so low that legitimate crawlers get challenged.

The tricky part is that WAF blocks are invisible in Google Search Console. Your coverage report might show everything as fine because Google simply stops trying after getting blocked.

Fig 4 – Screenshot from Cloudflare showing Security Rules block where you set up custom rules for your website.

Fig 5 – 3 Cloudflare settings that can quietly deindex your website.

The September 15, 2026 Update That Changes Everything

On July 1, 2026, Cloudflare announced a major overhaul of how it handles AI crawlers. The changes are live now, and a critical set of new defaults takes effect on September 15, 2026.

Here is what is changing and why it matters for your SEO.

3 New Crawler Categories

Cloudflare is replacing the old “block all AI bots” approach with 3 separate categories.

• Search – Crawlers that index your site to answer questions later. This is the traditional index and rank behavior.
• Agent – Bots that act in real time on behalf of a user. Think ChatGPT browsing the web for you, or Gemini doing research inside Chrome.
• Training – Crawlers that pull your content to train or adapt AI models.

This is a smart categorization. The real issue arises when a single crawler does more than one of these things.

The Problem With Crawlers That Do Two Jobs

This is where the Cloudflare SEO issues get serious. Googlebot is a crawler with multiple jobs. It indexes your site for Google Search and it collects data that feeds AI training for features like AI Overviews. Applebot and Bingbot work the same way.

Starting September 15, Cloudflare will apply the most restrictive applicable rule to any crawler that does more than one job.

That means if you have blocked Training (either through the new controls or through the old “Block AI Bots” toggle), Cloudflare will also block Googlebot on those pages. Because Googlebot does both search and training, the training block applies to the entire crawler.

This is not theoretical. A site owner on Reddit confirmed that setting AI Training to “Block” caused both Googlebot and Bingbot to receive HTTP 403 errors when trying to fetch the sitemap. Google’s John Mueller personally responded to investigate the issue and noted that the defaults will apply to new Cloudflare customers, new sites for existing customers, and all existing free tier customers who have not changed their settings by September 15, 2026.

Who Is Affected by the September 15 Defaults

How to Check If Cloudflare Is Hurting Your SEO Right Now

You do not need to wait for September 15 to find out if you have a problem. Here is the process we run for every client using Cloudflare.

Step 1. Check your Cloudflare AI Crawlers & Scrapers settings

Log into your Cloudflare dashboard. Go to Security, then look for the AI Crawlers & Scrapers section. Check whether any of the 3 categories (Search, Agent, Training) are set to Block. If Training is blocked, check whether Googlebot appears in the blocked list.

Step 2. Test with Google Search Console

Use the URL Inspection tool in Search Console to test a handful of your most important pages. If Googlebot cannot access them, you will see a “Blocked by robots.txt” or a server error. Also test your sitemap URL to make sure it is accessible.

Step 3. Run a curl test from outside your network

Run this command to simulate a Googlebot request.

curl -I -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com

If you see a 403 response or a cf-mitigated header, Cloudflare is blocking Googlebot.

Step 4. Review Bot Fight Mode and Super Bot Fight Mode

Under Security > Bots, check whether Bot Fight Mode or Super Bot Fight Mode is enabled. If it is, make sure verified bots (which include Googlebot) are set to Allow, not Challenge or Block.

Step 5. Audit your WAF custom rules

Go to Security > WAF and review every active rule. Look for rules that target user agents, IP ranges, or ASNs broadly enough to catch search engine crawlers. Document each rule and test its impact.

Fig 6 – Run this 5 step audit to catch Cloudflare SEO problems before they cost you traffic.

What to Do Before September 15, 2026

If you are using Cloudflare, here is your action plan.

• If you want search engines to keep crawling your site – Make sure Search is set to Allow. If you are blocking Training, understand that Googlebot will also be blocked under the strictest rule policy. You may need to explicitly allow Search crawlers alongside a Training block.
• If you are on the free tier and have not touched your settings – Cloudflare will change your settings to the new defaults on September 15. Log in and set them manually before that date so you stay in control.
• If you turned on “Block AI Bots” last year and forgot about it – Go check right now. The new strictest rule logic now covers that old toggle. It will block Googlebot, Applebot, and Bingbot on your site.
• If you want to block AI training and keep Google Search – This is the hard case. Googlebot combines both functions in a single crawler, so blocking one through Cloudflare blocks the other. Google offers Google-Extended as a robots.txt directive. It does not prevent your content from appearing in AI Overviews. The real fix needs to come from Google separating its crawlers, which Cloudflare is pushing for.

When Cloudflare Helps Your SEO

We do not want to leave the impression that Cloudflare is a risk to avoid. When configured correctly, it is one of the better things you can do for your site’s technical SEO.

• Page speed improvements – Cloudflare’s CDN serves your content from edge servers close to the user. Faster page loads mean better Core Web Vitals scores, and Google uses those as ranking signals.
• DDoS protection – If your site goes down from an attack, Google will stop crawling and your pages will eventually drop from the index. Cloudflare prevents that.
• Free SSL – HTTPS is a confirmed ranking factor. Cloudflare provides SSL certificates on all plans, including free.
• DNS speed – Cloudflare’s DNS is among the fastest in the world. Faster DNS resolution means faster time to first byte, which matters for crawl efficiency.

The takeaway is simple. Cloudflare is not bad for SEO. Ignoring your Cloudflare configuration is. The same tool that speeds up your site and protects it from attacks can also cut it off from Google entirely if you do not manage the settings with SEO in mind.

AI Search Visibility Depends on Getting This Right

AI search now matters as much as traditional Google rankings. Google AI Overviews, ChatGPT Search, Perplexity, and Claude all pull from web content to generate answers. If your Cloudflare settings block the crawlers that feed these systems, you are invisible in a channel that grows more important every month.

Our guides on What is Generative Engine Optimization and the GEO Implementation Checklist walk through the best practices for visibility in AI search results in 2026.

Frequently Asked Questions