AI bots in robots.txt, firewalls and Cloudflare: how not to block them by mistake

Many websites block ChatGPT, Perplexity or Claude without their owners knowing – in robots.txt, in the hosting firewall or through Cloudflare. I’ll show you which AI bots exist, which to allow, how to set it up and how to check it.

You can have the best content, structured data and reviews. But if an AI bot can’t load your website, it can’t recommend you. And blocking often happens silently – without you setting anything. That’s exactly what I had to deal with on my own website, where the hosting protection was rejecting “unknown” bots.

Two kinds of AI bots

Every big AI company now runs several bots with different purposes:

• Search and answer bots load a page when a user asks for current information. Recommendations and visits come from them. They include OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User (Anthropic) and Bingbot, which Copilot and ChatGPT both draw on.
• Model training bots collect text to train future AI versions: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Applebot-Extended. Google-Extended is not a separate bot but a token that tells Google whether it may use your content for Gemini.

For a small business that wants to be recommended, I suggest allowing all of them. If you don’t want models trained on your content, block only the training bots – never the search ones.

1. robots.txt

You’ll find the file at yourcompany.com/robots.txt. This is a setup that explicitly allows AI bots:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /admin/

Sitemap: https://yourcompany.com/sitemap.xml

What to watch out for:

• The line Disallow: / under User-agent: * blocks the whole site for every bot without its own section. It is often left on after a website is built.
• WordPress has the option “Discourage search engines from indexing this site” in Settings → Reading. It must be off.
• Some SEO plugins add AI bot blocking as “content protection”. Check their settings.

2. Hosting firewall and protection

robots.txt is only a request – the bot reads it and decides. A hosting firewall (WAF) stops the bot before that, and robots.txt doesn’t matter then. Typical signs:

• AI bots get a 403 error or an “I’m not a robot” check page.
• Bing Webmaster Tools reports crawl errors.
• Server logs show requests from GPTBot or PerplexityBot with status 403 or 429.

The fix: find bot protection (bot protection, anti-bot, WAF) in your hosting admin and allow verified search engine and AI bots. If you can’t do it there, write to your hosting support and send them the list of bots from this article.

3. Cloudflare

If your website runs through Cloudflare, pay special attention. Since July 2025 Cloudflare blocks AI bots by default on new domains and also offers “Bot Fight Mode”, which can stop the useful ones too.

• In Security → Bots, check the “Block AI bots” option and choose whether to block only training bots or nobody.
• Go through AI Crawl Control (formerly AI Audit): you’ll see which AI bots visit your site and can allow or block each one separately.
• If Bot Fight Mode is on, make sure verified bots get through.

How to check it

• Free website test: the website test with the AI visibility area checks robots.txt for you and shows which AI bots you block.
• Server logs: look at the access logs in your hosting admin and search for GPTBot, OAI-SearchBot, PerplexityBot. Status 200 is fine, 403 means blocking.
• Perplexity: ask about the content of a specific page on your website. If it says it can’t load the page, something is blocking it.

Once the door is open for AI, it makes sense to continue with the next steps: llms.txt, Bing Webmaster Tools and tracking AI traffic in GA4.

Don’t want to dig through hosting and Cloudflare settings yourself? Write to me via the free audit – I’ll check whether AI can see you and set it up.