0498 70 98 60AI Crawlers Blocked
AI Crawlers Blocked From Your Website? Make Sure It’s on Purpose
Your business never appears in ChatGPT search results or Perplexity answers, even though you rank on Google. One possible cause: AI crawlers blocked from your website by your robots.txt, a security plugin, your host or your CDN, sometimes without you knowing. Blocking some AI bots can be a deliberate choice; blocking all of them by accident can cost you visibility in AI search.
Signs AI crawlers can’t reach your site
AI assistants describe your business from directories or old articles, never from your own pages
ChatGPT search or Perplexity cite competitors’ sites but never yours, even for your brand name
Your server logs show no visits from bots such as OAI-SearchBot, or only blocked requests (403 errors)
Your CDN or security plugin dashboard reports AI bots as blocked
AI crawlers blocked without you knowing: common causes
- A robots.txt rule copied from a template that disallows bots like GPTBot or OAI-SearchBot
- A security plugin or web application firewall treating AI bots as threats
- CDN settings: Cloudflare started blocking known AI crawlers by default for new domains in July 2025. Since September 15, 2026, its settings separate Search, Agent and Training bots, and new domains block Training and Agent bots on pages with ads by default. Crawlers that serve both search and training can be caught by a training block.
- Host-level bot protection or rate limiting
- Pages that only render with JavaScript, which some crawlers can’t read well
Know the bots: training vs search
AI companies use different crawlers for different purposes. OpenAI, for example, documents GPTBot (used for model training), OAI-SearchBot (used to surface websites in ChatGPT search results) and ChatGPT-User (used when a user’s request makes ChatGPT visit a page). Google uses the Google-Extended token to control use of your content for its Gemini models; it doesn’t affect Google Search or AI Overviews, which rely on Googlebot. Other providers such as Anthropic and Perplexity publish their own crawler names. You can allow search-related crawlers while refusing training ones.
What to do right now
- Open yourdomain.com/robots.txt and look for rules naming AI user agents or a blanket Disallow: / for all bots.
- Check your CDN and firewall settings (Cloudflare, Sucuri, Wordfence, your host) for AI bot blocking options.
- Decide your policy: which AI tools should be able to read your content to cite you, and which ones you’d rather exclude.
- Don’t block Googlebot while trying to block AI bots: it would remove you from Google Search.
What we do
- Audit robots.txt, CDN, firewall and server logs to see which AI crawlers reach your site
- Define a clear policy with you (allow search and citation bots, decide on training bots)
- Implement it consistently across robots.txt and security tools
- Improve how your content is read by crawlers (server-rendered content, structured data)
- Combine with AI search optimization (GEO) to improve your chances of being cited, and see get mentioned in ChatGPT
How long does it take?
The audit and configuration changes can usually be done in a day or two. AI tools then need to recrawl your pages, at their own pace.
Send us your URL. We’ll check whether AI crawlers can access your site and help fix it.
Your questions, answered
It depends on your goals. Blocking training crawlers limits how your content is used for model training; blocking search crawlers can reduce your visibility in AI answers. Many businesses allow search bots and decide case by case for training bots.
OpenAI documents separate crawlers: GPTBot for training and OAI-SearchBot for search results. Check which ones your rules affect.
It can. Cloudflare has blocked known AI crawlers by default for new domains since July 2025, and since September 2026 its settings let you treat Search, Agent and Training bots separately. Check your dashboard rather than assuming.
Usually not noticeably. If a crawler is too aggressive, rate limiting is better than a full block. See hosting resource limit reached.