Technical Foundations · 1.1
AI Crawler Access
Whether the specific web crawlers that power AI search — OpenAI's, Perplexity's, and others — are actually allowed to read your website at all. Imagine putting up a "closed" sign on your shop door without realising it, while still technically being open for business inside.
Why this matters
The customer’s side
Invisible to them directly — but the consequence is total: if an AI crawler can't read your site, it simply cannot mention you, no matter how good your business genuinely is.
The AI’s side
Confirmed: OpenAI says a blocked site never appears at all
Roughly 1 in 4 business websites accidentally block the crawlers that power AI citations, usually through overly aggressive security settings that treat a legitimate AI crawler the same as a malicious bot. OpenAI states plainly that a blocked site will never appear in its search answers — not ranked lower, genuinely absent.
See it happen
Same file. One line is the entire difference between an AI crawler reading you, or not.
robots.txt — one line wrong
Disallow: /
Every page on the site, blocked from every search engine. Nobody's typo shows up as an error — the site just stops appearing.
robots.txt — correct
Allow: /
Same file, same structure, one word different. Fully visible to every search engine.
How to check yours
Method 1 — Check robots.txt directly
the file is always public, so this works for any site.
- 1.Visit yourwebsite.com/robots.txt.
- 2.Check for any line blocking user-agents like GPTBot, PerplexityBot, ClaudeBot, or Google-Extended.
Method 2 — Check your CDN or security settings
robots.txt isn't the only place this happens — security and CDN tools often block AI crawlers by default, separately from robots.txt entirely.
- 1.If you use Cloudflare or a similar security/CDN service, log in and check its bot-management settings specifically.
- 2.Paste both findings into ChatGPT or Claude and ask directly: "is anything here blocking a major AI crawler?"
How to fix what you find
Blocked in robots.txt
edit the file directly to remove the line disallowing the crawler.
- 1.Access robots.txt through your site's file manager, FTP, or — if you're on WordPress — an SEO plugin like Yoast or Rank Math that lets you edit it from the dashboard.
- 2.Find the line naming the blocked user-agent (e.g. "User-agent: GPTBot" followed by "Disallow: /") and delete that block, or change "Disallow: /" to "Allow: /".
- 3.Save, then reload yourwebsite.com/robots.txt in a browser to confirm the line is gone.
Blocked by Cloudflare or a similar CDN/security service
these tools block AI crawlers at a level robots.txt can't override, through their own bot-management settings.
- 1.Log into Cloudflare (or your CDN provider) and find "Bots" or "Security" in the sidebar.
- 2.Look for a setting called "AI Bots," "Verified Bots," or "Known Bots," and check whether AI crawlers are set to "Block."
- 3.Switch it to "Allow," then re-test using the robots.txt check and, if possible, your CDN's own "recent blocked requests" log to confirm the crawler is getting through now.
Blocked by your hosting provider's firewall or a security plugin
a security plugin like Wordfence or Sucuri, or a locked-down hosting firewall, can block crawlers independently of both robots.txt and Cloudflare.
- 1.Check your security plugin's firewall or bot-blocking rules for anything treating unfamiliar user-agents as suspicious by default.
- 2.Add GPTBot, ClaudeBot, PerplexityBot, and Google-Extended to its allow-list if such a list exists.
- 3.If you're not confident making this change yourself, ask your host's support team directly: "are AI crawlers like GPTBot allowed through your firewall?"
Not sure which layer is actually blocking it
paste what you've found into AI and let it narrow it down for you.
- 1.Copy your robots.txt content and a screenshot or description of your CDN/security settings.
- 2.Paste both into ChatGPT or Claude and ask: "based on this, what specifically is blocking AI crawlers, and what exactly do I need to change?"
Keeping on top of it
Security settings and CDN providers update their defaults periodically, sometimes re-blocking crawlers without you changing anything yourself — worth a check every couple of months.
How Reyse handles this
Done With You
we check this for you and tell you exactly what to change, and where.
Done For You
we monitor this continuously and fix it the moment anything changes.