GEO courseModule 3 of 10: Can AI reach you?Lesson 3.1
Let the right AI crawlers in: robots.txt, CDN and firewalls
AI companies send automated visitors (crawlers) to read websites. Which ones you let in decides whether AI tools can read your site, use it in their search answers, or fetch it when someone asks about you. Your robots.txt file states your wishes, but a CDN, firewall or security plugin in front of your site can overrule it without anyone deciding to.
By Morgan King, founder of Reyse. Facts checked October 2026. Jump to Your turn
Why it matters
If AI search crawlers can’t read your site, those tools can’t use it in their answers, however good your agency is. Most blocks are accidents.
The evidence
Evidence: Official guidance
What each crawler does comes from OpenAI, Anthropic, Perplexity, Apple and Google’s own pages. That letting search crawlers in leads to more mentions is likely but unproven for agents.
Do this first
Open yourdomain.co.uk/robots.txt and look for “Disallow: /” under User-agent: * or under OAI-SearchBot, Claude-SearchBot or PerplexityBot.
What it is
Your robots.txt is the guest list on the door. A security guard (your CDN or firewall) stands outside it. If the guard has been told to turn away anything that looks like a robot, the guest list never gets read.
Think of it like
A shop that’s open, lights on, stock out, with a “closed” sign on the door that nobody remembers putting there. People who look inside can see it’s open. The guidebook writer outside just walks past.
Three crawlers, three jobs. Switch them on and off and watch which one decides whether you can appear.
OAI-SearchBot
Search: builds the index behind ChatGPT search
Can appear in ChatGPT search answers
ChatGPT-User
User request: visits a page when someone asks
Can open your page when someone asks about you
GPTBot
Training: collects pages for future models
Not used for training. Search is unaffected
your-agency.example/robots.txt
User-agent: OAI-SearchBotAllow: / User-agent: ChatGPT-UserAllow: / User-agent: GPTBotDisallow: /
Blocking training is a separate choice from blocking search. Only the search crawler decides whether you can appear.
Why it matters to the people searching
Your customers never see . They only see the result: ask ChatGPT “who are the best letting agents in [town]?” and an agency whose site it can’t read is far less likely to be in the answer or linked as a source.
buyers
Ask AI which agents cover an area and what prices look like. An unreadable site can’t be linked.
tenants
Ask about rents and agents. Your rentals pages can’t be quoted if search crawlers are blocked.
sellers
Ask “best estate agents in [town]” before choosing who to invite. A site AI can’t read is less likely to be in that answer.
landlords
Ask about letting fees. An unreadable fees page can’t be quoted.
To Google, Bing and AI
Evidence: Official guidanceEvery AI crawler does one of three jobs. Search crawlers build the index behind an AI search feature: OAI-SearchBot (ChatGPT), Claude-SearchBot, PerplexityBot. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers. User-request fetchers visit a page because a person asked: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity say robots.txt may not apply to these. Training crawlers collect pages that may train future models: GPTBot, ClaudeBot. Blocking training doesn’t remove you from search. Google’s and are part of Google Search, so Googlebot is what matters there. Google-Extended is a separate setting for Gemini training and that, Google says, doesn’t affect Search. Above all of this sits your CDN: Cloudflare now has separate Search, Agent and Training switches, and announced that from 15 September 2026 new domains block Training and Agent bots by default on pages that show ads.
- Crawler jobs: search, user requests and training. Only blocking search removes you from answers
- 3Crawler jobs: search, user requests and training. Only blocking search removes you from answers
- About how long OpenAI says a robots.txt change takes to apply
- 24 hoursAbout how long OpenAI says a robots.txt change takes to apply
An agent example
A two-branch agency relaunches its website. The developer’s staging robots.txt, which blocked every crawler, goes live with it. Nobody notices, because the site works for people. Six months later the director asks AI assistants for agents in the town and the agency never appears. The fix takes two minutes; finding it took a robots.txt check.
Your turn: check yours, then fix it
- 1Type yourdomain.co.uk/robots.txt into your browser. “Not found” means nothing is blocked by it.
- 2Look for User-agent lines naming OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Applebot or Google-Extended, and read the line under each. Disallow: / means blocked.
- 3Look for “User-agent: *” followed by “Disallow: /”. That blocks every polite crawler, search engines included.
- 4Check every version of your site: www, without www, and any separate property search subdomain each have their own file.
- 1If your site uses Cloudflare, open Security and look for the AI crawler settings (Search, Agent and Training). Check Search is allowed.
- 2Check any security plugin (Wordfence, Sucuri and others) for rules that block bots or “unknown automated visitors”.
- 3Check for UK-only geoblocking or harsh rate limits. AI crawlers may not visit from UK addresses.
- 1Send your web provider the crawler log request from lesson 2.1.
- 2A 200 response means the page was served. 403 means refused. 429 means too many requests.
- 3Anyone can fake a crawler’s name, so a technical person can check visits against the IP lists OpenAI, Anthropic and Perplexity publish.
- 1Paste one of your page addresses into ChatGPT, Claude and Perplexity.
- 2Ask: “Open this page and tell me what fees it lists.”
- 3A refusal or error can point to a block, though it can have other causes. Repeat on two or three pages.
Here is my robots.txt. Which AI crawlers does it allow and block? Explain in plain English, separate search crawlers, user-request crawlers and training crawlers, and flag anything that looks accidental, such as a blanket block.
- 1Search crawlers: allow.
- 2User-request crawlers: allow (robots.txt may not stop them anyway).
- 3Training crawlers: your business choice. Some agencies block them on principle; others allow them, because an assistant answering from memory can only name a business it learned about. Neither choice has been shown to change mentions for agents.
- 4Save it with the date and who approved it. Review twice a year.
- 1Remove any blanket “Disallow: /” you didn’t intend.
- 2To block training only, add “User-agent: GPTBot” then “Disallow: /”, a blank line, then the same for ClaudeBot.
- 3Don’t add Google-Extended to that block unless you’ve decided to: Google says it also covers grounding in Gemini.
- 4Never block Googlebot or Bingbot as an “AI block”. That removes you from ordinary search too.
- 5WordPress: Yoast has a file editor under Tools; Rank Math has Edit robots.txt under General Settings.
- 1In Cloudflare, keep Search crawlers allowed and make a deliberate choice about Agent and Training. Be careful: blocking Training can also catch crawlers that do both jobs, such as Googlebot and Bingbot.
- 2In a security plugin, allow the search crawlers you chose by name, and by IP list where the tool supports it.
- 3Keep your attack protection on. The aim is to let the right visitors through, not to open the doors.
- 1Ask what robots.txt says, who can change it, and how long a change takes.
- 2Ask whether any firewall, CDN or bot protection blocks OAI-SearchBot, Claude-SearchBot, PerplexityBot or Bingbot.
- 3Ask for a log of refused requests from those crawlers.
Write a robots.txt that allows OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Bingbot and Googlebot, and blocks GPTBot and ClaudeBot. Keep this sitemap line: [paste]. Explain each line.
Then: After any change, reload robots.txt in a browser, re-run the open-a-page test, and ask your provider for a fresh log after a week. OpenAI says changes take about 24 hours; other systems vary.
Rather we did this for you? It’s part of Found First.
Keep it up
Check after any relaunch, new provider, new security plugin or CDN change, and review the policy twice a year. Security tools update their defaults and can re-block crawlers without you touching anything.
Go deeperrobots.txt is a request, not a lock
Reputable crawlers follow it. User-request fetchers may not. It’s also a public file, so never list anything private in it.
It’s not a legal shield either. The UK government’s March 2026 report on copyright and AI made no change to the law, so there’s no legislated opt-out from AI training. Treat blocking training crawlers as a business choice.
If you have no rules for Applebot but do have rules for Googlebot, Apple says Applebot follows the Googlebot rules, so a rule written for Google can affect Siri and Spotlight.
Sources
- 1.Overview of OpenAI crawlers, OpenAI
- 2.Does Anthropic crawl data from the web?, Anthropic
- 3.Perplexity crawlers, Perplexity
- 4.Introduction to robots.txt, Google Search Central
- 5.AI features and your website, Google Search Central
- 6.Report and impact assessment on copyright and artificial intelligence, GOV.UK, March 2026
Facts checked October 2026, and re-checked every quarter. Spotted something out of date? Tell us.
Worked through it? Mark it done and we’ll keep track on this device.