Skip to content

GEO courseModule 3 of 10: Can AI reach you?Lesson 3.1

Let the right AI crawlers in: robots.txt, CDN and firewalls

AI companies send automated visitors (crawlers) to read websites. Which ones you let in decides whether AI tools can read your site, use it in their search answers, or fetch it when someone asks about you. Your robots.txt file states your wishes, but a CDN, firewall or security plugin in front of your site can overrule it without anyone deciding to.

5.75% of your GEO effort7 min readAbout 15 minutes, twice a year and after any site change

By Morgan King, founder of Reyse. Facts checked October 2026. Jump to Your turn

Why it matters

If AI search crawlers can’t read your site, those tools can’t use it in their answers, however good your agency is. Most blocks are accidents.

The evidence

Evidence: Official guidance

What each crawler does comes from OpenAI, Anthropic, Perplexity, Apple and Google’s own pages. That letting search crawlers in leads to more mentions is likely but unproven for agents.

Do this first

Open yourdomain.co.uk/robots.txt and look for “Disallow: /” under User-agent: * or under OAI-SearchBot, Claude-SearchBot or PerplexityBot.

What it is

Your robots.txt is the guest list on the door. A security guard (your CDN or firewall) stands outside it. If the guard has been told to turn away anything that looks like a robot, the guest list never gets read.

Think of it like

A shop that’s open, lights on, stock out, with a “closed” sign on the door that nobody remembers putting there. People who look inside can see it’s open. The guidebook writer outside just walks past.

Three crawlers, three jobs. Switch them on and off and watch which one decides whether you can appear.

OAI-SearchBot

Search: builds the index behind ChatGPT search

Can appear in ChatGPT search answers

ChatGPT-User

User request: visits a page when someone asks

Can open your page when someone asks about you

GPTBot

Training: collects pages for future models

Not used for training. Search is unaffected

your-agency.example/robots.txt

User-agent: OAI-SearchBotAllow: / User-agent: ChatGPT-UserAllow: / User-agent: GPTBotDisallow: /

Blocking training is a separate choice from blocking search. Only the search crawler decides whether you can appear.

Why it matters to the people searching

Your customers never see . They only see the result: ask ChatGPT “who are the best letting agents in [town]?” and an agency whose site it can’t read is far less likely to be in the answer or linked as a source.

  • buyers

    Ask AI which agents cover an area and what prices look like. An unreadable site can’t be linked.

  • tenants

    Ask about rents and agents. Your rentals pages can’t be quoted if search crawlers are blocked.

  • sellers

    Ask “best estate agents in [town]” before choosing who to invite. A site AI can’t read is less likely to be in that answer.

  • landlords

    Ask about letting fees. An unreadable fees page can’t be quoted.

To Google, Bing and AI

Evidence: Official guidance

Every AI crawler does one of three jobs. Search crawlers build the index behind an AI search feature: OAI-SearchBot (ChatGPT), Claude-SearchBot, PerplexityBot. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers. User-request fetchers visit a page because a person asked: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity say robots.txt may not apply to these. Training crawlers collect pages that may train future models: GPTBot, ClaudeBot. Blocking training doesn’t remove you from search. Google’s and are part of Google Search, so Googlebot is what matters there. Google-Extended is a separate setting for Gemini training and that, Google says, doesn’t affect Search. Above all of this sits your CDN: Cloudflare now has separate Search, Agent and Training switches, and announced that from 15 September 2026 new domains block Training and Agent bots by default on pages that show ads.

Crawler jobs: search, user requests and training. Only blocking search removes you from answers
3Crawler jobs: search, user requests and training. Only blocking search removes you from answers
About how long OpenAI says a robots.txt change takes to apply
24 hoursAbout how long OpenAI says a robots.txt change takes to apply

An agent example

A two-branch agency relaunches its website. The developer’s staging robots.txt, which blocked every crawler, goes live with it. Nobody notices, because the site works for people. Six months later the director asks AI assistants for agents in the town and the agency never appears. The fix takes two minutes; finding it took a robots.txt check.

Your turn: check yours, then fix it

Illustration
Illustration of a CDN dashboard with three AI crawler groups: Search allowed, Agent allowed and Training blocked. Cloudflare’s AI crawler settings, in three groups. Keep Search allowed, and decide Agent and Training on purpose. Menu names change, so check Cloudflare’s own help. Example numbers.
  1. 1Type yourdomain.co.uk/robots.txt into your browser. “Not found” means nothing is blocked by it.
  2. 2Look for User-agent lines naming OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Applebot or Google-Extended, and read the line under each. Disallow: / means blocked.
  3. 3Look for “User-agent: *” followed by “Disallow: /”. That blocks every polite crawler, search engines included.
  4. 4Check every version of your site: www, without www, and any separate property search subdomain each have their own file.

  1. 1If your site uses Cloudflare, open Security and look for the AI crawler settings (Search, Agent and Training). Check Search is allowed.
  2. 2Check any security plugin (Wordfence, Sucuri and others) for rules that block bots or “unknown automated visitors”.
  3. 3Check for UK-only geoblocking or harsh rate limits. AI crawlers may not visit from UK addresses.

  1. 1Send your web provider the crawler log request from lesson 2.1.
  2. 2A 200 response means the page was served. 403 means refused. 429 means too many requests.
  3. 3Anyone can fake a crawler’s name, so a technical person can check visits against the IP lists OpenAI, Anthropic and Perplexity publish.

  1. 1Paste one of your page addresses into ChatGPT, Claude and Perplexity.
  2. 2Ask: “Open this page and tell me what fees it lists.”
  3. 3A refusal or error can point to a block, though it can have other causes. Repeat on two or three pages.

Copy this prompt

Here is my robots.txt. Which AI crawlers does it allow and block? Explain in plain English, separate search crawlers, user-request crawlers and training crawlers, and flag anything that looks accidental, such as a blanket block.

Rather we did this for you? It’s part of Found First.

Keep it up

Check after any relaunch, new provider, new security plugin or CDN change, and review the policy twice a year. Security tools update their defaults and can re-block crawlers without you touching anything.

About 15 minutes, twice a year and after any site changeSee it in your schedule
Go deeperrobots.txt is a request, not a lock

Reputable crawlers follow it. User-request fetchers may not. It’s also a public file, so never list anything private in it.

It’s not a legal shield either. The UK government’s March 2026 report on copyright and AI made no change to the law, so there’s no legislated opt-out from AI training. Treat blocking training crawlers as a business choice.

If you have no rules for Applebot but do have rules for Googlebot, Apple says Applebot follows the Googlebot rules, so a rule written for Google can affect Siri and Spotlight.

Sources

  1. 1.Overview of OpenAI crawlers, OpenAI
  2. 2.Does Anthropic crawl data from the web?, Anthropic
  3. 3.Perplexity crawlers, Perplexity
  4. 4.Introduction to robots.txt, Google Search Central
  5. 5.AI features and your website, Google Search Central
  6. 6.Report and impact assessment on copyright and artificial intelligence, GOV.UK, March 2026

Facts checked October 2026, and re-checked every quarter. Spotted something out of date? Tell us.