Technical Foundations · 1.6
Robots.txt
A file telling search engines which parts of your site they're allowed to look at, and which to skip — for example, deliberately hiding an internal staff testing area, but potentially, by accident, hiding your entire live website instead.
Why this matters
The customer’s side
Invisible directly — but if this is misconfigured, they'll experience the consequence as total invisibility: they search for your business, and you simply never appear, no matter how good everything else on your site is. It can also hide specific pages if configured wrong — the contact page, for example, quietly costing you leads nobody ever traces back to this file.
The search engine’s side
Can block your whole site, not just rank it lower
Getting this wrong can accidentally hide your entire site, or specific pages, from search engines — not a minor ranking penalty, a total block on being seen at all.
See it happen
Same file. One word is the entire difference between visible and invisible.
robots.txt — one line wrong
Disallow: /
Every page on the site, blocked from every search engine. Nobody's typo shows up as an error — the site just stops appearing.
robots.txt — correct
Allow: /
Same file, same structure, one word different. Fully visible to every search engine.
How to check yours
Method 1 — Check the file directly
the file is always public, so this works for any site.
- 1.Visit yourwebsite.com/robots.txt in your browser (e.g. reyselettings.co.uk/robots.txt).
- 2.Paste the contents into ChatGPT or Claude and ask if anything important is being blocked.
- 3.A clean result looks like “Allow: /” with no “Disallow:” lines blocking anything important — generally, the less restrictive, the better.
Method 2 — Google Search Console's URL Inspection tool
confirms whether a specific page — your homepage, or a key listing — is actually blocked, rather than reading the file yourself and hoping you've interpreted it right.
- 1.Go to Google Search Console and select your property.
- 2.Paste a specific page's URL into the “Inspect any URL” bar at the top.
- 3.Check the result doesn't say “Blocked by robots.txt.”
How to fix what you find
Keeping on top of it
Worth a re-check whenever a developer touches the site, or after any major rebuild — a single accidental line here can undo everything else in this entire guide.
How Reyse handles this
Done With You
we check it and tell you exactly what to fix if something's wrong.
Done For You
we monitor this continuously and catch any accidental change immediately.