SEO courseModule 2 of 10: Technical FoundationsLesson 2.1
Can Google Find and Read Your Pages?
Crawling is Google visiting your page. Indexing is Google deciding to keep it in its catalogue. If either one fails, the page can't appear in search at all, however good it is.
Why it matters
If Google can't visit a page and keep it, that page can't appear in search at all, however good it is.
Why Google cares
A page that isn't indexed can't rank at all
Do this first
Run your valuation, sales and lettings pages through Search Console's URL Inspection tool.
01What it is
A librarian can’t recommend a book they can’t find on the shelf, or one whose pages are glued shut.
Think of it like
A shop that's open for business, with the lights on and the stock out, but the front door is locked from the street. Nobody gets in, and nobody knows why the tills are quiet.
See it happen
Same file. One word is the entire difference between visible and invisible.
robots.txt: one line wrong
Disallow: /
Every page on the site, blocked from every search engine. Nobody's typo shows up as an error. The site just stops appearing.
robots.txt: correct
Allow: /
Same file, same structure, one word different. Fully visible to every search engine.
02Why it matters
To the people searching
If your valuation page isn't indexed, a seller searching "house valuation [town]" simply never sees it, and you never find out what you lost. Rentals go within days, so a new listing Google takes a week to find has often let before anyone could search for it.
buyers
Listings and area pages that aren't indexed can't be found directly, only through the portals.
tenants
A rental that's slow to be indexed is often let before anyone finds it on your site.
sellers
A valuation page missing from Google means valuations going to whoever is there instead.
landlords
Rental valuation and management pages must be indexed to catch landlords while they research.
To Google and Bing
A page that isn't indexed can't rank at allGoogle has to discover and access a page before it can judge it. The usual reasons it can't: a robots.txt rule blocking it, a noindex tag left on by mistake, server errors, password protection, no links pointing to it, or content that only appears after JavaScript runs and doesn't load properly for Google. Crawling and indexing are different steps: Google can visit a page and still decide not to keep it. Bing works the same way.
An agent example
An agency relaunches its website. The new valuation page looks perfect, but the developer left a noindex tag on it from the test site. For four months Search Console lists it under "Excluded by 'noindex' tag", and valuation enquiries quietly dry up. Removing one line fixes it.
03Your turn
Check yours, then fix it
- 1Open Search Console and choose your site.
- 2In the left menu, open Indexing, then Pages.
- 3Note how many pages are Indexed and Not indexed.
- 4Scroll to "Why pages aren't indexed" and click each reason to see example pages.
- 5Red flags: "Excluded by 'noindex' tag" or "Blocked by robots.txt" on pages you want found, "Server error (5xx)", "Not found (404)" on important pages, or lots of "Crawled, currently not indexed".
- 1Paste a page's address into the "Inspect any URL" bar at the top of Search Console.
- 2"URL is on Google" is good. "URL is not on Google" tells you why.
- 3Click Test live URL, then View tested page, to see the screenshot and code Google gets. Is your content there?
- 4Do this for your homepage, valuation page, rentals page, sales page, one listing, one area page and one guide.
- 5For an important new page, click Request indexing. It's a request, not a guarantee.
- 1Type yourdomain.co.uk/robots.txt into your browser.
- 2"User-agent: *" followed by "Disallow: /" blocks your whole site. Any other Disallow line should be something you meant to hide, like an admin area.
- 3In Search Console, Settings, robots.txt shows the file Google last read and any errors.
- 1On a key page, right-click and choose View page source.
- 2Press Ctrl+F (Cmd+F on a Mac) and search for noindex.
- 3If it's there, check whether it's meant to be.
- 4For the whole site at once, crawl it with Screaming Frog and open the Directives tab.
Here is my robots.txt and the <head> section of my page's source code. Is anything here stopping Google from crawling or indexing my important pages (homepage, valuation page, listings)? Explain in plain English and tell me exactly what to change.
- 1WordPress: Settings, Reading, and untick "Discourage search engines from indexing this site".
- 2Then check the page itself: in Yoast or Rank Math, the setting that allows search engines to show this page.
- 3Other platforms: ask your web provider to remove the noindex tag, and send them the Search Console screenshot.
- 1Remove any Disallow line blocking pages you want found.
- 2Test the result with URL Inspection before you relax.
- 3If you're at all unsure, ask a developer: this file carries real risk.
- 1Add genuinely useful, original content (lessons 4.2 and 4.4).
- 2Link to it from relevant pages on your site (lesson 2.2).
- 3Merge thin pages that say the same thing (lesson 4.5).
- 1Open a support ticket with screenshots from Search Console.
- 2Ask them to confirm that listings and key pages are indexable, and that listing content is in the page's HTML rather than loaded later by scripts.
Then: After any fix, use URL Inspection and Request indexing on the affected pages, then check the Pages report again in a week or two.
04Keep it up
Glance at the Pages report monthly, and always after a relaunch, a new provider or a developer touching the site. A single stray line here can undo everything else in this course.
Go deeperrobots.txt in more detail
robots.txt tells search engines which parts of your site to stay out of. A clean file is usually short, and the less it blocks, the better.
It stops crawling, not indexing. A blocked page can still appear in Google (just without a description) if other sites link to it. To keep a page out of Google, use a noindex tag instead, and don't block that page in robots.txt, or Google never sees the noindex.
It's a public file and a polite request, not a lock. Never use it to hide anything private.
Whether to allow AI tools like ChatGPT to read your site is covered in our free GEO course.
Go deeperJavaScript and rendering
Many property sites build listings and search results with JavaScript after the page loads. Google can run JavaScript, but it's an extra step that can be delayed or fail, and Bing is less forgiving.
Test it: on a listing, View page source and search for the price or a line of the description. If it isn't in the source but shows on screen, the content depends on scripts. URL Inspection's View tested page shows whether Google gets it.
Fix it: ask your provider to put key content and links in the HTML from the start (server-side rendering), and to use real links (<a href>) for any filter or page you want found.
That's lesson 2.1
Worked through it? Mark it done and we’ll keep track on this device.