GEO courseModule 2 of 10: Your Prompts and BaselineLesson 2.3
Testing Fairly, and Running Your Baseline
How to test what AI says about you in a way you can trust: the same prompts, run several times, on several platforms, from several places, with the conditions written down. Your first full run is your baseline, the "before" every later change is measured against.
Why it matters
One AI answer is one roll of the dice. Act on it and you'll chase noise, or relax when you shouldn't.
The evidence
Evidence: Study
Vendor studies (SparkToro and Gumshoe, BrightLocal) show answers vary run to run and place to place. The test method is practitioner advice.
Do this first
Run your standard prompts on two platforms, three times each, in private windows, and log every answer before you change anything.
01What it is
One coin toss tells you nothing about whether a coin is fair. Toss it fifty times and write each result down, and you start to know.
See it happen
Same prompt, run again and again. Watch which number you can actually trust.
“Best letting agent in Hallowford?”
Green: named (and where in the list). Grey: not named.
The honest number
–
run the prompt
The misleading one
–
this run’s “position”
One run is one roll of the dice. Keep going.
02Why it matters
To the people searching
Your customers don't see your answer. They see theirs: on their phone, logged in, from their street, on the day they asked. A director testing on the office laptop, where they've asked about the agency fifty times, can see something no customer ever does.
buyers
Search from where they live or work, often on a phone, so location changes what they see.
tenants
Ask "near me" questions where the AI guesses their location.
sellers
Name their own road or postcode and expect agents who cover it.
landlords
Often compare two or three tools before choosing who to call.
To Google, Bing and AI
Evidence: StudyAnswers vary for real reasons. Location: BrightLocal found moving across one town changed which businesses were named, and Claude says it may use your approximate location from your internet connection. Repeats: the SparkToro study found the same list of brands less than once in 100 runs, but how often a brand appeared held steady. Search: if the assistant didn't search the web, it can only name you from training memory. None of the platforms tells you how many people saw an answer.
- Runs that returned the same list of brands (SparkToro, 2,961 runs)
- <1 in 100Runs that returned the same list of brands (SparkToro, 2,961 runs)
- Businesses shared between repeat runs of the same local prompt in ChatGPT (BrightLocal)
- ~1 in 4Businesses shared between repeat runs of the same local prompt in ChatGPT (BrightLocal)
An agent example
A director finds the agency named when he tests on his office laptop. A negotiator tests in a private window on her phone and doesn't see it. They each run the same prompt five times: 3 of 5 on the laptop, 1 of 5 in the private window. They log both, and treat the laptop result as unreliable.
03Your turn
Check yours, then fix it
- 1Take your standard set from lesson 2.2 (12 to 20 prompts).
- 2Pick two or three platforms to start: ChatGPT with search on, Google AI Mode, and Copilot or Perplexity.
- 3Run each prompt three to five times per platform, each in a fresh private window, spread over two to four weeks.
- 4Log every answer with the columns below.
- 5Write the headline as "named in X of N runs", never a position.
- 1Run five local prompts from the office, from a phone on mobile data, and from a colleague's home on the other side of your patch.
- 2Compare "estate agents in [town]" with "estate agents near me" on the same assistant.
- 3Log the location every time.
- 1Run one prompt five times in a private window, then five times logged in.
- 2If the tool lets you switch memory or custom instructions off, note the setting.
- 3A big difference means your own history is colouring what you see.
- 1"Who is the best estate agent in [town]?" "Which letting agent should I use in [town]?"
- 2"[Your agency] vs [your main rival]" and "What do customers say about [your agency]?"
- 3Note the sources each answer cites. They're your map for module 7.
- 1Date, platform, logged in or out, search on or off, location, run number.
- 2Prompt ID, audience, exact prompt.
- 3Named (yes/no), cited with a link (yes/no, and the address), other agencies named, sources shown.
- 4Accurate (yes, no, partly), errors about you, screenshot file name, notes.
- 1Light baseline: 12 prompts, 2 platforms, 3 runs each = 72 answers. Treat moves under about 10 percentage points as noise.
- 2Standard baseline: 20 prompts, 2 to 3 platforms, 3 to 5 runs each = roughly 120 to 300 answers.
- 3Zero mentions in 20 runs doesn't prove you're never named. Keep going before you conclude anything.
- 1Don't change the wording of your core prompts for at least six months.
- 2Add new prompts as a separate list.
- 3Never reword a prompt to get a better answer.
Here's my AI prompt test log (no client data). Count how often my agency was named and cited, per platform and per audience, and list every website cited, with how many runs it appeared in. Don't guess any value that isn't in the log.
Then: Repeat the same set every month, with three runs per prompt. Compare totals, not single answers, and look at the pattern over three months before you claim anything moved.
04Keep it up
Baseline once, over two to four weeks, before you make changes. Then a monthly run of the same set. Module 10 turns the log into a one-page monthly report.
Go deeperWhy "we're number 2 in ChatGPT" is meaningless
Order changes from run to run, so a position is a snapshot of one roll. "Named in 11 of 48 answers" survives a re-test; "second" doesn't.
Two months of 30 answers each can easily differ by 10 points through chance alone. Only compare months with similar numbers of answers, and only believe differences that repeat.
See also
That's lesson 2.3
Worked through it? Mark it done and we’ll keep track on this device.
← Back to 2.2 What People Actually Ask AI: Your Prompt Bank and Prompt Map