An AI visibility score lands in your inbox: 82%, green arrow, no prompt list. Ask the same buyer question twice and the brand list may change, so what exactly did that arrow measure?

I wrote The AI Visibility Glossary for Google Ads People: 18 Terms and Their Traps to put names to the terms. We are taking a second look because knowing the words did not make the scores checkable. More businesses are being handed numbers they cannot verify. This time, the job is to run a small test in-house before trusting any number, yours or a vendor’s: ask the same buyer prompts several times across each engine and log the answers like a search terms report.

The question is narrow on purpose: for a fixed set of buyer prompts, how often is your brand mentioned or cited, and how much does that rate move between identical rounds? First measure the movement you get without changing anything. Then decide whether a later change deserves your attention.

Setup: turn paid search terms into buyer prompts

Pull your last 90 days of search terms, sort by spend and start with queries that already cost you money. That report is a prompt list in raw form, not a finished test. I would aim for 20 to 30 prompts across four buckets:

  • Category: best CRM for small law firm
  • Comparison: HubSpot vs Pipedrive for service business
  • Problem-led: how to reduce no-shows for dental clinic
  • Branded: is [your brand] good for [use case]

Practitioners suggest starting with 5 to 15 prompts split across branded, comparison and non-branded questions. I would go higher and repeat the prompts, because a small set asked once gives you little chance of separating a real move from ordinary variation. Lean toward non-branded prompts. A branded question already contains your name; it tells you less about buyers who have not heard of you.

Rewrite each search term the way a buyer would ask an assistant. Keywords are fragments. Prompts have context: budget, location, constraint. cheap project software becomes what project software under $50 a month works for a 5-person remodeling crew. Do that editing before the first run. Once testing starts, freeze the wording. Changing it between rounds breaks the comparison.

Give each prompt one row, an owner and a version. If a better question occurs to you next week, save it for the next version of the test. Do not quietly swap it into this one and call the resulting chart a trend. The prompt set is the control, not a draft you improve between measurements.

Controls: make each engine answer under repeatable conditions

Test what buyers actually see: ChatGPT on the web, Gemini on the web and Google AI Overviews in a normal search. Keep each engine’s results separate. Do not mix API answers with consumer-interface answers in one series; access method, wording, login state, memory and location can change the answer.

Write a short control sheet before you start. Use the same logged-out or fresh profile, location and device type each round. Clear history and log the date. If you cannot hold a condition steady, record what changed rather than pretending it did not. An answer produced through a different interface is a new series, not the next dot on the old chart.

For AI Overviews, record whether an Overview appeared and whether you had to expand it. No Overview is a result worth keeping, not a row to delete. Search Console includes those impressions in its totals without a filter for Overview presence, so it cannot fill that gap in your log. No appearance and no citation are different outcomes. Keep both visible.

Repeats: ask the same question until a single answer stops looking definitive

Same prompt, different answer is normal. A lab sample cited in research on AI visibility score accuracy ran one prompt 1,000 times at temperature zero and got 80 different completions. In a separate field test, 600 volunteers ran buyer prompts across ChatGPT, Claude and Google AI; fewer than 1 in 100 runs produced the same brand list. Those are not results from your account. They are a reason not to treat one answer as your baseline.

Start with five repeats per prompt per engine. Treat anything below about 20 runs per category as a screenshot, not a dependable trend. The practitioners behind the field test suggest 20 to 30 runs per category for directional trends and 50-plus for competitive claims. If the workload gets uncomfortable, cut the prompt list before you cut the repeats. Fewer frozen prompts with repeat runs beat a long list asked once.

Run the full set twice in week one, without changing your site between rounds. You are not trying to produce a launch result. You are measuring how far the answers move while your own setup stays put. Keep both rounds, including the inconvenient ones.

Log: record mentions, citations and competitor movement separately

The first sheet can be plain. Put each run on its own row and record:

  • Prompt and prompt version
  • Date, engine and access method
  • Your brand mentioned: yes or no
  • Your site cited: yes or no, with the URL
  • Competitors named
  • For AI Overviews: whether one appeared and whether it needed expanding

Save the answer as well, so you can check how you scored it. A mention means your name appeared in the answer text. A citation means a clickable link pointed to your site. Do not award yourself a citation because the answer named you, or a mention because your page appeared somewhere you expected to see it. Define those rules once and use them on every row.

One manual approach uses 40 to 50 buyer prompts across three engines for about 150 checks a month. In its example, 34 mentions from 150 checks work out to roughly 23%. That arithmetic is useful, but I would report the rate for each engine, with its run count, rather than blend everything into one headline. ChatGPT, Gemini and AI Overviews do not give you the same answer surface. AI Overviews, for example, cite three to five sources per answer on average. A combined percentage can hide the place where you are absent.

Track two or three competitors beside your brand on each run. They are controls, not a separate scoreboard to admire. When your mentions rise, check whether theirs rose in the same bucket and engine. If everyone moved, be cautious about crediting your site. If you moved and they did not, you have a more useful question to investigate.

Now compare the two unchanged week-one rounds. Calculate the mention and citation rates for each engine and bucket, and note how far they shifted. That shift is your working noise floor, not a universal statistical guarantee. A category rate that moves eight points between unchanged rounds does not make a six-point rise next month look persuasive. Put the run count beside every claimed win. A green arrow without it has not earned your attention.

Reading the result: a percentage is not a verdict

Three mentions out of ten runs is 30% of those ten runs. It is not a dependable estimate of what the next ten will do. At around 20% visibility, an estimate from 20 prompts has a band of about plus or minus 17.5 points. At 100 prompts, that band tightens to about plus or minus 7.8 points. Small samples leave plenty of room for a chart to climb without your visibility improving.

The same research estimates that a five-point gain can be buried in roughly 16 points of noise when two periods of 50 prompts are compared. Detecting a real five-point move takes something like 1,100 prompts per period in that analysis; a ten-point move takes around 290. I would not ask a local client to run 1,100 prompts to settle a small wiggle. I would stop calling the wiggle a win.

Read your sheet in rounds, not isolated points. Repeat the frozen set weekly, or monthly at minimum, and look for a pattern after three or four cycles, a cadence practitioners recommend for spotting movement. A change becomes worth investigating when it exceeds your unchanged-round movement, appears on the same engine in consecutive rounds and is not mirrored by the competitors you track. That is a directional signal, not proof that one page edit caused it.

The useful patterns are specific. You may be present on branded prompts but missing on category prompts. You may be named in answers without a link to your site. You may appear in ChatGPT but not in Overviews. Keep those findings separate: they point to different work, while one blended score points nowhere. Report the range with the rate, and say what happened to the controls.

Automation: save time without changing the test

The manual version needs a sheet and a calendar reminder. Keep the prompt text, answer, timestamp and scoring decision together. If you automate, send the frozen list on a schedule, store full answers, flag your brand name and export the results. That is the shape of a DIY tracker MVP: repeat the same prompts, log competitor appearances beside yours and add other classifications later if you need them. A manual benchmark is 2 to 3 hours a month for roughly 150 checks.

ChatGPT and Gemini APIs can make scheduled runs easier, but API results must remain a separate series from the web-interface test. Otherwise, the time you saved comes at the cost of a comparison you can no longer interpret. Pick the access method you intend to measure, label it and keep it steady.

AI Overviews are the awkward part. There is no official API in this setup that tells you whether your site was cited. For a manual spot-check, use an incognito session, clear cookies each time and log whether the Overview appeared, whether you expanded it, its citations, competitor appearances and a screenshot. Budget 2 to 3 minutes per keyword. One costing exercise puts 100 keywords across 3 locations at roughly 15 hours a week by hand. That is a poor use of the week. Its breakdown also describes pulling Overview HTML through a SERP data source, citing a free tier around 250 searches a month, a couple of hours of setup and five minutes of weekly review.

Whichever route you choose, retain the same prompt wording and log how the answer was collected. Automate the asking, not the definition of a comparable run.

Vendor check: ask for the controls before the score

Once you have your own noise floor, you can interview a tracking vendor like a buyer instead of an audience. Ask for the run count beside every reported change. Ask to see results by engine, the exact prompt list and its version dates, and whether each answer came from a consumer interface or an API. Ask how mentions differ from citations in their report. Those answers tell you more about the chart than the color of its arrow.

If a demo cannot show those controls, keep your spreadsheet. You have a stronger case for paying when the prompt list passes 50, weekly repeats eat half a day or you need Overviews coverage across locations you cannot reasonably spot-check. I made the fuller cost argument in Is AI Prompt Tracking Worth Paying For? The Case for It, the Case Against It, and My Verdict.

My bias is plain: tracking does not move CPA by itself. In groas work, the point of a prompt log is to inform the pages and fixes that get tested in the next round. A tool that only reports is a halfway step. Pay for a measurement process you can inspect, then use its findings to decide what to change.

Monthly report: three outputs, then a decision

Do not end this test with a dashboard full of percentages. Give the reader of your monthly report three things:

  1. Mention rate by engine, with runs and range. Write mentions over runs for ChatGPT, Gemini and AI Overviews separately. Show how each compares with the unchanged-round movement.
  2. Citation rate, with the linked URLs. Name the pages that earned a click path. Keep a mention without a citation in its own column.
  3. A one-sentence noise call. Say how far the frozen set moved between unchanged rounds and whether your tracked competitors moved with you.

Then make the call the sheet was built for. If you are missing on category prompts while competitors hold steady, put the next effort into the page that answers those prompts and test again. If every brand swings together between rounds, do not buy a story about your new green arrow. Change nothing on its account and keep the fee in your pocket.

Either result is useful. One tells you where to work; the other tells you when to wait. Both beat paying for a number you cannot check.