Why AI Search Answers Can Change Even When Your Website Has Not

Why Do AI Answers Change When Nothing on Your Site Did?

AI answers change because the system that produces them has several moving parts: randomness in text generation, changing retrieval results, model and product updates, personalization and shifts in the sources it reads. Your website is only one input. A change in output does not, by itself, tell you anything about your own pages.

A useful comparison is a weather forecast. Nobody concludes their house changed when tomorrow's forecast differs from today's. The forecast depends on many inputs, most outside your control. AI answers are the same, with the added twist that even identical inputs can produce different wording.

Ten Reasons the Answer Moves

1. Sampling randomness. Language models pick each next word from a probability distribution. Unless a product forces identical output, the same prompt can produce different phrasing and different examples on every run.

2. Retrieval differences. When an assistant searches the web, it decides what to search for, gets a ranked list and reads passages. Small differences in the queries it writes or the pages it fetches change what ends up in the answer.

3. Query fan-out. Some systems break a question into several sub-searches. Which sub-searches they choose can vary, and each pulls different sources.

4. Index and ranking updates. Search indexes refresh constantly. A page you compete with may have been updated, or a third-party article that mentioned you may have moved down.

5. Model and product updates. Providers update models, system instructions and search features, often without announcements. Behavior can shift overnight.

6. Model routing. Products may route a question to different models depending on load or complexity, so two runs may not even use the same model.

7. Personalization and memory. Location, language, past conversations and saved preferences can shape answers for one person and not another.

8. Tool settings. Web search on versus off, deep research versus a quick reply, and different modes all produce different answers.

9. Conversation context. Earlier messages in a chat change later replies. A fresh session and a long session are different tests.

10. Changes elsewhere. The sources cited about you can change without your involvement: a review is edited, a directory listing is removed, a competitor publishes a better guide.

What Studies Have Found

Research supports what practitioners see. A 2025 SparkToro study collected thousands of prompt runs from hundreds of volunteers across several AI tools. It found that asking for brand recommendations repeatedly almost never produced an identical list: fewer than 1 in 100 runs matched, and order varied even more. The study is not peer reviewed and used a limited prompt set, but its central finding fits everyday experience. It also found that the share of runs in which a brand appears was steadier than its position in any single list.

The variation exists across Google's own products too. One 2026 analysis reported that AI Overviews and AI Mode shared only about 13.7 percent of the same citations, so a good result on one surface says little about the other. Our article on how AI search citations get selected explains why retrieval-based systems differ in what they cite.

How Sure Can You Be From a Small Sample?

The practical tool for cutting through variability is sample size. If your brand appears in about 30 percent of answers, how many runs do you need before a measured percentage means something? A standard approximation gives the range around your estimate.

Ten runs can report 30 percent when the true rate is anywhere from near zero to nearly 60 percent. That is why a "drop" from 40 to 30 percent measured on ten runs is meaningless. The margins narrow slowly, so plan to pool across many prompts, run more samples for important questions and only call something a change when the difference is clearly larger than the noise. These are rough figures that assume independent runs, which real prompts only approximate, so treat them as a guide, not a guarantee.

How to Tell Real Movement From Noise

1. Use the same prompts, settings and run counts. Change only one thing at a time.
2. Track rates, not single answers. Report percentages with the number of runs beside them.
3. Compare across competitors. If every brand in your category moved the same direction, suspect the platform.
4. Record the environment. Log the date, product, mode, logged-in state, region and model name if shown.
5. Save the raw text. When a number changes, the saved answers show what replaced you.
6. Look at trends over weeks. A pattern across several months means more than any single round.
7. Check your other data. Do referrals from that assistant, in your AI referral analytics, move in the same direction?

A Checklist for a Sudden Disappearance

Sometimes a change is real. If your brand disappears from answers it used to appear in, work through this list before blaming the algorithm.

1. Confirm it with more runs. Repeat the prompt 20 or more times in a fresh session. If you still appear in a fair share, the "drop" was noise.
2. Check the settings. Was web search on last time and off now? Different mode? Different account state?
3. Look at the cited sources. Did a page that used to mention you change or disappear? Open it.
4. Check your own pages. Look for accidental blocks: a robots.txt change, a noindex tag, a broken redirect, a page that now returns an error to crawlers, or content that moved behind a script.
5. Compare with competitors. Did others rise or fall? A category-wide shift points to the platform.
6. Check platform announcements. Providers sometimes publish notes about search or model changes.
7. Review server logs. Are the relevant assistant crawlers still fetching your pages successfully, or are they receiving errors?

If the checklist finds a technical fault on your side, fix it. If it finds nothing, resist the urge to rewrite pages in a panic. Changes made in response to noise add noise of their own and make later results harder to interpret.

How a False Alarm Looks in Practice

Imagine a team that checks one prompt each Monday. In week one their product is named. In week two it is not. In week three it is back at the bottom of the list. Someone proposes rewriting the product page. Instead, an analyst runs the same prompt thirty times over two days and finds the product named in about a third of the runs. The weekly check was sampling a coin that lands heads a third of the time. The story changes from "we lost visibility and won it back" to "our visibility is moderate and noisy."

That is a less exciting story, and a truer one. It also suggests better work: instead of chasing the coin, improve the pages and sources that would push the rate up, and check the rate monthly with enough runs to be meaningful.

What You Can Do About It

You cannot make an inherently variable system consistent. You can improve the inputs.

• Keep your facts clear and current. Plain statements of what you do, dated and easy to quote, are what retrieval systems pick up.
• Publish pages that answer specific questions directly. Systems reward pages they can quote without heavy interpretation.
• Correct stale third-party sources. Answers often lean on directories and reviews, not just your site. See how to find incorrect descriptions of your company.
• Make sure crawlers can read your pages. If key content is hidden behind scripts, some AI crawlers may never see it. We cover that in how AI search handles dynamic JavaScript websites.
• Build authority over time. Being mentioned reliably in reputable places matters more than any tweak.

How to Explain Variability to Colleagues

Managers like clean numbers, and AI visibility does not provide them. A few habits make the conversation easier. Show ranges, not points. Put the sample size on every chart. Explain the forecast comparison in one sentence. Agree in advance on a rule for calling a change real, for example "a difference of more than 10 points sustained for two consecutive months with at least 40 runs per group." Having the rule beforehand prevents overreaction to a single bad week and avoids claiming credit for a lucky one.

Also decide which decisions depend on the numbers. If the choice is whether to invest in improving product comparison pages, a rough estimate is enough. If you are about to report results to a client, you need a larger sample and a clear method statement.

Frequently Asked Questions