2026-09-18, updated 2026-11-11, GEO · Fundamentals · Analytics · Research

AI answer volatility: how much answers change and how often to check

AI answers vary because every answer is a fresh sample: the engine generates text probabilistically over search results that change from day to day. One run shows what an answer can say, not how often it says it. The measurement that holds up is a rate over repeated runs, reported with its uncertainty. This page shows how much answers moved in a 600-answer study, how many runs a rate needs, and how to pick a cadence and budget for ongoing monitoring.

Definitions

• Cell: one prompt, one market, one engine. It is the unit you repeat.
• Run: one collected answer for a cell.
• Mention rate: the share of runs in a cell whose answer names the brand, p = mentions / n.
• Standard error (SE): the typical distance between an observed rate and the true rate you would see with infinitely many runs.
• Margin of error: the half-width of a confidence interval; for a 95% interval under the normal approximation it is 1.96 × SE.
• Cadence: how often a cell is collected.
• Detection delay: the time between a real change in answers and the moment your data confirms it.

How much answers change: 30 questions, 10 runs each

On 2026-09-28 we asked ChatGPT and Google AI Mode the same 30 buying questions (“best CRM for a small business”, “best password manager”, “best running shoes for flat feet”) ten times each, from the US. For each question we tracked six well-known brands of its category and the sites each answer cited. 596 of the 600 answers came back.

Take “best password manager”. Over ten ChatGPT runs, 1Password and Bitwarden appeared every time, Dashlane eight times, Keeper twice, NordPass and LastPass never. Over ten AI Mode runs, NordPass appeared every time, Keeper nine times and Dashlane never.

The cited sources moved more than the brands. Two ChatGPT answers to the same question shared under a third of their cited domains on average, so a report built on one run’s sources would show a different outreach list every time.

For the rates observed, the average width of the 95% interval around a measured rate was:

Ten runs are the practical floor for a number you put in front of a client. For a decision between two brands that sit close together, run twenty or more and compare intervals, not single rates.

Method: 30 buying-intent questions, six tracked brands per question, country US. Ten runs per question and engine were sent as independent async tasks in one batch, so the study measures variation within a few hours; runs spread over days are likely to vary more. A brand counts as mentioned when one of its names appears as a whole word in the answer text, ignoring case, typographic quotes and dashes, and brands are tracked by names that are not everyday words (“Copilot Money”, not “Copilot”). Source overlap is the Jaccard index of the cited domains of two runs, averaged over every pair of runs of a question. Interval widths are Wilson 95% intervals at the observed rates. Every question’s mention rates and source overlap are in answer-variance-2026-09.json.

Why answers vary

Four mechanisms act at once. None of them has a published rate you can plan around, so you measure the outcome instead of predicting it.

Sampling during generation

Language models pick each next token from a probability distribution. Google’s Gemini API documentation describes temperature as controlling “the degree of randomness in token selection” and notes that only a temperature of 0 always selects the highest-probability response (Gemini prompting strategies, checked 2026-09-17). Consumer assistants do not publish the sampling settings they use, so treat answer wording, list order and the set of named brands as random variables.

Retrieval drift

When an assistant searches the web before answering, the answer is conditioned on whatever those searches return at that moment. New pages get indexed, rankings move, and the pool of candidate sources shifts. You see this layer through the cited sources: sources[] lists the pages an answer used, in order, and that list moves as retrieval moves. ChatGPT’s searchQueries[] comes back empty on the answers served today; the query fan-out post covers how to reason about its searches anyway.

Market and session context

The same prompt in a different country or language can produce a different answer. Every request names its market with country. ChatGPT, Gemini, AI Mode and Google Search answer from 213 countries; Copilot, Perplexity and Grok take any country; Google News takes the 71 countries with a Google News edition. ChatGPT and Gemini also answer from inside a requested US state; the other engines accept state and answer for the whole US. Keep those inputs identical across the runs of a cell. See state-level AI answers for market design.

Model updates

Providers change the models behind their products, and a change can shift answer style, length and source selection for many prompts at once. ChatGPT and Grok responses include a model field; store it with every row so you can check whether a step change lines up with a model change.

Quantifying variance: the binomial standard error

If each run of a cell either mentions the brand or does not, and runs are independent, the count of mentions follows a binomial distribution:

SE = sqrt( p × (1 − p) / n )
95% margin ≈ 1.96 × SE

Worked values, computed from the formula (not measured data):

Uncertainty is largest when the true rate is near 50%, which is where a brand is contested. The margin shrinks with the square root of n: halving it takes four times the runs.

Sizing n for your own margin

For a target margin E at 95% confidence:

n = p × (1 − p) × (1.96 / E)²

Use p = 0.5 when you have no prior estimate, since it gives the largest n:

• E = 0.15: 0.25 × (1.96 / 0.15)² = 42.7, so 43 runs
• E = 0.10: 0.25 × (1.96 / 0.10)² = 96.0, so 97 runs
• E = 0.05: 0.25 × (1.96 / 0.05)² = 384.2, so 385 runs

If a pilot shows p near 0.2, recompute: E = 0.10 gives 0.16 × 384.2 = 61.5, so 62 runs.

Detecting a change between two periods

A drop from last month to this month is only a finding if it exceeds the noise in both measurements:

SE_diff = sqrt( SE_1² + SE_2² )

With p = 0.5 and n = 100 in each period, SE_diff = 0.0707 and the 95% margin is 0.139. A change smaller than about 14 points is not distinguishable from sampling noise at that sample size.

Rates near 0% or 100%

The normal approximation breaks down when p is close to 0 or 1 or n is small. Use the Wilson interval:

center = (p + z²/(2n)) / (1 + z²/n)
half   = z × sqrt( p(1−p)/n + z²/(4n²) ) / (1 + z²/n)

For 0 mentions in 20 runs with z = 1.96, the interval is 0% to 16.1%. “Never mentioned” after 20 runs means “mentioned in fewer than about one in six answers”.

When runs are not independent

The formulas assume independent runs. Runs collected in the same minute share search results and index state, so the true uncertainty is larger than the formula suggests. Spread a cell’s runs across hours and days. If you must report from a burst, treat the interval as a lower bound on uncertainty.

Choosing a cadence

The right cadence is the slowest one that still tells you about a change before it costs you. One run does not confirm a change:

worst-case detection delay ≈ cadence interval × runs needed to confirm

Suppose a brand is normally mentioned in 70% of runs. With independent runs, a miss on a healthy day has probability 0.3, two in a row 0.09, three in a row 0.027. A “three misses in a row” rule fires on noise about 2.7% of the time per sequence; at daily cadence it confirms a real disappearance on day three, at weekly cadence in three weeks. For a brand mentioned in 90% of runs, two misses in a row have probability 0.01, so a two-run rule is enough. The more stable the baseline, the lower the cadence can go for the same delay.

Metrics from the same prompts carry different noise. Mention rates are binary per run and noisy near 50%. Citation share aggregates many sources per answer, but sources within one answer move together, so the answer is still the sampling unit. Decouple schedules: a daily alert set, a weekly reporting set and a monthly research set can share prompts without sharing cadence. For crises, see brand crisis monitoring.

A method

1. Pick the cells and group them by consequence: brand and high-intent prompts, category prompts, long tail. The prompt set design guide covers selection.
2. Write down the acceptable detection delay per group: a day, a week, a month.
3. Measure each cell’s baseline rate at a daily cadence for two to four weeks, sizing n from a target margin.
4. Set cadence = acceptable delay ÷ runs to confirm, rounded to a schedule you can run.
5. Submit as async batches with idempotencyKey values that include the run or slot, so a retried submission cannot double-collect.
6. Extract one row per run: cell, run time, model where present, mentioned (0/1), cited domains, brands named.
7. Compute p and its interval per cell and period, and classify cells by the interval (table below).
8. Revisit monthly. Promote cells that turned contested to a faster cadence; demote cells that stayed locked.

What it costs

monthly cost = prompts × markets × engines' price per run × runs per month

Prices per request at the Starter tier: ChatGPT $0.0015, Gemini $0.0015, Google AI Mode $0.00206, Google Search $0.0015, Google News $0.001. Google Search adds $0.00071 per page beyond the first and $0.00071 when the response holds an AI Overview; ChatGPT’s include flags and Google News pages beyond the first are free. Prices drop on larger tiers, reached by 30-day spend (Growth from $95.00, Max from $1,900); a new account starts with 100 free requests.

A pilot, 20 prompts in one market at 43 runs each (E = 0.15):

ChatGPT: 20 × 43 × $0.0015 = $1.29
Gemini:  20 × 43 × $0.0015 = $1.29
total                 = $2.58

Ongoing monitoring, 200 prompts, US, ChatGPT and Gemini, 30-day month:

Engines and markets multiply every prompt, so removing one that does not change decisions can save as much as a large cut in cadence. Ten news queries collected hourly on Google News cost 10 × 24 × 30 × $0.001 = $7.20 a month, and pages: 2 costs the same. Cost planning has more combinations, and /pricing every tier’s price.

Collecting the runs

Run the schedule from your own job runner (cron, a workflow tool, a queue worker) and submit each slot as one batch of up to 500 tasks:

import datetime, os, requests

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}", "Content-Type": "application/json"}
ENGINES = {"CHATGPT": "chatgpt", "GEMINI": "gemini"}

def daily_batch(prompts, markets, webhook_url):
    day = datetime.date.today().isoformat()
    tasks = [
        {
            "taskType": task_type,
            "idempotencyKey": f"cadence:{day}:{slug}:{country}:{pid}",
            "payload": {"prompt": text, "country": country},
            "webhook": {"url": webhook_url},
        }
        for pid, text in prompts
        for country in markets
        for task_type, slug in ENGINES.items()
    ]
    for start in range(0, len(tasks), 500):
        r = requests.post(f"{API}/v1/async/task/batch", json=tasks[start:start + 500], headers=HEADERS, timeout=30)
        r.raise_for_status()
        for item in r.json()["results"]:
            if not item["success"] and item["error"]["code"] != "RESOURCE_ALREADY_EXISTS":
                print("task", start + item["index"], item["error"]["code"], item["error"]["message"])

For repeated runs of one cell in a slot, add the run index to the key (...:{pid}:r01). Queue rules that shape the schedule:

• The queue holds about 100,000 tasks of a typical prompt, with no concurrency cap. A queued task waits until it can start; clear the queue if you no longer need a run.
• priority (1 to 10, higher first) orders only your own queue: give daily brand prompts a higher priority than the weekly long tail.
• RESOURCE_ALREADY_EXISTS on a batch item means that slot was already submitted, which is what you want on a retry.
• Failed tasks are not charged.

Results arrive at your webhook as the finished task JSON; the engine result of a COMPLETED task is at response.result. See webhook reliability design for the receiver and verify webhook signatures for the check.

Computing the rate and Wilson interval from stored rows:

import math
from collections import defaultdict

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (max(0.0, center - half), min(1.0, center + half))

def mentioned(result, brand):
    text = result.get("text") or ""
    return brand.lower() in text.lower()

# rows: (cell_key, result) pairs taken from COMPLETED tasks' response["result"]
def rates(rows, brand):
    counts = defaultdict(lambda: [0, 0])
    for cell, result in rows:
        c = counts[cell]
        c[0] += 1 if mentioned(result, brand) else 0
        c[1] += 1
    return {cell: (k / n, *wilson(k, n)) for cell, (k, n) in counts.items()}

Substring matching is a starting point; handle aliases and false positives before you publish numbers (measuring brand mentions in ChatGPT). The measuring visibility guide shows the SDK helpers: visibility.measure runs the samples and brandCheck returns each brand’s mention rate with its 95% interval.

Reporting and alerting

Show the interval with the point. “Mentioned in 58% of runs (95% interval 48 to 68%)” is a finding; “58%” invites a comparison with last week’s 61% as if the difference meant something.

• Report n next to every rate, and plot rates with interval bars per cell and period.
• Call a change only when the difference exceeds its own margin.
• Alert on sustained patterns: N consecutive misses sized from the baseline rate, or a weekly rate whose change exceeds its margin.
• Annotate periods where the model value changed.
• Log credits.creditsCharged per run so the bill and the schedule stay reconciled.

The interval tells you what kind of prompt you are looking at:

The thresholds are illustrative; set your own. See how engines choose citations, sampling vs. census and AI share of voice.

Pitfalls

• Calling a single run a drop. A missing mention at p = 0.7 happens three times in ten.
• Mixing markets or engines into one rate. Compute per cell, then aggregate with the counts visible.
• Bursty collection. Correlated runs understate uncertainty.
• Reading optional arrays without a default. They are omitted when empty.
• Rewording the prompt mid-series. A reworded prompt is a new cell.

Set up collection with the async task guide, or start with the ChatGPT engine page.

Keep reading