Sampling vs. census in AI-answer monitoring
An AI answer is a sample, not a fact. The same prompt asked twice can return different wording, different citations and different recommendations — so any monitoring program has to decide how many runs per prompt are enough to trust the measurement.
The mistake of n=1
A single run tells you what one user might have seen at one moment. For reporting “was our brand recommended”, n=1 produces false alarms in both directions: a lucky run hides a problem, an unlucky run invents one. Yet n=1 per prompt per period is what most dashboards quietly assume.
What variance actually looks like
From repeated-run monitoring, the stable patterns are:
- Answer text drifts constantly — wording changes nearly every run even when conclusions don’t.
- Citations are moderately stable — the top sources for a prompt tend to repeat, with a rotating tail.
- Recommendations (which products get named) sit in between — the leading answer is fairly stable, ranks 2–5 churn.
The practical consequence: you need fewer runs to measure “are we cited at all” than to measure “are we ranked #1 vs #2 in the recommendation”.
A sampling strategy that holds up
- Collect a variance baseline first. Run each prompt 5–10 times over a week on your primary engine. Measure how often each citation appears (appearance rate) and how often the brand is mentioned.
- Set the cadence by volatility. If a citation appears in 9 of 10 runs, weekly collection is enough. If it appears in 4 of 10, you need more frequent runs to detect a real drop vs. noise.
- Track rates, not booleans. “Cited in 80% of runs this month, down from 95%” is a signal. “Cited: yes/no” is a coin flip wearing a costume.
- Rotate seeds implicitly. Different times of day and different days give natural sampling; a daily run already captures more variance than 5 runs in the same hour.
When census makes sense
For compliance, legal, or crisis workflows, sampling is wrong — you want every run, every engine, every region. That is what async batches with webhooks are for: the full census, delivered when it completes, without paying sync-latency costs.
The reporting rule
Whatever n you choose, report it. “ChatGPT recommends us” means nothing without “in 17 of 20 runs across the past month”. The executive report template includes a variance column for exactly this reason — a metric without its sample size invites the wrong conclusion.