AI Visibility Sample Size: How Many Runs a Number Actually Needs

TL;DR:

• A single capture of an AI answer surfaced only 49.6% of the cited domains that five captures of the same prompt surfaced, measured across a 100-capture matrix of five answer engines, four prompts, and five repeats with every input held constant.
• Cited-domain sets overlapped across repeats at a mean pairwise Jaccard of 0.384, but individual engine-and-prompt cells ranged from 0.114 to 0.956 — there is no single stability figure that covers "AI visibility."
• Brand mentions inside the answer text were the least trustworthy layer: of 137 brand-by-cell observations, 44.5% appeared in all five repeats and 24.8% appeared in exactly one.
• Set-membership questions converge quickly — three repeats recovered 83.1% of the cited domains that five repeats found — so a citation audit needs a handful of runs, not a quarter of daily captures.
• Rate questions converge slowly: at an observed 40% appearance rate, five runs give a 95% interval 65 points wide, and reaching ±10 points takes roughly 100 runs per engine per prompt.
• The entire matrix ran inside 4 minutes 9 seconds, so these figures measure model-level nondeterminism alone; a program sampling across days should expect at least this much movement, never less.
• Run the same matrix against your own prompt set on the Scrapeless free plan and get your own number instead of borrowing this one.

Every AI-visibility dashboard reports a number. Share of citation, share of voice, recommendation rank, sentiment — each arrives as a clean figure with a trend arrow next to it. None of them arrive with an error bar.

That omission matters more here than in classic search, because the surface being measured regenerates its answer on every request. Researchers who pinned five models to nominally deterministic settings across eight tasks still recorded accuracy swings of up to 15% between runs, with the gap between the best and worst possible run reaching 70%. A model that will not repeat itself when it is told to will not hand you a repeatable visibility number from one capture either.

The rest of this post fires the same prompts at the same engines repeatedly, holds every input constant, and measures how far the answers actually move. That measured movement then converts into the only output a monitoring program can act on — a number of runs.

Pipeline at a Glance

The pipeline has four stages, and the last one is the deliverable:

capture the repeat matrix → reduce each answer to comparable layers → score stability per cell → convert stability into a required sample size

Stage 1 fires an identical request many times per engine and stores every raw response. Stage 2 strips each response down to three things that can be compared across repeats: the set of domains it cited, the text of the answer, and which watchlist names the text mentions and in what order. Stage 3 measures how much each of those three layers moves between repeats. Stage 4 takes the measured movement and answers the question a reporting team actually has, which is how many captures a claim needs behind it.

This sits one layer above the capture pipelines it depends on. If you have not built the capture side yet, the share-of-citation pipeline across six AI answer engines is the foundation; this post measures whether the series that pipeline produces is readable at the sample size you are running it at.

What This Measurement Answers

Four decisions change depending on the answer, and all four are being made today without it.

Whether a week-over-week move is real. A share-of-citation figure that drops from 40% to 25% looks like a regression. At five runs per cell, both figures sit inside the same confidence interval, so the drop is indistinguishable from the engine regenerating its answer.

How much capture budget a program needs. Sample size sets cost. A team that needs a rate estimate and a team that needs a citation inventory have requirements that differ by more than an order of magnitude, and buying the larger one for both wastes most of the budget.

Which metric to build a target around. Some layers of an AI answer are stable enough to set a goal against. Others are not, and building an OKR on the unstable one produces a quarter of noise interpreted as progress.

Which engine to weight. Per-engine stability varies by roughly a factor of two in this matrix. An engine whose citations barely move gives you a usable reading from a few captures; an engine that reshuffles needs several times the sampling for the same confidence.

The Sample Design, Stated Up Front

The matrix is five engines, four prompts, five repeats: 100 captures, with 20 engine-and-prompt cells holding five repeats each.

Two design choices carry the result. The first is that every input is pinned: same prompt string, same country, same reasoning mode, same web-search flag. Varying any of those would mix input differences into the variance and make the measurement meaningless. The fingerprint count in Stage 1 exists to prove the pinning held.

The second is the compressed window. All 100 captures landed inside four minutes. That is deliberate, and it is also the sharpest limitation of the whole exercise — see the limits section before quoting any figure here.

Prerequisites

• Python 3.10 or newer, standard library only.
• A Scrapeless API key exported as SCRAPELESS_API_KEY. The LLM Chat Scraper actors under the Universal Scraping API put all five engines behind one endpoint and one response envelope, which is what makes a cross-engine matrix practical.
• A watchlist.txt file with one brand name per line — your own brand and the names you expect to compete with in the category. Keep it out of version control if the list is sensitive.
• Roughly 100 actor calls of budget for a first pass. Usage-based pricing is on the Scrapeless pricing page.

export SCRAPELESS_API_KEY="your_scrapeless_api_key"
mkdir -p ai-visibility-variance && cd ai-visibility-variance
printf 'YourBrand\nRivalOne\nRivalTwo\n' > watchlist.txt

Stage 1 — Capture the Repeat Matrix

Stage 1 fires every cell's five repeats and writes each raw response to disk untouched. Storing the full envelope rather than a parsed summary matters, because the layers you want to compare are not obvious until you have looked at the spread.

Each engine takes the same prompt and country; only the actor-required extras differ, and each extra is pinned so it cannot drift between repeats. The cells run concurrently because a serial pass over 100 captures would stretch the window and let real-world drift leak into a measurement that is supposed to exclude it.

# capture.py — capture a fixed engine x prompt x repeat matrix
import hashlib
import json
import os
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor

API = "https://api.scrapeless.com/api/v2/scraper/execute"
KEY = os.environ["SCRAPELESS_API_KEY"]
OUT = "runs"
REPEATS = 5

PROMPTS = {
    "P1": "What are the best web scraping APIs for developers in 2026?",
    "P2": "Which services provide cloud browsers for automated data collection?",
    "P3": "What tools do developers use to collect Google search results programmatically?",
    "P4": "Recommend a proxy provider for large-scale public web data collection.",
}

# Same prompt and country everywhere. Only actor-required extras differ, and
# every one of them is pinned: an input that moves between repeats would be
# measured as engine variance.
ENGINES = {
    "perplexity": ("scraper.perplexity", {}),
    "grok":       ("scraper.grok",       {"mode": "MODEL_MODE_FAST"}),
    "gemini":     ("scraper.gemini",     {}),
    "aimode":     ("scraper.aimode",     {}),
    "chatgpt":    ("scraper.chatgpt",    {"web_search": True, "shopping": False}),
}


def execute(actor, payload, timeout=300):
    request = urllib.request.Request(
        API,
        data=json.dumps({"actor": actor, "input": payload}).encode(),
        headers={"Content-Type": "application/json", "x-api-token": KEY},
        method="POST",
    )
    started = time.time()
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return json.loads(response.read().decode()), round(time.time() - started, 1)


def capture(job):
    engine, prompt_id, repeat = job
    actor, extra = ENGINES[engine]
    payload = {"prompt": PROMPTS[prompt_id], "country": "US", **extra}
    # Hash the actor with the payload. Three of these engines take an identical
    # payload, so a payload-only hash would collapse them into one fingerprint
    # and stop proving that each cell repeated the same request.
    fingerprint = hashlib.sha256(
        json.dumps({"actor": actor, **payload}, sort_keys=True).encode()).hexdigest()[:16]
    try:
        body, elapsed = execute(actor, payload)
    except Exception as exc:
        # A cell that returns nothing is logged as missing, never dropped.
        # A silent hole would bias every statistic computed downstream.
        return {"ok": False, "note": f"{engine} {prompt_id} r{repeat}: {type(exc).__name__}"}
    with open(f"{OUT}/{engine}__{prompt_id}__r{repeat}.json", "w") as handle:
        json.dump({
            "engine": engine, "prompt_id": prompt_id, "repeat": repeat,
            "elapsed_s": elapsed, "input_fingerprint": fingerprint,
            "captured_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "response": body,
        }, handle)
    return {"ok": True, "fingerprint": fingerprint}


os.makedirs(OUT, exist_ok=True)
jobs = [(e, p, r) for e in ENGINES for p in PROMPTS for r in range(1, REPEATS + 1)]
print(f"{len(jobs)} captures = {len(ENGINES)} engines"
      f" x {len(PROMPTS)} prompts x {REPEATS} repeats")

with ThreadPoolExecutor(max_workers=12) as pool:
    results = list(pool.map(capture, jobs))

stored = [r for r in results if r["ok"]]
print(f"stored {len(stored)} captures in {OUT}/")
print(f"missing cells: {[r['note'] for r in results if not r['ok']] or 'none'}")
print(f"{len({r['fingerprint'] for r in stored})} distinct input fingerprints "
      f"across {len(stored)} captures")

The fingerprint line is the honesty check. Twenty distinct fingerprints across 100 captures means each cell's five repeats were byte-identical requests, so anything that differs in the responses came from the engine.

Stage 2 — Reduce Each Capture to Three Comparable Layers

Stage 2 flattens five different response envelopes into one row shape with three measurement layers, because "AI visibility" is not one quantity and treating it as one is where the error starts.

The three layers answer different questions and, as the results show, need different sample sizes:

Each engine reports its sources under its own key — Perplexity uses web_results, Grok keys footnotes by citation ID, Gemini and AI Mode both use citations, and ChatGPT's search panel arrives as search_result. Normalizing to the registrable host drops the difference between a deep article URL and a homepage link, which is what a share-of-citation metric actually counts.

# layers.py — reduce every capture to three comparable measurement layers
import json
import os
import re
from urllib.parse import urlparse

WATCHLIST = [line.strip() for line in open("watchlist.txt") if line.strip()]

CITATIONS = {
    "perplexity": lambda r: [c.get("url") for c in r.get("web_results") or []],
    "grok":       lambda r: [c.get("url") for c in (r.get("footnotes") or {}).values()],
    "gemini":     lambda r: [c.get("url") for c in r.get("citations") or []],
    "aimode":     lambda r: [c.get("url") for c in r.get("citations") or []],
    "chatgpt":    lambda r: [c.get("url") for c in r.get("search_result") or []],
}
ANSWER_KEY = {
    "perplexity": "result_text", "grok": "full_response", "gemini": "result_text",
    "aimode": "result_text", "chatgpt": "result_text",
}


def registrable(url):
    netloc = urlparse(url).netloc.lower()
    return netloc[4:] if netloc.startswith("www.") else netloc


def mentions(text):
    """Watchlist names present in the answer, keyed by first-mention offset."""
    lowered = text.lower()
    found = {}
    for name in WATCHLIST:
        hit = re.search(rf"(?<![a-z0-9]){re.escape(name.lower())}(?![a-z0-9])", lowered)
        if hit:
            found[name] = hit.start()
    return found


rows = []
for filename in sorted(os.listdir("runs")):
    record = json.load(open(f"runs/{filename}"))
    result = record["response"].get("task_result") or {}
    engine = record["engine"]
    answer = result.get(ANSWER_KEY[engine]) or ""
    found = mentions(answer)
    rows.append({
        "engine": engine,
        "prompt_id": record["prompt_id"],
        "repeat": record["repeat"],
        "cited_hosts": sorted({
            host for url in CITATIONS[engine](result)
            if url and (host := registrable(url))
        }),
        "answer_chars": len(answer),
        "answer_tokens": sorted(set(re.findall(r"[a-z0-9]+", answer.lower()))),
        # Ordered by first mention, so list position survives into stage 3.
        "brands_present": sorted(found, key=found.get),
    })

with open("layers.jsonl", "w") as handle:
    for row in rows:
        handle.write(json.dumps(row) + "\n")

cited = sum(len(r["cited_hosts"]) for r in rows)
print(f"reduced {len(rows)} captures -> layers.jsonl")
print(f"L1 citations: {cited} cited hosts, {cited / len(rows):.1f} per capture")
print(f"L2 prose:     {sum(r['answer_chars'] for r in rows) / len(rows):.0f} chars per answer")
print(f"L3 brands:    {sum(len(r['brands_present']) for r in rows)} watchlist hits total")

Stage 3 — Score Stability Per Engine and Prompt

Stage 3 measures movement inside each cell, and it reports per engine rather than pooling, because the pooled figure hides the thing you most need to know.

Three statistics do the work. For the citation and prose layers, the mean pairwise Jaccard index across the five repeats — the size of the intersection over the size of the union, averaged over all ten pairs in a cell. For answer length, the coefficient of variation, which expresses the standard deviation of a measured spread as a percentage of the mean so answers of different lengths stay comparable. For brands, the share of names that appeared in all five repeats rather than some of them.

# stability.py — how far each layer moves inside a cell
import json
import statistics as stats
from collections import defaultdict
from itertools import combinations

cells = defaultdict(dict)
for line in open("layers.jsonl"):
    row = json.loads(line)
    cells[(row["engine"], row["prompt_id"])][row["repeat"]] = row


def jaccard(left, right):
    union = left | right
    return len(left & right) / len(union) if union else None


def mean_pairwise(sets):
    scores = [j for a, b in combinations(sets, 2) if (j := jaccard(a, b)) is not None]
    return stats.mean(scores) if scores else None


per_engine = defaultdict(lambda: defaultdict(list))
brands = defaultdict(lambda: {"stable": 0, "flipping": 0})

for (engine, _prompt), reps in sorted(cells.items()):
    keys = sorted(reps)
    hosts = [set(reps[k]["cited_hosts"]) for k in keys]
    tokens = [set(reps[k]["answer_tokens"]) for k in keys]
    lengths = [reps[k]["answer_chars"] for k in keys]

    per_engine[engine]["cite"].append(mean_pairwise(hosts))
    per_engine[engine]["prose"].append(mean_pairwise(tokens))
    per_engine[engine]["cv"].append(stats.pstdev(lengths) / stats.mean(lengths) * 100)
    per_engine[engine]["union"].append(len(set().union(*hosts)))
    per_engine[engine]["core"].append(len(set.intersection(*hosts)))

    for brand in set().union(*[set(reps[k]["brands_present"]) for k in keys]):
        seen = sum(1 for k in keys if brand in reps[k]["brands_present"])
        brands[engine]["stable" if seen == len(keys) else "flipping"] += 1

header = f"{'engine':<12}{'citeJaccard':>12}{'proseJaccard':>13}{'lengthCV%':>11}"
print(header + f"{'hostsEveryRun':>15}{'brandsStable%':>15}")
for engine, series in per_engine.items():
    core, union = sum(series["core"]), sum(series["union"])
    counts = brands[engine]
    stable_pct = counts["stable"] / (counts["stable"] + counts["flipping"]) * 100
    print(f"{engine:<12}{stats.mean(series['cite']):>12.3f}"
          f"{stats.mean(series['prose']):>13.3f}{stats.mean(series['cv']):>11.1f}"
          f"{f'{core}/{union}':>15}{stable_pct:>14.1f}%")

Stage 4 — Convert Stability Into a Sample Size

Stage 4 turns the spread into a run count, and it does so twice because the two question types behave completely differently.

For a set-membership question — which domains does this engine cite for my category — the useful output is a coverage curve: what share of the domains that five runs surface does one run surface, or two, or three. That is computed directly from the captures by subsampling every combination of repeats.

For a rate question — what share of answers mention my brand — the output is a confidence interval on a proportion. The Wilson score interval is the right tool at these sample sizes: NIST's reference implementation makes it the default method for proportion confidence limits, and a systematic comparison across sample sizes from 1 to 1,000 found the Wilson interval outperforms the textbook Wald interval, particularly at small n and at proportions near 0 or 1. Both conditions describe brand-mention data exactly.

# samplesize.py — measured stability -> a required number of runs
import json
import math
import statistics as stats
from collections import Counter, defaultdict
from itertools import combinations

cells = defaultdict(dict)
for line in open("layers.jsonl"):
    row = json.loads(line)
    cells[(row["engine"], row["prompt_id"])][row["repeat"]] = row


def wilson(hits, runs, z=1.96):
    proportion = hits / runs
    denominator = 1 + z * z / runs
    centre = (proportion + z * z / (2 * runs)) / denominator
    half = z * math.sqrt(
        proportion * (1 - proportion) / runs + z * z / (4 * runs * runs)) / denominator
    return max(0.0, centre - half) * 100, min(1.0, centre + half) * 100


coverage = defaultdict(list)
for reps in cells.values():
    keys = sorted(reps)
    universe = set().union(*[set(reps[k]["cited_hosts"]) for k in keys])
    if not universe:
        continue
    for n in range(1, len(keys) + 1):
        coverage[n].append(stats.mean([
            len(set().union(*[set(reps[k]["cited_hosts"]) for k in combo])) / len(universe)
            for combo in combinations(keys, n)
        ]))

print("SET QUESTION — share of the 5-run cited-domain union that n runs surface")
for n in sorted(coverage):
    print(f"  n={n}: {stats.mean(coverage[n]) * 100:5.1f}%")

observed = Counter()
for reps in cells.values():
    keys = sorted(reps)
    for brand in set().union(*[set(reps[k]["brands_present"]) for k in keys]):
        observed[sum(1 for k in keys if brand in reps[k]["brands_present"])] += 1

total = sum(observed.values())
print(f"\nRATE QUESTION — {total} brand-by-cell observations at n=5")
for seen in sorted(observed):
    low, high = wilson(seen, 5)
    print(f"  seen {seen}/5 in {observed[seen]:>3} cases ({observed[seen] / total * 100:4.1f}%)"
          f"  95% interval {low:5.1f}%-{high:5.1f}%  width {high - low:4.1f}pt")

print("\nRATE QUESTION — interval width at a 40% appearance rate, by run count")
for n in (1, 5, 10, 20, 30, 50, 100, 200):
    low, high = wilson(round(0.4 * n), n)
    print(f"  n={n:<4} {low:5.1f}%-{high:5.1f}%  width {high - low:4.1f}pt")

What the Matrix Actually Showed

Everything moved, and the engines did not move by the same amount.

Pooled across all 20 cells, the mean pairwise Jaccard on cited domains was 0.384. The cell-level spread is the more useful figure: 0.114 at the low end and 0.956 at the high end. A stability claim taken from one engine on one prompt can land anywhere in that range, which is precisely why a three-run probe on a single engine is a lead and not a finding.

The coverage curve is the number most worth writing down:

One capture shows you about half the domains that five captures show you. Per engine at n=1 the range runs from 73.6% on Perplexity down to 35.1% on ChatGPT, so a single-capture citation audit on the least stable engine is missing roughly two thirds of what it is supposed to inventory. The curve also flattens fast: the third run adds 12.8 points, the fourth adds 9.2. For a citation inventory, three to four repeats buys most of what is available.

Google AI Mode produced by far the widest domain union — 182 distinct hosts across four prompts, with only 11 appearing in all five repeats of their cell. Google's own guidance explains the mechanism: both AI Mode and AI Overviews may use a query fan-out technique that issues multiple related searches across subtopics and can therefore surface a wider and more varied link set than classic search. When an engine is built to spread its citations across subtopics, one capture samples only part of that spread.

The rate layer tells the opposite story. Across 137 brand-by-cell observations, only 44.5% were present in all five repeats. 24.8% appeared in exactly one of five — a name that a single capture would report either as a confident presence or a confident absence, depending entirely on which capture you took.

A brand seen twice in five runs has a true appearance rate somewhere between 12% and 77%, which is too wide to support any statement a stakeholder would act on. Widening the sample narrows it slowly:

So the answer to "how many runs" is: it depends which layer you are reporting, and the two answers differ by more than an order of magnitude. A citation inventory is readable at three to four repeats. A rate you intend to compare week over week needs on the order of 100 captures per engine per prompt before the interval tightens to ±10 points. Daily captures for two to three weeks land at roughly 14 to 21 — comfortably enough for the set question, and about five times short for the rate question.

Two smaller results are worth recording. Rank moved less than mention did: among names present in more than one repeat, the first-mention position shifted by a mean of 0.89 places on ChatGPT and 1.95 on Grok, so a rank-tracking metric inherits the mention instability but adds only a little of its own. And one cell — a single engine on the cloud-browser prompt — named no watchlist brand at all in any of its five repeats, answering in capability categories instead of vendors. A consistent absence is a stable reading too, and a monitor that treats an empty result as a failed capture will throw away real signal.

The measured spread also lines up with published research rather than contradicting it. A variance-components study that decomposed 12,933 LLM brand responses across 20 brands, 8 languages, and 3 models found resampling alone accounted for 34.8% of total variance, with the reliability of a single response sitting near 0.01. That study used a different corpus and a different statistical method and still landed in the same place: one answer carries almost no brand-discriminating signal.

If you are already running the per-engine brand sentiment pipeline or the AI recommendation rank tracker, both read the prose layer, and both therefore inherit the slow-converging sample size rather than the fast one.

Getting your own version of this table takes about a hundred calls. Start on the Scrapeless free plan and run the matrix against your own prompts before you set a target on any of these metrics.

What This Measurement Cannot Tell You

The compressed capture window is the limitation that governs every figure above. All 100 captures landed inside 4 minutes 9 seconds on a single day. That isolates one source of movement cleanly — the model regenerating its answer — and it excludes several others entirely: index updates, news cycles, content you publish, model version changes, and any genuine day-to-day drift in what the engines retrieve.

The consequence runs in one direction only. These numbers are a floor, not an estimate of total variance. A program sampling the same prompts across weeks is exposed to model-level nondeterminism plus everything the four-minute window held constant. Required sample sizes for a real monitoring program are therefore at least this large, and the gap between a four-minute figure and a four-week figure is itself a measurement nobody here has made.

Four narrower limits apply:

• Twenty cells is a small design. Per-engine figures are means over four prompts, so an engine's number is not a precision estimate of that engine. The cell-level range from 0.114 to 0.956 is the honest summary of how much a single cell can differ.
• One category, one language, one country. All prompts sat in one product category, in English, pinned to US. The variance-components work cited above found query language carried 26.5% of total variance, so a different language should not be assumed to behave like this one.
• The Wilson interval assumes independent draws. Five requests fired concurrently in one session may share upstream state, which would make the effective sample smaller than the nominal one and the true intervals wider than the table shows.
• The brand layer depends on the watchlist. Only names on the list are counted, so a competitor you did not list is invisible to L3 while still being fully visible to L1 if the engine cites its domain.

None of these break the headline result, which is a comparison between layers measured under identical conditions. They do mean the specific percentages belong to this matrix, not to AI answers in general.

Putting the Number Into a Monitoring Schedule

Schedule the two question types separately, because running one cadence for both overpays for the set question and underpowers the rate question.

For citation inventory, three to four repeats per engine per prompt, weekly. That recovers 83% to 92% of the cited-domain universe per capture round, and the week-over-week diff on that set is readable. Store the union per round rather than a single run's list; comparing two single runs generates changes that are mostly resampling.

For rate metrics — mention share, share of voice, sentiment share, rank — accumulate rather than snapshot. Twenty captures per cell per week gives an interval near ±20 points on a weekly figure and a pooled monthly figure near ±10. Report the interval next to the number every time. A dashboard that shows 42% without showing 31%–50% is asserting a precision the capture design does not support.

Whichever cadence you choose, pin everything. Same prompt strings, same country, same reasoning mode. The input fingerprint from Stage 1 belongs in your stored rows, so that a future analyst can tell a real change from a prompt someone edited.

Re-measure when an engine changes. Stability is a property of the current model and retrieval stack, not a constant. When an engine ships a visible change, the run count that was sufficient last quarter is a hypothesis again. The same discipline applies here as in any model evaluation, where curating a representative dataset and choosing a meaningful methodology come before the metric, not after it.

Conclusion: Report the Interval, Not Just the Number

The useful finding is not that AI answers move. Every guide already says that. The useful finding is that they move by different amounts at different layers, and that the required sample size splits accordingly: a handful of runs for a citation inventory, around a hundred for a rate you intend to defend in a meeting.

The design behind that split is four files and a hundred calls. Capture with the inputs pinned, reduce to comparable layers, score the movement, convert to a run count. Run it once against your own prompt set and you stop guessing at a cadence, because you will have the one number every AI-visibility dashboard is currently missing — the error bar.

Ready to Measure Your Own AI-Visibility Variance?

Five answer engines sit behind one endpoint and one response envelope on the Scrapeless Universal Scraping API, which is what makes a hundred-capture matrix a single afternoon rather than five separate integrations. Parameters and response shapes for every actor used here are in the Scrapeless API documentation.

Create a free Scrapeless account and put an error bar on your visibility numbers.

FAQ

Q: How many runs does an AI-visibility number actually need?

It depends on the layer, and the two answers differ by more than an order of magnitude. In this matrix, three to four repeats per engine per prompt recovered 83% to 92% of the cited domains that five repeats surfaced, so a citation inventory is readable at that sample size. A rate metric such as mention share needs roughly 100 captures per engine per prompt before a 95% interval narrows to about ±10 points, and five captures leave it around 65 points wide.

Q: Why do AI answers change when the prompt and settings are identical?

The models sample from a probability distribution and the retrieval layer feeding them is not fixed either, so identical inputs do not guarantee identical outputs. Work that pinned five models to nominally deterministic settings across eight tasks still recorded accuracy swings of up to 15% between runs, and attributed the effect to how inference batches are processed rather than to a configuration mistake.

Q: Are cited sources more stable than the answer text?

Not reliably, and not by enough to skip repeat sampling on either layer. Cited-domain sets in this matrix overlapped at a mean pairwise Jaccard of 0.384 while a single capture surfaced only 49.6% of the domains five captures surfaced, so the citation layer moves substantially in absolute terms. It converges faster than a rate metric does, which is a different claim from being stable.

Q: Does the engine I track change how many runs I need?

Yes, by roughly a factor of two on the citation layer. A single capture surfaced 73.6% of the five-run domain union on the most stable engine in this matrix and 35.1% on the least stable, so a run count chosen for one engine will be wrong for another. Compute the coverage curve per engine rather than applying one figure across the board.

Q: Does this measure day-to-day drift as well?

No. All 100 captures ran inside 4 minutes 9 seconds, which isolates model-level nondeterminism and excludes index updates, news cycles, content changes, and model versions. Treat every figure here as a floor: a program sampling across weeks carries this variance plus everything the short window held constant.

Q: Is capturing AI answers this way legal?

Capturing publicly available AI answers to prompts you author is ordinary public-web data collection, and the prompts and responses here contain no personal data. Keep volume bounded and proportionate to the measurement, store only what the analysis needs, and check the terms that apply to your own jurisdiction and use case before running a program at scale.

Q: Can I run this without an AI agent or SDK?

Yes. Every script here is Python standard library against one HTTPS endpoint, so the whole pipeline runs from a cron job or a CI schedule with nothing installed beyond Python. The only external dependency is the API key.