# How many runs

> One run of a prompt is a sample, not a measurement: the same question sent to the same model returns a different set of brands often enough that a single answer cannot be read as the model's opinion.

Canonical page: https://llmaudit.app/learn/how-many-runs

One run of a prompt is a sample, not a measurement: the same question sent to the same model returns a different set of brands often enough that a single answer cannot be read as the model's opinion. How many runs a measurement needs depends on what it is trying to resolve, and the honest answer for most teams is "more than one, and fewer than you fear".

## What our data shows

Across 4,500 calls, the same question sent to the same model returned the same set of brands in 15 percent of cases. The first-named brand held in 75 percent of runs. Read together: the leader is fairly stable, the rest of the list is not, and a single run says little about anyone below first place.

## What three runs can and cannot resolve

With 3 runs, a brand's appearance on a question reads as 0, 1, 2, 3 of 3. That is enough to separate "always named", "never named" and "sometimes named", which is the distinction that decides whether there is a problem. It is not enough to tell 40 percent from 60 percent, to rank brands below the leader, or to treat a change of one appearance between two measurements as movement. Those need many more runs per question, and for most teams that budget is better spent covering more questions.

## How LLM Audit runs

The snapshot runs 3 times per provider at a low temperature (0.2) with a fixed JSON format. The buyer-question map runs once per question per provider at temperature 0 on every model since 2026-08-23, because it trades depth on one prompt for breadth across the buyer's questions, and every count is shown with the number of runs it came from. Re-runs are self-serve, so a team can repeat a measurement after publishing instead of waiting for a schedule. The methodology page has the models, parameters and limits.

## Prior work

Other people measured the same thing and reached the same place.

- SparkToro and Gumshoe, reported by Search Engine Land: AI recommendation lists rarely repeat. https://searchengineland.com/ai-recommendation-lists-rarely-repeat-study-468076
- Schulte, Bleeker and Kaufmann, "Don't Measure Once", arXiv 2604.07585. https://arxiv.org/abs/2604.07585
- Sielinski, arXiv 2603.08924. https://arxiv.org/abs/2603.08924
- Our own repeatability study, with the 4,500 calls and how they were counted, is on the research page linked below.

## Measured

**15 percent**: of cases, across 4,500 calls, in which the same question to the same model returned the same set of brands; the first-named brand held in 75 percent of runs. Source: https://llmaudit.app/research/how-repeatable-are-ai-recommendations

## FAQ

### Is one run ever enough?

Only as a first look, and only if it is labeled as one run. It is enough to read what an assistant says about a brand and to find the questions worth measuring properly. It is not enough to state a share, a rank or a change.

### Why not ten runs per question?

Cost, and the trade against breadth. Ten runs of one prompt resolve that prompt well and say nothing about the other eleven questions a buyer asks. The audit puts the budget on the map first and keeps re-runs cheap.

### Does temperature 0 fix it?

It reduces variation; it does not make the set of brands repeat every time, and the model behind an API can change between calls. Run counts are the honest answer; temperature is a setting.

## Related

- [Share of voice in AI answers](https://llmaudit.app/learn/share-of-voice-in-ai-answers.md)
- [AI visibility audit](https://llmaudit.app/learn/ai-visibility-audit.md)
- [How repeatable are AI recommendations](https://llmaudit.app/research/how-repeatable-are-ai-recommendations.md)
- [Methodology](https://llmaudit.app/methodology.md)
- [All definitions](https://llmaudit.app/learn.md)

---

Machine-readable index: https://llmaudit.app/llms.txt · Full context: https://llmaudit.app/llms-full.txt · All URLs: https://llmaudit.app/sitemap.xml · Developer and agent resources: https://llmaudit.app/developers.md