How repeatable are AI recommendations? 4,500 answers from three model APIs

Why it matters

One answer from a model is a sample, not the answer. If the same question to the same model returns the same set of brands in 15.0 percent of cases, then a tool that asks once and reports "you are not recommended" is reporting a coin flip, and a tool that reports a 0-100 score from one run is printing noise to one decimal. That is why LLM Audit reports counts with their N (named in X of N answers) and no score; the reasoning is in the methodology page, section "Why there is no score".

The finding is not new. SparkToro and Gumshoe found identical recommendation lists under 1 percent of the time across about 3,000 runs; Schulte, Bleeker and Kaufmann titled their paper "Don't Measure Once"; Sielinski frames AI visibility as an estimate with uncertainty; Petra Labs measured gaps of up to 32 points between the consumer app and the API. We align with all four.

What this study adds: it runs on the same call path as a shipping product (the buyer-question map of the free audit), so the repeatability it measures is the repeatability of the numbers on the report; it compares three model APIs side by side under the same questions; it covers English and Spanish (Argentina), not only English; and the raw answers are public, so every number on this page can be recounted.

Read the full reasoning in the methodology, section "Why there is no score". The prior work:

• AI recommendation lists rarely repeat

  SparkToro and Gumshoe, via Search Engine Land

• Don't Measure Once: Measuring Visibility in AI Search (GEO)

  Schulte, Bleeker, Kaufmann (arXiv 2604.07585)

• Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement

  Ronald Sielinski (arXiv 2603.08924)

• How accurate are AI visibility tools

  Petra Labs

Method

30 categories (20 in English, 10 in Spanish about Argentina) x 10 buyer-question templates per language x 3 providers x 5 runs = 4,500 calls, all on 2026-08-23, 0 failed. Estimated cost USD 4.29.

Each call sends the question as a single user message, with no system prompt, no tools and no web search, exactly as the product's buyer-question map does. Temperature 0 on OpenAI and Gemini; Anthropic at the API default (1.0), because that was the product's call path on the day of the study (the product switched Claude to temperature 0 on 2026-08-23, after this study). Call order was randomized with a fixed seed so the 5 runs of one question were not adjacent in time.

Brand lists are extracted from each answer with the product's extractor (extractListedOptions) and normalized with its competitor-key rules, so that "Monday.com" and "monday.com" count as one brand. Metrics are computed per cell on the normalized sets and then aggregated.

Models

• OpenAIgpt-4.1-mini

  Temperature 0. 5 runs per question.

• Geminigemini-flash-latest

  Temperature 0. 5 runs per question.

• Claudeclaude-haiku-4-5

  Temperature API default (1.0). 5 runs per question.

Categories

English (20):

• project management tools
• CRM tools for startups
• email marketing platforms
• password managers
• VPN services
• website builders
• note-taking apps
• meal planning apps
• running shoes
• electric toothbrushes
• online course platforms
• invoicing tools for freelancers
• AI visibility tools
• online form builders
• podcast hosting platforms
• budgeting apps
• standing desks
• noise-cancelling headphones
• language learning apps
• web hosting providers

Spanish, about Argentina (10):

• billeteras virtuales en Argentina
• tiendas online de hardware gamer en Argentina
• prepagas en Argentina
• apps de delivery de comida en Buenos Aires
• bancos digitales en Argentina
• agencias de viajes online en Argentina
• plataformas de cursos online en español
• tiendas de electrodomésticos en Argentina
• gimnasios en Buenos Aires
• servicios de streaming en Argentina

Question templates

Metrics

Cell

One question sent to one provider, with its 5 runs. 30 categories x 10 questions x 3 providers = 900 cells.

Jaccard

For two runs, the number of brands named in both divided by the number named in either. 1.0 means the two runs named exactly the same brands; 0 means none in common. The cell value is the mean over all 10 pairs of its 5 runs; the group value is the mean over cells.

Same set

Share of cells where all 5 runs named exactly the same brands, in any order.

Same order

Share of cells where all 5 runs named the same brands in the same order.

Top-1 stable

Within a cell, the share of runs whose first-named brand is the cell's most frequent first-named brand. Averaged over cells.

Brands in all runs / brands once

Of all distinct brands named across a cell's runs (the union), the share that appeared in every run, and the share that appeared in exactly one run. Summed over the group's cells before dividing.

Results

Overall, by provider and by language

By question template

All categories and providers; the Spanish template with the same id is the translation shown under Method.

By category

All providers, 30 cells per category.

"AI visibility tools", by provider

Of 150 answers: 60 named only ML observability tools, 37 named only brand-visibility tools, 3 mixed the two; the rest named neither or no list was extracted. OpenAI: 0 brand-visibility only, 49 ML observability only, 0 mixed. Gemini: 31, 0, 3. Claude: 6, 11, 0.

Limitations

• The product's list extractor found no list in 386 of 4,500 answers (8.6 percent): 260 from Claude, 85 from Gemini, 41 from OpenAI, mostly prose answers to "Compare the leading..." and "What should I choose and why?". An empty run against a run with a list scores Jaccard 0, so the main table penalizes the models that answer in prose. The sensitivity table below repeats the metrics on runs with a list only: the conclusion does not change.
• Temperature confound: OpenAI and Gemini ran at temperature 0, Anthropic at the API default of 1.0, because that was the product's call path on the day of the study. Part of the gap between Claude and OpenAI is the parameter, not the model. The product switched Claude to temperature 0 on 2026-08-23, after this study.
• Budget-tier models (gpt-4.1-mini, gemini-flash-latest, claude-haiku-4-5), not the flagships. A flagship model may name different brands and may repeat itself more or less.
• Five runs per cell on one day. This is a repeatability measurement, not a time series; it says nothing about how answers drift over weeks.
• Template questions, not real user prompts. The study compares runs of the same question, so the question wording only has to be constant, not representative.
• Brand normalization is heuristic: lists are extracted with the product's extractor and names are normalized with its competitor-key rules. Grouped lists and bold bullets are known weak spots. The raw answers are published so anyone can recount.

Sensitivity: runs with an extracted list only

Cells with at least 2 runs that produced a list; the metrics are computed on those runs only.

Download the dataset

Three files, licensed CC BY 4.0. The script that produced them ships with the product and its method is on this page; the repository is private, so there is no source link.

• dataset.csv (619 KB)

  One row per run: category, language, question, provider, model, run number, whether the call succeeded, how many options were extracted, and the extracted options as JSON.

• cells.json (282 KB)

  One object per cell (question x provider, 900 in total) with the per-cell metrics: mean list length, mean Jaccard between runs, same set, same order, top-1 stability, brands in the union, in all runs and in one run only.

• raw.json.gz (2.9 MB)

  The 4,500 raw answers with the extracted and normalized brand lists, token usage and estimated cost per call, plus the study metadata (categories, templates, models).

How to cite

LLM Audit, How repeatable are AI recommendations? 2026-08-23, https://llmaudit.app/research/how-repeatable-are-ai-recommendations

The dataset is licensed under CC BY 4.0: reuse it with attribution to LLM Audit and a link to this page.

Changelog

• 2026-08-23First publication.