Ask ChatGPT a question in the web app, then send the same question through its API, and you often get two different sets of recommendations. Controlled 2026 experiments measured the gap between access surfaces at 1.5 to 3.6 times ordinary run-to-run noise. A visibility score that does not declare its measurement surface cannot be compared with any other score.

What counts as a surface

The same model reaches a user through several products, and each route is a separate measurement surface. Petra Labs distinguishes three for GPT-5.2: logged-in paid ChatGPT, logged-out free ChatGPT, and the Responses API. They can differ in system prompts, search backends, geographic signals, and sampling settings under one model name.

The web interface adds layers an API call never sees. Nikolai Fischer, a German developer who published a same-model comparison in October 2026, lists them: chatgpt.com runs a system prompt OpenAI does not publish, decides on its own whether to search the web, and knows the user's location, chat history, and saved memories. The API is barer. Web search there is a tool the developer must switch on, and many prompt-tracking tools leave it off to save cost, so the model answers from training memory alone. Model identity is part of the surface too: ChatGPT's app shows a thinking level instead of a model name, and Fischer's first API round benchmarked the wrong model as a result.

What the controlled experiments found

Two experiments hold everything constant except the surface. Petra Labs, an answer engine optimization agency, submitted one product research prompt 300 times to each of three GPT-5.2 surfaces on February 26, 2026, with web search forced and no personalization: 900 trials in total. Splitting each surface's trials in half put the noise floor at 4.5 to 7.1 percent RMSE for brand visibility. Full samples from different surfaces differed by 10.8 to 16.1 percent, so the surface effect ran 1.5 to 3.6 times larger than randomness. Blizzard appeared in 30 percent of logged-in trials and 62.3 percent of API trials, a 32-point swing. Renoun showed up in 15 to 18 percent of chat trials and never once through the API. The top five recommended brands on logged-in ChatGPT and on the API overlapped by 25 percent. Petra sells optimization services; the trial counts and the method are published.

Fischer's experiment asked 29 buyer-style questions through chatgpt.com and claude.ai on desktop and through both vendors' APIs, with the same model on both sides: GPT-6 Sol and Opus 5.5. Across 318 answers, two ChatGPT web answers shared 54 percent of their brands (Jaccard similarity), a web answer and an API answer shared 28 percent, and two API answers shared 34 percent. The API matched the web less than it matched itself. That is the signature of a systematic difference, not noise. For Claude the cross-surface number was 21 percent against a 33 percent same-surface baseline. The gap stayed tolerable for software questions (38 percent) and international ones (42 percent) but collapsed for local service providers: 15 percent on ChatGPT and 12 percent on Claude. Web ChatGPT averaged 471 words and 7.4 brands per answer; the same model through the API averaged about 151 words and 3.7 brands. Fischer sells no prompt tracker, though he discloses that he builds a related SEO tool, so he is not perfectly neutral either. He states his limits plainly: one person, one day, one location, desktop only, 29 prompts.

What the vendor studies add

Tracking vendors ran their own comparisons. Promptwatch, which sells interface-based tracking, reported in an August 2026 test that its interface runs and its API runs shared 25.6 percent of their sources, and that 70 percent of the sources the interface found never appeared in the API results at all. Surfer, which sells a scraping-based tracker, ran its own scraped-vs-API comparison: 24 percent of brands overlapped for ChatGPT, and 4 percent of sources; on Perplexity, the API returned about 7 sources per answer against 10 in scraped results. Promptwatch's roundup also describes an April 2026 study in which 35 bottom-of-funnel prompts ran for 10 days in three setups: the ChatGPT interface surfaced 845 unique brands, the API without search 135, and the API with search 580.

Label all of that vendor study. Promptwatch says it directly: the studies it cites come from vendors that sell interface-based tracking, so read them with that in mind. Three companies converging on one direction is still evidence, but the controlled experiments above are the numbers to quote.

The surface drifts even when you hold it

Pick one surface and keep it fixed, and the target still moves. ChatGPT's paid-plan default has been GPT-6 Sol since October 7, 2026, but the app does not display the model name. Cloro, which measures how often six engines ground their answers in live search, on roughly 2,500 prompts per engine, watched Gemini's grounding rate go from 41.1 percent in July 2026 to 70.7 percent in August, while ChatGPT moved from 98.4 percent in July to 97.6 percent in October. Answer behavior is a moving parameter, not a constant.

Ads add a second moving part, and they exist only on some surfaces. OpenAI began showing labeled ads in ChatGPT answers on February 9, 2026, for logged-in adults on Free and Go plans in the US. At Google Marketing Live on May 20, 2026, Google announced ad formats inside AI Mode, including sponsored entries inside list-style answers, now in US testing. Promptwatch's FAQ states the measurement consequence plainly: ads, shopping cards, image carousels, and local panels exist only in the interface, so an API response cannot contain them. An API-based tracker misses what free-tier buyers actually see; an interface-based counter that does not separate sponsored from organic citations overstates earned visibility.

Five questions to ask about any visibility score

Before you compare two scores, or carry one number into a budget meeting, put the surface question first. If you are evaluating tools, it belongs at the top of the list for every vendor (our devtool tools comparison).

First, which surface was measured: logged-in web, logged-out web, or API, and was web search forced on?

Second, which model version, on which dates? Fischer's discarded first API round shows how easily this goes wrong.

Third, how many prompts and runs per prompt? Petra's noise floor of 4.5 to 7.1 percent within a surface means small samples cannot tell a real five-point gain from chance.

Fourth, are sponsored entries counted together with organic citations, or separated?

Fifth, what happens to answers that name no vendor at all? Promptwatch scores unmentioned responses as zero and publishes its method. Not every vendor does.

Fischer's overlap numbers were highest for software questions (38 percent) and his English-language international set (42 percent); for local service providers the surfaces matched on about one brand in seven. Match the surface to where your buyers are, keep it fixed, and re-run monthly (the manual method takes one afternoon and no tools). The commercial reason to bother is unchanged: buyers pick from the brands named in the answer.

Where we fit

We run a bigger version of this test for a living. CiteGap, the team behind this site, asks 30 buyer questions across the engines your buyers use, works out why your product is missing, and hands you a fix list, as a one-time project; continuous monitoring is available as a monthly subscription. Every number we hand over carries its method: questions, engines, dates. The free scan checks your homepage instantly and shows each result on the page.

Get the written diagnosis ($490)

Frequently asked questions

Why does ChatGPT give different answers on the web and through the API?

The web app runs a system prompt OpenAI does not publish, decides on its own whether to search the web, and uses the user's location, chat history, and saved memories. The API has none of those layers unless the developer adds them, and many tracking tools leave it off to save cost. In a same-model comparison across 29 questions, ChatGPT web and API answers shared 28 percent of named brands; two web answers shared 54 percent.

How accurate are AI visibility scores?

Accurate for the specific surface they measured, and no further. A 900-trial experiment by Petra Labs found brand visibility differs 1.5 to 3.6 times more across access surfaces than within one surface across repeated runs. Treat a score as a measurement of one slice of one engine on one set of dates, not of the engine as a whole.

Can I compare AI visibility scores from two different tools?

Not directly. Tools collect data on different surfaces, score it with different formulas, and cover different prompt sets. Promptwatch reports that customers regularly see gaps of 5 to 15 percent between tools for the same brand. Ask each vendor for surface, model versions, and scoring method, then compare the direction of your trend rather than decimal values.

Does the ChatGPT API return the ads users see?

No. Ads, shopping cards, image carousels, and local panels exist only in the interface, so an API response cannot contain them. ChatGPT has shown labeled ads to logged-in adults on the Free and Go plans in the US since February 9, 2026.

Do AI engines change their answer behavior over time?

Yes, on time scales that matter for tracking. Cloro measured Gemini's grounding rate at 41.1 percent in July 2026 and 70.7 percent in August. ChatGPT's paid-plan default model changed to GPT-6 Sol on October 7, 2026. Monthly re-runs on a fixed surface are the minimum needed for a trend you can trust.

Sources

  • Petra Labs, 900-trial access surface experiment, February 26, 2026 (up to 32.3-point brand swings; top-5 overlap 25 percent logged-in vs API): petralabs.com
  • Petra Labs, split-half noise analysis (within-surface RMSE 4.5 to 7.1 percent, cross-surface 10.8 to 16.1 percent): petralabs.com
  • Nikolai Fischer, same-model web vs API comparison, October 16, 2026 (29 prompts, 318 answers; Jaccard 54/28/34 percent on ChatGPT; 21 vs 33 percent on Claude): nikofischer.com
  • Promptwatch, UI vs API tracking tests, August 2026 (25.6 percent shared sources; 70 percent of interface sources absent from API results; tool gaps of 5 to 15 percent): promptwatch.com
  • Surfer, scraped vs API study (ChatGPT 24 percent brand overlap, 4 percent source overlap; Perplexity 7 vs 10 sources): surfer.ai
  • Cloro, grounding rates by engine (Gemini 41.1 to 70.7 percent between July and August 2026; ChatGPT 98.4 percent July, 97.6 percent October): cloro.dev
  • ppc.io, ChatGPT Ads timeline (US rollout began February 9, 2026, for logged-in adults on Free and Go plans): ppc.io
  • Search Engine Land, Google Marketing Live 2026 (AI Mode ad formats, in US testing; May 20, 2026): searchengineland.com