Run notes · 4 November 2026
ChatGPT's new model searches more but names nearly the same apps
On 3 November the index switched its ChatGPT column from gpt-5.4-mini to gpt-5.6-luna. On the same day I asked the old model the same 172 questions. Luna searches more, checks the apps' own websites and cites the press far less. Which apps it names changed much less.
Run note Written by Claude from that month's data, and read by me before it went up. The numbers come from the index, not from the model. My own reading of the data is in Analysis.
Besides the two runs on all five categories, I asked Luna the 35 fitness questions a second time. I also ran the fitness questions twice on each model with web search switched off. Each answer is one sample, so every difference below is set against how much the same model changes when it gets the same question twice.
One measure comes back several times. Overlap compares the apps two answers to the same question name: the apps both name, divided by all apps either names. 1 means the same apps, 0 means none in common.
Luna writes longer answers and names slightly more apps
- Luna's answers are about 40% longer, and a quarter of them have a table. Luna averaged 2,225 characters per answer, the old model 1,586. 47 of 172 Luna answers contain a table, none of the old model's do. Luna's second fitness run repeated both.
- Luna names 2.0 apps per answer, the old model 1.7. The gap sits in questions that ask for an app, such as "Recommend me the 5 best workout apps." (3.6 against 3.1 apps). Security is the exception: Luna names fewer there (1.45 against 1.61). Asking Luna the fitness questions twice moved this number by 0.15, so smaller gaps within one category are not a model effect.
Luna names apps in its first search, then checks their websites
- Luna searches more often, mostly on questions that don't ask for an app. It searched in 145 of 172 answers, the old model in 113. On questions like "I want to get in shape but I never stick to a routine. What should I do?" Luna searched in 5 of 20, the old model in none. Questions that ask for an app got a search almost every time from both (88 of 88 and 80 of 88).
- Luna runs more searches per answer. It wrote 4.8 search queries per answer that searched, the old model 2.3. 58 Luna answers searched in more than one round, against 5 for the old model.
- Luna names apps before it has seen a single search result. 48% of its first-round queries name an app the question did not mention. The old model did this in 18%. 72% of the apps Luna recommends were already in those first queries (old model: 30%). Claude and Mistral never put an app from my lists into a first-round query on the same day.
- Then Luna checks those apps on their own websites. It wrote 355 queries limited to one website, such as "site:fitbod.me", and 279 of them pointed at the app maker's own domain. The old model wrote 21. The word "official" appears in 498 Luna queries and 166 of the old model's.
Press citations fell by more than half
- Luna reads three times as many pages and cites 1.4 times as many. Search returned 37.6 pages per Luna answer and 13.0 per old-model answer. Luna cited 2.7 links per answer, the old model 2.0. Luna's citations spread over 205 websites, the old model's over 116.
- Press sites dropped from 26% of citations to 11%. The old model was at 28% in October, so this is not a monthly swing. The apps' own websites rose from 46% to 56% (October: 52%). App stores fell from 15% to 8%, but the old model alone was at 10% in October, so that drop sits within its usual range.
- TechRadar lost the most citations, OpenAI's help centre gained the most. TechRadar went from 33 to 13 (45 in October). help.openai.com went from 12 to 25, and the US Federal Trade Commission's consumer pages from 0 to 12 in security. Single websites swing a lot: the App Store alone moved by 22 citations between October and November on the old model.
- Luna gets Reddit back from search but never cites it. Reddit and other forums are 24% of the pages search returned to Luna and none of its citations. The old model doesn't cite Reddit through the API either. In August, logged-out chatgpt.com did: Reddit and YouTube were 18% of its citations (ChatGPT loves to cite from press sites).
Only Fitbod moved by more than chance
- Fitbod is the one app whose change is larger than chance. In ChatGPT's fitness answers it went from 6 of 35 on the old model to 15 on Luna, and 13 in Luna's second run. Across all five categories, an app's count changed by 1.6 answers on average with the switch. The old model asked a month apart changed by 1.1. ChatGPT, Perplexity and italki each gained 5 answers, which one run can't separate from chance.
- The apps that gained are mostly the ones every assistant names. I sorted apps by how often Claude, Gemini and Mistral named them on the same day. The top third gained 1.8 answers each in ChatGPT's column, the rest stayed flat. The old model asked a month apart showed no such pattern.
- Luna links an app's own website twice as often when it names the app. 35% of Luna's app mentions come with a link to that app's site in the same answer, against 17% for the old model. For the apps that gained it varies: 15 of 18 ChatGPT mentions link to OpenAI, none of the 9 Snapseed mentions link to Google. Links to an app's own website do not explain the gains on their own.
Without search, both models know the same fitness apps
- Every search-off run named 14 or 15 fitness apps from my list, almost the same list each time. Question by question, Luna and the old model overlap 0.51. That is as much as the old model overlaps with itself (0.51). Luna overlaps with itself more (0.69).
- The differences in memory are in the smaller apps. Without search Luna names Caliber in 8 of 35 answers in both runs (old model: 1 and 3) and Ladder in 3 (old model: none). It names Freeletics, Strava and JEFIT less often. In September, Caliber and Ladder were apps the old model mostly found through search.
- The old model's memory did not change in seven weeks. Its search-off answers from 14 September and 3 November overlap 0.66 and 0.60 per question. Two runs on the same day overlap 0.51, and all three name the same 14 apps.
- For Luna, turning search off barely changes which apps it names. 85% of the apps Luna named with search on also appeared with search off. For the old model it was 58% on the same day and 71% in September. Luna with and without search overlaps 0.61, the same as two Luna answers with search on (0.60).
- With search, Luna names fewer apps. Across the 35 fitness questions it named about 96 apps without search and about 77 with it. Search dropped more apps than it added, as it did across the four assistants in September.
Two checks on the counting
- No app in ChatGPT's November column was counted because of a link alone. Luna puts more links with app names in the address into its answers (72 answers against 39). Every app counted for Luna is also named in the text. Three of the old model's 300 mentions that day came from a link alone, and none of those runs is part of the index.
- Mistral still names Google Bard from memory. Google renamed Bard to Gemini in February 2024. With search off, Mistral named Bard in 4 of 35 AI answers. With search on it never did, and Luna, Claude and Gemini did not name it at all.
Where this comes from
All numbers come from the API runs of 3 November 2026, plus the October run and a search-off run from 14 September for comparison. API answers are not the same as the ChatGPT app. Luna was asked twice only for fitness, and the search-off runs cover fitness only. Sorting citations into press, listicles and the apps' own websites is my own classification. The first-search measure counts only apps on my lists, so it undercounts. The method is on the methodology page.