How much of a month's movement is real?

Asking more didn't change the answer was about the first axis of noise: a single sample per question is coarse, so the index leans on the number of questions, not repeats, to average it out. A second monthly run adds a second axis. Now a score can move between snapshots, and the same question applies: is a five- or ten-point swing a real change in what the assistants recommend, or is it the kind of wobble you'd get re-running the same prompts on a stochastic model twice?

There's a specific reason to worry about this more than the k=1 case alone. Google's AI Overview is captured on only six prompts a month, against roughly thirty-five for each API engine, but it counts as one of five equal columns in the score average. One extra mention on Google AI is a 17-point swing in that column. Averaged in equally with four engines that each sample six times as many questions, a single extra AI Overview hit can move a score more than a real, broad shift on the larger engines does. That's not a flaw I'm fixing this month, it's a property of the design worth naming so a reader can weigh a score change accordingly.

The check I used

For every app that moved more than a few points, I looked at the per-engine breakdown rather than the headline score. A swing counts as broad if at least two of the four 35-question API engines moved several points in the same direction. A swing where the API engines are flat or split, and the movement is really the six-question Google AI column, gets flagged as unconfirmed rather than written up as a finding.

App	Category	Score Δ	Largest engine moves	Read
Strong	fitness	+7	ChatGPT+11pt, Claude+5pt	Broad
Hevy	fitness	+5	ChatGPT+8pt, Claude+6pt	Broad
Picsart	photo	−10	Claude-9pt, Mistral-9pt	Broad
Aura	security	+6	Claude+9pt, ChatGPT+3pt	Thin
Fitbod	fitness	+6	Google AI+33pt, Mistral+3pt	Concentrated
Boostcamp	fitness	−9	Gemini-20pt, Google AI-33pt	Concentrated

A score that moves for a real reason and a score that moves because a six-question capture caught one different answer look identical in the headline number. They only separate once you open the engine columns.

Where I got this wrong the first time

The fitness run record originally grouped Hevy, Fitbod and Strong together as a broad, three-app gain. That was wrong for Fitbod. Its ChatGPT rate actually fell three points this month and Claude was flat; the entire headline gain is the Google AI column going from one hit in six to three in six. I caught it writing the comparison table above, corrected the run record, and split Fitbod out of the Hevy/Strong write-up. I'd rather show the correction than the version that quietly reads better.

Aura, in security, is the case in between. Three of its four API engines ticked up this month, Claude the most at nine points, but none of the individual moves are far from what a count or two of difference produces on 33 questions. Nothing here contradicts the "broad gain" story, but nothing confirms it either. I'm logging it as thin rather than either broad or concentrated, and I'd want a third month before treating Aura's rise as established.

What this doesn't cover

The reversals written up in AI self-preference and CapCut or Splice are a related but different case: there, Google's AI Overview didn't just shift a count, it changed which app it named first, in the actual verbatim text, on the same query both months. A qualitative flip like that is stronger evidence than a raw count swing on the same six samples, even though it's still a small sample. The broad/concentrated check above is for reading a leaderboard's score movement; it isn't the only way a small sample earns your trust.

This whole check is a working rule, not a validated statistical threshold. With two months of data there's no way to compute how often a "broad" two-engine agreement itself reverses by a third month. The honest position is that this run gave me a way to sort this month's movement into "worth a claim" and "worth watching," not a proof that the sorting is correct. I'll know more after August.

What this is

A cross-category read of the June-to-July movement in all five indexes, using the per-engine counts behind each published score. Not a new data collection: everything here comes from the same snapshots and run records already published for fitness, photo and security.