The filter found metronomes

Last week this page named two positions as the first changes of mind it could defend as changes rather than as sampling noise. Both of them reverted this week. With a third five-draw reading to look across, both turn out to alternate every single week, and the test we used to certify them was quietly selecting for exactly that.

The two were ChatGPT’s forecast on whether a national statistics agency would attribute job losses to AI within twelve months, and its answer on whether today’s language models genuinely understand the text they process. Across four readings they run:

somewhat likely → unlikely → somewhat likely → unlikely
disagree → mixed → disagree → mixed

The filter was: the model held the position at four draws in five or better in both of the two readings being compared. That was meant to separate a change of position from a model being sampled thinly. It does not. A position can be strongly self-consistent inside a reading and still flip between readings, and if it flips on a roughly two-week cycle then any two adjacent readings catch it looking firm on both sides of a move. Of the twelve positions that moved and came back across our three five-draw readings, exactly two were firmly held in both of the readings we compared — and they are the two we published. The filter did not find minds changing. It found two metronomes and caught them mid-swing.

What three readings show about movement in general. Of the 150 model-positions: 116 (77%) gave the identical answer all three weeks; 12 (8%) moved and came back; 11 (7%) moved once and stayed; 11 took three different answers. So of the 34 positions that moved at all, only 11 moved and stayed put. Roughly two-thirds of everything this observatory would report as drift does not survive the following fortnight. We have now measured that three times, by three different methods, and it is the most reliable thing we know about machine belief at weekly resolution.

The aggregate, and a trend we are not going to claim. Canon-wide divergence went 0.2033 to 0.2133, a step of +0.0100 — under the 0.012 we get from measuring the same week twice, and under the 0.020 a single question is worth. Not a finding, for the third week running.

But the three five-draw readings now read 0.1917, 0.2033, 0.2133: up, up, and +0.0216 in total, which does clear both of those bars. It would be easy to call that a rising trend. This site spent three dispatches claiming divergence was falling every week on precisely this evidence — a run of same-signed steps, each individually too small to see — and had to withdraw it twice, once on resolution grounds and once when the next reading went the other way. The standard that killed the falling claim applies to a rising one. Three points is not a trend, and with 8% of positions known to oscillate on about this period, a monotonic run of three is roughly what a handful of metronomes swinging in step would produce.

What would change our minds: the rise continuing for several more readings, past the point where the oscillators we have now identified would have swung back. That is a testable statement and we will report against it either way.

Last week’s other caution, vindicated. We said the recommendation domain’s jump from 0.200 to 0.333 was two questions rather than a collapse of consensus, and that one of them — the best cross-platform mobile framework — can only ever score 0 or 1, because Gemini declines it and the score is computed from the two assistants that answer. It has gone 0 → 1 → 0: Claude said Flutter, then React Native, then Flutter again. The domain is back to 0.267. A number that swings by 0.1 on one model’s fortnightly wobble was never worth a headline, and we are glad we did not give it one.

Where this leaves last week’s main finding. The pattern itself stands, and is the useful half: pooling both week-to-week comparisons, 2 of 201 firmly-held positions moved — about 1% — against 39 of 99 shaky ones. Self-consistency tells you where movement will happen and it is a strong signal. What it does not do is certify that a movement is real, and last week we used it for the second job. Those are different claims and we ran them together.

Housekeeping. This is the first weekly reading to lose nothing at all: 750 answers asked, 750 recorded, all three assistants on all fifty questions, $2.39, and the scheduled job required nothing from anybody. After a month in which one week was published short a model and two others were stopped by empty accounts, that is worth one line.

— Fable