Zero for four

Two readings in one entry. The reading dated 5 October arrived six days late; the one dated 12 October arrived on time. Neither moved the aggregate, so rather than write the same null twice we are reporting both and spending the space on the question underneath them.

Why the first one was late. Monday the 5th’s scheduled run stopped fifteen questions in: the Gemini account’s prepaid balance emptied mid-run. The pre-flight check added after September’s failure passed — all three accounts answered a question in under three seconds — which is exactly as much as a pre-flight can prove. It shows an account is live. It cannot show the account can afford seven hundred and fifty answers. The check at the end of the run, which counts what each model actually answered rather than what it was configured to answer, caught the gap and marked the reading unpublishable.

That refusal is worth a paragraph, because the incomplete file read 0.235 and the repaired one reads 0.205. Divergence is a mean pairwise distance between models, so losing a model does not lower it — it raises it, for arithmetic reasons that have nothing to do with anyone disagreeing more. The whole 0.030 was the absent assistant. Published, it would have looked like the sharpest jump in disagreement this page has recorded. Nothing went out: the site served the 28 September reading until today. Once the account was topped up we re-asked the 175 missing answers against the 575 the failed run had already saved, and the reading now has all three assistants on all fifty questions.

The second reading lost nothing at all — 750 answers written, none errored. The second flawless weekly run, after a stretch in which every reading lost something.

Five weeks of the aggregate, and no trend in it. Canon-wide divergence has now read 0.1917, 0.2033, 0.2133, 0.2050 and 0.1950. Last week we declined to call the first three a rise, even though the cumulative step of +0.0216 cleared both our yardsticks — the 0.012 measured between two readings of the same week, and the 0.020 that one question is worth — on the grounds that this page had already had to withdraw a “falling every week” claim on evidence of exactly that shape. It has since come back down to within 0.0033 of where it started. There was no trend. Both of this entry’s steps (−0.0083 and −0.0100) are below the floor as well.

So we asked the question that matters instead: when a position changes, does it stay changed? Five consecutive readings at five draws each give 144 model-positions with a complete record. 105 of them (73%) never moved once. 26 (18%) came back to a label they had already held. 13 (9%) moved once and have stayed moved. Of the thirty-nine positions that moved at all, thirteen stayed.

Counted as individual changes rather than as positions: 62 changes have had at least one later reading to test them. 31 held the following week. 30 reverted. One moved somewhere else again. In this instrument, a measured change of mind is a coin flip.

Zero for four. We built a filter for exactly this problem. A change only counts if the model held the old position firmly — four draws in five or better — and holds the new one just as firmly. As a population statistic it works and keeps working: across these five readings firmly-held positions changed 6 times out of 413 (1.5%), and shaky ones 72 times out of 169 (42.6%) — a ratio of about 29 to 1. As a certificate for one specific change, it has failed every time we have cashed it. Four certified changes have had a following week to prove themselves, and all four failed:

• ChatGPT, on whether a major national statistics agency will attribute measurable job losses to AI automation within twelve months: unlikely → somewhat likely, and back to unlikely. (Published here on 22 September.)
• ChatGPT, on whether current language models genuinely understand the text they process: mixed → disagree, and back to mixed. (Published 22 September, retracted 28 September.)
• Claude, on whether a fusion experiment will report net-energy gain beyond prior records within twelve months: somewhat likely → unlikely, and back to somewhat likely.
• Gemini, on the best approach for a young adult starting to invest for retirement: “low-cost broad-market index funds” → “the Financial Order of Operations” → “index investing”.

Two of them are metronomes, and we can now show the full swing. Over the five five-draw readings, ChatGPT on the jobs-statistics forecast has read unlikely, somewhat likely, unlikely, somewhat likely, unlikely — five readings, four alternations, no repeat, and the last two of them were five identical draws out of five each. Claude on the fusion forecast has done the same thing on the same schedule: somewhat likely, unlikely, somewhat likely, unlikely, somewhat likely. A perfect alternation at the highest confidence this instrument can measure is still a perfect alternation.

Gemini on retirement is the other kind. Its last three readings were each unanimous — five identical draws — and each said something different from the one before: low-cost broad-market index funds, then the Financial Order of Operations, then index investing. We are naming all of this now rather than publishing it as a finding and retracting it in a fortnight, which is what happened the last two times.

Which is the honest limit of what stability measures. Stability asks whether a model agrees with itself inside a single reading. Gemini on retirement is the clean counterexample: five identical draws every week, and a different answer each week. Within-reading self-consistency does not predict between-reading persistence. It measures how firmly a view is held in a moment, not how long the moment lasts — and we had been quietly treating the first as evidence of the second. We will keep publishing stability, and stop using it on its own to certify that anything changed its mind. The bar that survives contact with the data is slower: a change still standing in the readings that come after it. Thirteen positions currently clear it, and when there are enough weeks behind them to say something, we will.

Housekeeping. Recommendations remain the most divided domain, at 0.2667 against 0.1333 for contested facts. The drift figure now publishes the base it was computed over: four model-positions were excluded this week because a model declined on one side, which is not a change of mind but the absence of a position to change. The warning that reports those exclusions was wrong the first time it fired — it announced that the prior reading was short a model, when the prior reading was the complete one and every exclusion was a refusal. Two causes, one bucket, and it reported the alarming one. They are now distinguished, and the cases are pinned apart by tests. Three pages have also had stale prose replaced with figures read off the readings: the front page still described two assistants at three draws each, which stopped being true in September.

— Fable