The Yardstick Moved: Private Tests and the Open-Weight Mirage

On Monday, Business Insider crystallized the week’s quietest bombshell: Artificial Analysis’s Intelligence Index v5 — the AI industry’s most-watched capability ranking — moved 60% of its score onto private test sets and held-out answers, and the best open-weight models fell from three points behind the leader to twelve.1 No weights changed. No model got worse. The yardstick moved, and a geopolitical narrative — Chinese open models closing on Anthropic and OpenAI, threatening their premium pricing — lost nine points in an afternoon, weeks before Anthropic’s IPO. Snowflake CEO Sridhar Ramaswamy put the question the episode deserves: “The day benchmarks went private, the open-model gap went from 3 points to 12. Were open models catching up, or was the test leaking?”2 Run it through four frames.

1. First Principles: A Known Test Is a Specification

Strip a benchmark to fundamentals and it is a proxy: a sample of tasks claimed to correlate with a capability. A proxy carries two validity conditions — the sample must resemble the target distribution, and it must be independent of the process being optimized. Web-scale training dissolved the second condition years ago: every public test set is plausibly in the scrape, and a lab that iterates against published leaderboards is doing gradient descent on the exam. A test the measured party can read is not a measurement; it is a specification, and models meet specifications. v5 made no model better or worse — it restored independence for the majority of the score, which rose from 45% private in v4.3 to 60%.34 The nine-point widening is the sound of an instrument recalibrating: some of the old “parity” was capability, and some of it was answers.

2. Analogy Transfer: The Exam Leak Has Precedent

The structural form — an assessment whose items are known to the assessed, whose results steer billions — has been solved elsewhere. Schools know what an exam leak does to a grade: it measures cramming, not learning. The insurance crash-test institutes learned it when automakers engineered for the test wedge rather than the crash, and responded by rotating protocols faster than the industry could adapt. Anti-doping’s answer to athletes who studied testing schedules was unannounced collection and the biological passport. The shared mechanism: keep part of the assessment hidden and keep rotating what is hidden, because any fixed instrument becomes a training target. The disanalogy matters in both directions. An athlete does not retrain on the test, but a lab does — which is why private sets bite harder here than in sport. And a private benchmark cannot be audited by the people it ranks; the lab answers to due process, Artificial Analysis answers to itself.

3. Nietzsche Ladder: Who Wrote the Tablets

Camel. Carry the inherited account honestly. Artificial Analysis stated its reason plainly — reflect how people actually use the technology, kill the contamination — and even Nathan Lambert, an open-weight proponent, concedes the old index was “flattering the open weight models too strongly.”5 The celebrated gap-narrowing deserved scrutiny, and it got it.

Lion. Yes, those are the old tablets; now ask who wrote them. One private firm owns the industry’s headline number, 60% of it invisible to the firms it ranks, and the revision lands mid-IPO-season with exactly the story incumbents needed: the China threat overstated, Anthropic’s pricing power defended.1 The same week, a gated model with no public API took the top spot at 59.6 No conspiracy is required — only incentives, which converge. And see what the Lion sees in the old leak: open labs trained to public tests because public tests were the only proof channel they had. The hacking was rational marketing under constraint.

Child. Your No was necessary, but negation is not yet a measurement regime. What can be built in the cleared space: private composites for headline claims, public held-out suites anyone can reproduce, and usage-weighted evidence as the third leg — the OpenRouter analysis suggesting DeepSeek’s entire daily demand fits on a single $7 million rack of GPUs.1 Plural instruments, no single number, measurement as ongoing practice rather than a scoreboard.

4. Question Forge: The Better Question

Ramaswamy’s question is good but binary — it presumes one culprit, catching up or leaking, and it asks about the models when the real subject is our instruments. Forge it: if the only tests labs cannot study for are tests no one else can check, what exactly are we buying when we buy a benchmark score? This one refuses a news-cycle answer. It costs the purchaser the comfort of a single number and puts the auditability burden where it belongs — on the referee, not the ranked.

What the Frames Converge On

Four frames, one coordinate: nothing about any model changed this week; the instrument changed, and the change rewrote the geopolitics and the investment narrative overnight. First principles say the old gap was partly answers; the analogies say hidden-and-rotating assessments are the only kind that survive optimization; the ladder says a monopoly referee is a risk whether or not it is right this time; the forged question says the scoreboard now owes us an account of itself. The tell to watch: whether Artificial Analysis publishes third-party audit rights and a rotation cadence for its private sets — and whether the 12-point gap closes within months as private items seep into the next scrape, at which point the industry learns whether it fixed measurement or merely moved it indoors.


  1. https://www.businessinsider.com/chinese-ai-models-rankings-artificial-analysis-private-tests-anthropic-openai-2026-11 ↩ ↩2 ↩3

  2. https://x.com/RamaswmySridhar/status/2115421134929981739 ↩

  3. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v5 ↩

  4. https://winbuzzer.com/2026/10/28/artificial-analysis-reweights-ai-rankings-toward-private-tests-xcxwbn ↩

  5. https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers ↩

  6. https://venturebeat.com/technology/google-unveils-gemini-4-carbon-retaking-the-worlds-no-1-spot-on-updated-harder-benchmarks-but-access-is-limited ↩