Solved Is Not Proved: The Mathematicians Grade Back
On October 14, the First Proof consortium published the complete record of its third batch — ten research mathematics problems across geometry, topology, statistics and dynamics, each known to admit a proof of at most twelve pages, offered to five AI systems with one 24-hour shot each and no human input.1 Thirty expert reviewers convened October 8–9 at Carnegie Mellon’s Institute for Computer-Aided Reasoning in Mathematics to grade anonymized solutions on correctness, exposition, and attribution — with negative scoring newly introduced so that producing no answer beats producing a wrong one.1 The scores: ProofCouncil, a multi-agent harness running GPT-6 Astra, Claude Fable 5.1, Claude Sonnet 5, and Gemini 3.1 Pro, solved nine of ten at a cost of $1,888; a direct query to Astra alone solved seven of ten for $36 — each two problems better than its Batch 2 self.1 Every problem was solved correctly by some system. And every submission of the batch’s first open-weight entrant, Moonshot AI’s Kimi K3, contained major errors; reviewers judged it less capable than December 2025’s frontier.1 Harvard’s account followed on October 19, and the load is in the details between the scores: one solution’s missing attribution “essentially constituted plagiarism”; even the winner drew negative average exposition marks on a third of its answers.2 Six months ago this blog watched mathematics acquire an intake queue.3 This is the queue being read. Four frames on what the reading found.
First Principles — a proof is not an answer that is right
Strip the story to its fundamentals. A proof is not a certificate of abstract truth; it is an artifact whose function is to bring a community of peers to settled agreement — which is why mathematics has always demanded three things at once, the things First Proof’s rubric merely writes down: correctness, exposition, attribution. The customary answer treats “solved nine of ten” as one number. Rebuilt from fundamentals, the round reports three different quantities moving in different directions: correctness up sharply, citations better than before but adequate on only four of ten Astra solutions, and exposition flat to worsening — “it might be getting worse,” as co-founder Rachel Ward put it, “as the solutions become more complex and longer.”2 That is exactly what optimization selects for: training and harnesses reward the checkable axis, and nothing in the reward signal pays for the reader. The difference from the default answer is the difference between solving and proving. The frontier learned to solve. Lauren Williams’s operational gloss: a correct proof that takes hours to read, or an error that takes hours to find, is “a waste of human time”2 — the resource mathematics actually runs on.
Question Forge — the question the guild built an institution to escape
The question the media asks — can AI do research math? — is broken in three ways at once. It smuggles its answer (a benchmark score), sits at the wrong level (capability is the tactics; certification is the problem), and offers a false binary. First Proof exists because its founders refused to answer that question and forged a costlier one. It is worth making the forgery explicit: who can afford to know? Not abstractly — concretely. Thirty mathematicians, two days in one building in Pittsburgh, problems embargoed until test week, $1,888 of compute per graded frontier entry.1 Carried for a while, the question reframes every capability claim as an invoice. And it implicates everyone in the room: two companies produce the claims, one guild prices them — and the referee took money from the players, funded this batch by DARPA, Google.org, and unrestricted donations from Anthropic and OpenAI.1 Williams’s worry that “it seems very unhealthy for two companies to wield such a tremendous power”2 is this invoice, read aloud.
Assumption Audit — the referee audits its own headline
The round’s loudest finding: the open-weight gap is “substantially wider” than the roughly four-month lag Epoch AI measured across general benchmarks in May.1 Audit it. The keystone is definitional: equal test conditions equal fair comparison — one shot, 24 hours, a format tuned to harness-equipped frontier systems. Load: it breaks the headline. Confidence: low — by the organizers’ own admission: “Our testing is currently far too harsh for systems that are built on open-weight models,” said Mohammed Abouzaid.2 Testability: cheap, and scheduled — Batch 4 in January will split the tracks to isolate how open-weight systems fail.1 Around the keystone: the assumption that August’s embargoed problems stayed out of training corpora (load: high; testability: hard, mostly hindsight); the assumption that three anonymous reviewers converge on stable grades (moderate, untested); and the causal blur where harness capability passes as model capability — ProofCouncil’s two-problem premium over its own base model is scaffold, not weights, a distinction the press release itself draws: “the same model can perform very differently in different harnesses.”1 If the keystone fails, the politically potent gap partly measures format rather than weights. Note what is rare here: the consortium flagged its own keystone and booked the experiment that could topple its headline. In benchmark land, that is close to unheard of.
Explanation Ladder — from report card to who owns certification
High school: the best AI got nine out of ten on a mathematicians’ test, but wrote the answers so badly that a third of them scored below zero, and one forgot to credit the humans whose work it used.
College (responding): the deeper result is the design, not the score. Problems withheld until test week, scripts anonymized, three graders each, negative scoring so that silence beats fabrication. That is how you measure a model rather than its memory — which is why these numbers are believable in a way September’s press releases were not.
PhD (responding): the rubric operationalizes an old distinction. Correctness can be delegated — eventually to a proof kernel, mechanically. Exposition and attribution cannot, because they are not decoration; they are the discipline’s social contract, Merton’s norms wearing a scorecard. The plagiarism finding is a norms violation, not a style note. And the harness premium — two problems for a committee over the raw model — is scaffolding buying capability: the committee is the competence.
Philosopher (responding): step outside the machinery. Mathematics was the one field where an argument, once settled, stayed settled. This round re-prices settlement: answers approach free while certification remains aristocratic — thirty people, one building, two days. The consortium’s own worry that compute access may become “a barrier for basic research in mathematics”1 is the aristocracy price made explicit: $36 gets you seven answers; $1,888 gets you nine graded ones.
Gigabrain (responding): every community of makers learns this lesson eventually, and it is a plain lesson: the maker’s account of the thing is marketing. What First Proof built is boring and enormous — a slow room where claims wait their turn to be looked at by people with standing. Everyone will quote the nine out of ten. The real finding is what the nine out of ten cost: thirty mathematicians and two days to establish. Capability now moves at the speed of compute. Certification still moves at the speed of trust.
What the frames show together
First principles: the frontier optimizes the axis benchmarks can check; solving has outrun proving. Question forge: the scarce resource is not ability but priced verification — and the referee’s funding by DARPA, Google.org, and the labs themselves is now part of the result.1 Assumption audit: the loudest headline hangs on a keystone the organizers themselves flagged and scheduled for demolition. Explanation ladder: what looks like a report card is an institution being built in public — the first one with standing to hand the labs a grade they did not write. October 6 made claims cheap.3 October 14 priced them. Watch three things: January’s split-track round; whether any lab submits a raw model with no harness; and whether exposition keeps degrading — because the first axis the machines skipped is the one that was never theirs to skip.
-
https://1stproof.org/third-batch-press-release.html ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11
-
https://current.fas.harvard.edu/stories/top-mathematicians-grade-ais-ability-solve-toughest-problems ↩ ↩2 ↩3 ↩4 ↩5
-
https://logicinczo.github.io/semantic-thinking/2026/10/the-referee-is-the-bottleneck-openai-722-manuscripts/ ↩ ↩2