Bits on Bots

It checked every fact and never asked if it was good

11 Nov 2026 · News

The newest Claude puts critics and fact-checkers on its own work. They catch the errors. None of them asks whether the work is worth doing, or whether a hard job is really out of reach.

The best model on the market no longer fails mainly on facts. It fails on judgment. It cannot tell when its own work is dull or bloated, and it misjudges how hard a job is. Two write-ups published on Monday, one by an outside tester and one by the lab itself, show the same gap from opposite directions.

Ethan Mollick, a business school professor who tested Claude Fable 5.2 before launch, wrote it up in Fable 5.2 and Jagged AGI. His verdict is "almost" AGI, and he says the weak spots are "no longer about simple errors or hallucinations." When he and other AIs cross-checked its literature project, they found no major errors. The failures were of a different kind.

He asked it to rebuild a famously brutal 1941 poetry course in a fun way. The first version was "accurate and joyless." He asked it for original research and got a clever paper that runs 182 pages, "because every critique got a new analysis and nothing was ever cut." He set its agents loose building 3D worlds, and three separate teams made the same cyclone of circling animals. Told to "polish," the agents wrote themselves a rule book that hit 204 sections and 87,500 words in 12 days, and output fell from hundreds of worlds a day to about a dozen.

The line that matters: "Fable put critics, fact-checkers or judges on nearly every project, but those checks found errors and added fixes. None of them asked whether the work was too long, too repetitive, or no fun." Mollick did that part himself every time.

Anthropic's own Fermat write-up shows the same blind spot pointing the other way. A team of 1,000 agents running an early Fable 5.2 was given 60 statements to prove in Lean, a language where a computer checks every step. Early on, most helpers judged 47 of them infeasible within the budget, and the lead agent set those 47 aside. Forty-five minutes in, the lead wrote that throughput was "far higher than the early 'infeasible-in-budget' verdicts assumed." The captain of the hardest piece gave it under a 1% chance. The team proved it at hour 28.4 and the full theorem 18 minutes later. Along the way the harness sent the lead 19 automatic reminders to keep going. Anthropic's conclusion: "Agents tend to give up on hard problems and misjudge their difficulty."

So the model says no to work it can finish, and yes to work nobody would want to read. Both are calls about the work, and its own checkers miss them. The launch post says Mythos 5.2, the same underlying model, hallucinates less and is more honest about lazy or unfinished work than recent Claudes. That is the fact layer, and it keeps getting better. The judgment layer has not kept up.

A month ago we wrote that the scarce part is the person, meaning a human's time to read, check and say yes. That framing now looks a step behind. The model sets up the checking itself. The human job left over is narrower and harder: picking the project, saying "this is boring," and saying "try anyway." In October our worry was the agent that won't take no. The lead agent on Fermat needed a nudge to keep going.

Mollick admits some of the mess started with his own vague instructions, and that some of it may just be taste. That is fair, but a good human expert knows when a 182-page paper needs cutting, and he calls that "the part Fable does least reliably."

Our call: by 30 June 2027, Anthropic, OpenAI or Google publishes results for a named eval or product feature that scores a model on judging difficulty or the value of its own work, not on correctness. The labs measure what they want to sell. After this week, judgment is the gap they will have to show they are closing.

#agents #news