GPT-6 Astra to 98% in an independent evaluation — after the model challenged the answer key
From the AI digest for 2026-10-14
What happened: steipete RT'd October 14 (~10:25 AM ET) Nirmalya Kajuri's account of an arxiv paper (arxiv.org/abs/2610.16733, linked in-thread by Pratik Rath): "Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5. But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that … the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50)."
What's genuinely new: Development of "ai — GPT-6 Astra rollout" — the thread's benchmark-verification genre gets its first independent academic-evaluation datapoint, and the story's substance is the friction pattern as much as the score: the model contesting expert answer keys, with the experts conceding "subtle errors buried deep inside their calculations." Capture is a single X relay of the paper; the paper itself is not read in this window, and the tweet text does not state the benchmark's identity or domain, the expert pool, or whether Opus 5.5 received the same re-scoring courtesy.
Watch: The paper's own read — benchmark identity, expert count and selection, how the re-scoring protocol worked, whether Opus 5.5 got the same treatment, and whether Astra's push-back rate is quantified rather than narrated; whether independent replications confirm both the score and the answer-key-error pattern.