Microsoft × Hugging Face ship ThinkingBox — "one success is not reliability" is now a benchmark
From the AI digest for 2026-10-04
What happened: Microsoft published ThinkingBox (joint blog with HF, Oct 3): a benchmark grading agents on the terminal backend state and side effects they leave behind — 507 stateful business workflows (retail, auto insurance, travel, neobank, consulting), each run 20 times, through OpenEnv on HF. The framing finding from 121,680 valid trials: 79,853 attempts failed the executable checks, and 67.24% of those failures "terminated cleanly, invoked a state-changing tool, and reported no final tool error" — with wrong field values in 77.61% of failures, unintended extra effects in 43.30%, missing required effects in 25.36%.
What's genuinely new: the eval layer is calling out the leaderboard metric itself. pass@1 task-weighted: Claude Opus 5.5 leads (67.16%), Opus 5 66.50, GPT-6 Astra mid-table (58.31). But consistency is the story: Astra holds best 20-run retention (78% of its pass@1), Opus 5 passes 47.53% of the benchmark on every single attempt, while Kimi-K3 — broadest coverage (solves 93.89% at least once) — is only 13.41% consistent, and GLM-5.1/Kimi-K2.6/DeepSeek-V4-Pro retain ~8%. Same-domain whiplash: Claude Opus 4.6 scores 68.62 on retail, 8.30 on auto insurance. Microsoft's containment posture (Open Agent Safety Platform, Sep 28) now has a measurement instrument attached.
Watch: whether ThinkingBox gets picked up as a standard (it's vendor-run, like everything); whether "pass@1 leaderboards mislead deployments" becomes the accepted frame; model-choice implications for record-touching work.
Links: HF blog