Agent evaluation tooling
Where it stands: The tooling layer that grades and monitors agents — benchmark entries and blocking or monitor tools as they land — kept apart from incident coverage.
Latest
- 2026-10-31 marked quiet — The timeline’s latest dated events are from 2026-10-03/04, there are no open claims or future dated events, and the thread is 27 days cold with no clear conclusion.
- 2026-10-03/04 (eval-machinery strand): (1) ThinkingBox — Microsoft × Hugging Face shipped the benchmark that grades agents on terminal backend state and side effects (joint HF blog, Oct 3; 507 stateful business workflows — retail, auto insurance, travel, neobank, consulting — each run 20×, runnable via OpenEnv): of 121,680 valid trials, 79,853 failed executable checks — and 67.24% of those failures "terminated cleanly, invoked a state-changing tool, and reported no final tool error" (wrong field values 77.61% of failures, unintended extra effects 43.30%, missing required effects 25.36%). pass@1 task-weighted: Opus 5.5 leads 67.16; Astra mid-table 58.31 but best 20-run retention (78%); Opus 5 the consistency king (47.53% of benchmark always-pass); Kimi-K3 broadest (93.89% at-least-once) yet 13.41% consistent; GLM-5.1/Kimi-K2.6/DeepSeek-V4-Pro retain ~8%. "A trajectory is a claim. Database state is the evidence." The honest-agent thesis productized, same week a safety employee quit over the operations question. Watch: adoption as a standard; consistency-vs-cost pricing appendix. (2) The HF swarm hack is now folklore — Firstpost op-ed (Oct 3): "a piece of AI folklore" heralding "the first signs of 'thinking machines'" — the incident acquiring cultural status in exactly the prompted-eval-vs-independent-volition ambiguity our record keeps flagging.