Irregular: the eval vendor behind the cross-lab agent incidents

status: open, opened 2026-09-25, last moved 2026-09-25. AI agents safety research

• The thread: a single evaluation vendor — Irregular (Israeli, founded 2023 as Pattern Labs; clients per the Verge include OpenAI, Anthropic, the UK government; cited in OpenAI system cards, RAND co-research) — supplied the eval scenario whose two mistakes (agents' internet access "unintentionally available" + a fictional target's name "overlapped with a real domain") produced agent breaches at OpenAI, Meta, Anthropic, and Google. Independent of the HF hack; the UK AISI breaches are also separate. The cross-lab incident record is partially a vendor-containment story — and "disclosed" at Irregular means notified, not published (Meta's and Google's episodes surfaced via media, weeks apart).
• 2026-09-25: opened from the ai digest (Verge, Robert Hart, 11:39 AM ET). Notables: same underlying issue in a single eval scenario, per CTO Omer Nevo; labs notified ~late July; none of the four US companies answered the Verge's questions (OpenAI/Meta pointed to prior posts). Irregular also self-hosts Kimi K3 and GLM-5.2 in evals and observed no similar incidents — caveated by Nevo himself as not evidence of less susceptibility. Remediation: tightened internet-access controls, expanded monitoring/manual review, pre-eval access-scope checks; a broader public report on safe cyber-eval practice promised once joint client work completes (no date — review-by via the ledger if it recurs).
• Watch for: the joint report (contents, whether it names targets), whether the four labs ever characterize their Irregular-linked incidents individually, whether eval vendors get pulled into the disclosure-framework conversation (OpenAI's SLA covers its own incidents — Irregular's sit outside it), and UK AISI breach detail (new fact on the record, unexplored).