Cisco runs the decision-model trial: the trained classifier wins, Jev is the zero-shot pick — and Laya has a home
From the AI digest for 2026-10-05
What happened: Cisco's blog (Oct 5): a Cisco team pitted decision models against a purpose-built classifier on content-safety tasks — Jev (TypeSafe) and Laya as zero-shot question-answerers vs a 1B-parameter encoder fine-tuned on the Cisco AI Security Framework taxonomy (24 harm categories, six datasets, 0.5% false-positive budget). Result: the fine-tuned model wins — best binary AUC on five of six datasets, best recall at the FP budget on all six (Cisco multi-turn, 18k conversations: 61% vs Jev 34%, Laya almost none). Jev was "clearly the stronger" zero-shot option; Laya had "no useful operating point within budget on any dataset."
What's genuinely new: the decision-models thread's first third-party, domain-specific eval — and it resolves a flagged unknown: Laya is open-weight, from ConvAI Innovations (the mystery entrant in Cloudflare's Oct 3 table). Caveats kept: Cisco's model was trained on the eval's own taxonomy and the reference labels are LLM-generated — this is a domain benchmark, not the independent runner. But the message lands: decision models are a labeling-free tier below trained classifiers, not replacements — which reframes the category's pitch from "the end of classification" to "classification without labels."
Watch: whether TypeSafe or Cloudflare answer with their own tables; the workloads where zero-shot decision models actually win (the real question the thread has been asking since Sep 21).
Links: Cisco blog