Zero-shot benchmark: Laya's advertised win over Jev does not reproduce on four datasets
Frozen protocol, raw predictions, calibration and latency for Jev 1.13, PrismNLI-0.4B and Laya. Laya ties Jev on dair-ai/emotion and trails by 10–33 points elsewhere.
# Jev vs open decision models: a zero-shot classification benchmark This repository measures how well three "decision" systems classify short English texts when they are given nothing but the text and a list of label names: **Jev 1.13** (TypeSafe's proprietary decision model, reached through OpenRouter), **PrismNLI-0.4B** (an open NLI classifier) and **Laya** (an open typed-decision model). We ran them under one frozen protocol on four public datasets, then added supervised reference models and a label-efficiency study to answer the question a practitioner actually has: *is a zero-shot decision model good enough, and when does it pay to label data instead?* > **TL;DR** > - **Task type beats model family.** On the two topic datasets Jev is the best zero-shot system by a > wide margin (accuracy 0.793 vs 0.633 / 0.632 on `tweet_topic`, 0.670 vs 0.352 / 0.342 on > `fin_topic`). On the two emotion datasets nobody is good: all three are below the 0.817 > majority-class baseline on `daily_dialog`, and PrismNLI's 0.725-vs-0.587 lead on `dair-ai/emotion` > comes on the one dataset whose train/validation splits are in its training lineage. > - **Laya's advertised win over Jev does not reproduce.** On `dair-ai/emotion` the two tie exactly > (0.587 vs 0.587, McNemar p = 1.000); on the three other datasets Laya trails Jev by 10 to 33 points. > - **With labels, a plain linear model wins on accuracy.** Trained on each dataset's own labels (not > zero-shot), logistic regression beats the best zero-shot system on every dataset (accuracy 0.860 / > 0.848 / 0.828 / 0.852 vs 0.725 / 0.793 / 0.670 / 0.765), with ECE 0.018 to 0.038 for those four > winning models after a one-parameter temperature fit (0.012 to 0.061 across all supervised > configurations; the zero-shot ECE