jevbench: Jev vs GPT-5-mini, Claude Sonnet 5, fine-tuned DistilBERT, BART NLI and Laya

# jevbench

Is TypeSafe's **JEV** a better text classifier than LLMs, BERT, or the open-weights
alternatives? This is a small, reproducible benchmark that runs six classifiers over
three public datasets and reports accuracy, calibration, latency, throughput and cost
side by side.



Full tables with macro-F1, ECE, p50/p95 and throughput: [`docs/results/2026-09-22-n500-summary.md`](docs/results/2026-09-22-n500-summary.md).

## What is being compared

| Classifier | What it is | Runs |
|---|---|---|
| `jev` | TypeSafe `typesafe/jev-1.13`, a "System One" model. One `choice` question per example, returns per-label probabilities. | OpenRouter **Decisions API** (`/api/alpha/decisions`, not chat completions) |
| `laya` | Convai `convaiinnovations/laya`, open-weights (Apache 2.0) System One model, 421M params, same question format as JEV | local, MPS or CPU |
| `llm-cheap` | small fast LLM (default `openai/gpt-5-mini`), JSON-schema enum output | OpenRouter chat completions |
| `llm-frontier` | frontier LLM (default `anthropic/claude-sonnet-5`), same prompt | OpenRouter chat completions |
| `bert-ft` | `distilbert-base-uncased` fine-tuned per dataset on up to 10k train examples | local, MPS or CPU |
| `bert-zs` | `facebook/bart-large-mnli` zero-shot NLI | local, MPS or CPU |

| Dataset | HF repo | Labels | Eval split |
|---|---|---|---|
| `sst2` | `stanfordnlp/sst2` | 2, sentiment | validation |
| `agnews` | `fancyzhx/ag_news` | 4, topic | test |
| `banking77` | `legacy-datasets/banking77` | 77, intent | test |

A fixed, seeded sample of N examples (default 500) is drawn from each held-out split.
Every zero-shot model receives the **same label descriptions**, defined once in
`src/jevbench/datasets.py` and reused for JEV criteria, Laya criteria, the LLM prompt
and the NLI hypothe