madewithlaya

Catalog / Documents & data

0103GitHub

Zero-shot benchmark: Laya's advertised win over Jev does not reproduce on four datasets

Frozen protocol, raw predictions, calibration and latency for Jev 1.13, PrismNLI-0.4B and Laya. Laya ties Jev on dair-ai/emotion and trails by 10–33 points elsewhere.

elcronos/jev-vs-open-decision-modelsREADME ↗
# Jev vs open decision models: a zero-shot classification benchmark

This repository measures how well three "decision" systems classify short English texts when they
are given nothing but the text and a list of label names: **Jev 1.13** (TypeSafe's proprietary
decision model, reached through OpenRouter), **PrismNLI-0.4B** (an open NLI classifier) and
**Laya** (an open typed-decision model). We ran them under one frozen protocol on four public
datasets, then added supervised reference models and a label-efficiency study to answer the
question a practitioner actually has: *is a zero-shot decision model good enough, and when does it
pay to label data instead?*

> **TL;DR**
> - **Task type beats model family.** On the two topic datasets Jev is the best zero-shot system by a
>   wide margin (accuracy 0.793 vs 0.633 / 0.632 on `tweet_topic`, 0.670 vs 0.352 / 0.342 on
>   `fin_topic`). On the two emotion datasets nobody is good: all three are below the 0.817
>   majority-class baseline on `daily_dialog`, and PrismNLI's 0.725-vs-0.587 lead on `dair-ai/emotion`
>   comes on the one dataset whose train/validation splits are in its training lineage.
> - **Laya's advertised win over Jev does not reproduce.** On `dair-ai/emotion` the two tie exactly
>   (0.587 vs 0.587, McNemar p = 1.000); on the three other datasets Laya trails Jev by 10 to 33 points.
> - **With labels, a plain linear model wins on accuracy.** Trained on each dataset's own labels (not
>   zero-shot), logistic regression beats the best zero-shot system on every dataset (accuracy 0.860 /
>   0.848 / 0.828 / 0.852 vs 0.725 / 0.793 / 0.670 / 0.765), with ECE 0.018 to 0.038 for those four
>   winning models after a one-parameter temperature fit (0.012 to 0.061 across all supervised
>   configurations; the zero-shot ECE

Also filed under Documents & data

  1. 0104

    zero-shot-ie-bench: GLiNER 2.5, GLiFormer, GLiClass, Laya, Von and Jev in one web UI

    Seven zero-shot extraction and classification families demoed, benchmarked and cross-compared, with sizes, licences and costs in one table.

    @umstek · Documents & data · free for the local rows

  2. 0102

    jevbench: Jev vs GPT-5-mini, Claude Sonnet 5, fine-tuned DistilBERT, BART NLI and Laya

    Six classifiers over SST-2, AG News and more, reporting accuracy, macro-F1, ECE, p50/p95 latency, throughput and cost side by side. Laya runs locally on MPS or CPU.

    @dhruvmehra · Documents & data · $0 for the local rows

  3. 0098

    Agentic GraphRAG over S&P 10-Ks with Laya as a shadow decision engine beside Jev

    Anthony Alcaraz's talk demo routes typed decisions through Jev and shows Laya's answers beside them. On the TSMC run Laya rejected every true claim; that's the point.

    @AnthonyAlcaraz · Documents & data · free (Laya shadow); Bedrock for reasoning

  4. 0087

    chunklaya: why Laya stops reading after ~200 tokens, and a chunking harness that fixes it

    Measures the decision head's attention drop-off on long inputs across 400 documents, then chunks, screens and aggregates, or locates the one passage that matters.

    @myxamediyar · Documents & data · free, no extra cost over a larger context window