madewithlaya

Catalog / Documents & data

0104GitHub

zero-shot-ie-bench: GLiNER 2.5, GLiFormer, GLiClass, Laya, Von and Jev in one web UI

Seven zero-shot extraction and classification families demoed, benchmarked and cross-compared, with sizes, licences and costs in one table.

Open source ↗ github.comcostfree for the local rowstime-
umstek/zero-shot-ie-benchREADME ↗
# zero-shot-ie-bench

Zero-shot information-extraction and classification benchmark: a curated
catalog of decision, classification, NER and reranking systems — hosted
APIs and local runner islands — benched against versioned, multilingual
suites, with an append-only run history and generated result tables.

- **Catalog** (`catalog/`): one YAML per system; providers, tracks,
  languages, machines. Schema-validated (`bun run validate`).
- **Suites** (`suites/`): versioned parallel-item pools with content
  hashes. Canary suites are the cheap scheduled nerf-detector.
- **Runs** (`runs/`): append-only; every table below is generated from
  them (`bun run lake && bun run render`). Rebenching adds history; a
  nerf or improvement shows up as drift.
- **Bench** (`bench/`): TypeScript + Bun — provider adapters (one per
  API contract), scorers per track, `run | canary | doctor`.
- **Runners** (`runners/`): one independent Python/uv project per
  dependency island, all speaking one HTTP protocol; a machine that
  benches needs Bun next to uv, nothing else.
- **Board / Playground** (`apps/`): static decision board and a
  try-it UI for hosted + live-local models.

Quickstart: `bun install` · `bun run check` (typecheck + tests + lint) ·
`bun run render` (lake + README tables) · `bun run --cwd bench dev
doctor`. Deeper docs: [methodology](docs/methodology.md) (scoring,
deferred runs, cost model) · [adding a model](docs/adding-a-model.md)
(the 3-step recipe) · [the restructure plan](docs/plan/00-index.md).

The v0 era (2026-09-20 → 2026-10-24, 81 systems / 38 families) is
preserved verbatim at tag `v0-final`; history since lives in
[CHANGELOG.md](CHANGELOG.md) and the run records themselves.

## Mixed-pool spectrum benchmark (all 78 systems)

Every system answers the same 48 mixed c

Also filed under Documents & data

  1. 0103

    Zero-shot benchmark: Laya's advertised win over Jev does not reproduce on four datasets

    Frozen protocol, raw predictions, calibration and latency for Jev 1.13, PrismNLI-0.4B and Laya. Laya ties Jev on dair-ai/emotion and trails by 10–33 points elsewhere.

    @elcronos · Documents & data

  2. 0102

    jevbench: Jev vs GPT-5-mini, Claude Sonnet 5, fine-tuned DistilBERT, BART NLI and Laya

    Six classifiers over SST-2, AG News and more, reporting accuracy, macro-F1, ECE, p50/p95 latency, throughput and cost side by side. Laya runs locally on MPS or CPU.

    @dhruvmehra · Documents & data · $0 for the local rows

  3. 0098

    Agentic GraphRAG over S&P 10-Ks with Laya as a shadow decision engine beside Jev

    Anthony Alcaraz's talk demo routes typed decisions through Jev and shows Laya's answers beside them. On the TSMC run Laya rejected every true claim; that's the point.

    @AnthonyAlcaraz · Documents & data · free (Laya shadow); Bedrock for reasoning

  4. 0087

    chunklaya: why Laya stops reading after ~200 tokens, and a chunking harness that fixes it

    Measures the decision head's attention drop-off on long inputs across 400 documents, then chunks, screens and aggregates, or locates the one passage that matters.

    @myxamediyar · Documents & data · free, no extra cost over a larger context window