Does a typed-judgment model rerank Finnish legal passages better than a cross-encoder?

# jev-rerank-bench

Does reranking with a **typed-judgment model** beat a dedicated cross-encoder?

The picture is the problem. A corpus of Finnish collective agreements contains
hundreds of documents that repeat each other section for section, so
"32 § Ikääntyneiden työntekijöiden työajan lyhentäminen" exists, nearly word for
word, in dozens of them. The passage text does not identify the right answer —
the document it belongs to does. A bi-encoder embedding cannot see that
distinction, because it never sees the query and the passage at the same time.

That is what a reranker is for. This repository measures whether
[TypeSafe Jev](https://docs.typesafe.ai) — which returns a calibrated
probability instead of generated text — does it better than the alternatives, on
two Finnish legal corpora, over the same shortlists, with the same question.

## What it measures

| | |
| --- | --- |
| **First stage** | Postgres FTS (`ts_rank_cd`, `finnish`), pgvector HNSW, and RRF of both |
| **Rerankers** | Jev (noul / score / batched / cascade), Voyage `rerank-2.5`, `gpt-4.1-mini` via logprobs, `gpt-5.6-luna` graded, [Laya](https://github.com/NandhaKishorM/laya) open weights |
| **Metrics** | Recall@1/3/5/10, MRR@10, nDCG@10, shortlist ceiling, latency, \$/1000 queries, abstention curve |
| **Corpora** | `mupler` — [MuPLeR-fi](https://huggingface.co/datasets/mteb/MuPLeR-retrieval), 10 000 EU legal passages, 200 queries, EUPL-1.2, reproducible by anyone. `fi-tes` — 160 Finnish collective agreements and statutes, 37 440 chunks, 84 verified questions with section-level ground truth; not redistributable, so the code path is here and the documents are not. |

Every reranker receives the **identical shortlist**, and every model is asked the
**identical question** —