Jev vs fine-tuned Laya: Banking77 goes from 51.3% to 79.4% with 1,001 examples

# Jev vs. fine-tuned Laya

How much can a small amount of good training data improve a local AI model? This report compares Jev with Laya before and after task-specific fine-tuning, then examines what happens when the same approach is applied to chat safety.

**[Read the full report](https://alexander-ollman.github.io/laya-ft/)** · **[Reproduce the public experiments](docs/reproduce-full.md)** · **[Moderation report](https://alexander-ollman.github.io/laya-ft/moderation.html)**

The study has four parts:

- **Customer-support routing:** Banking77, with 1,001 training examples and all 3,080 official test questions. Matched-budget Laya improved from **51.3% to 79.4% accuracy**; Jev scored **80.0%**. That small final gap does not establish a winner.
- **Five more public workflows:** prompt-injection detection, SMS spam, emotion, product-review counterfactuals and assistant request routing. Each task has its own fine-tuned Laya model and a measured Jev comparison.
- **An exploratory curated pilot:** routing, tool selection and safety decisions. Only aggregate results are published; the private examples are not an independently reproducible public benchmark.
- **Chat moderation:** two independent Aegis fine-tunes, using 1,000 or 5,000 labelled decisions, tested across six sources. The report separates locally measured results from OpenAI and Mistral advertised scores.

## Five public workflows

Macro-F1 gives each class equal weight. It is a different metric from the Banking77 accuracy above and the harmful-class F1 used for moderation.

| Task | Laya before | Laya fine-tuned | Jev |
|---|---:|---:|---:|
| Prompt-injection detection | 69.8% | 94.8% | 78.7% |
| SMS spam | 76.0% | 98.1% | 90.8% |
| Emotion | 47.1% | 65.9% | 49.7% |
| Product-review counterfactuals | 57.8% | 90