The at-home Darija showdown: Jev vs self-hosted Laya on Moroccan Arabic sentiment

# Darija Sentiment Benchmark

### Jev vs Kev-9B vs Laya on Moroccan Arabic reviews

Can a self-hosted model match a hosted AI service on informal Moroccan Darija? I built an end-to-end, zero-shot benchmark using [Darija Reviews](https://huggingface.co/datasets/ohidaoui/darija-reviews) to answer that question with the **same reviews, labels, and three-class task** for every model. It evaluates positive, neutral, and negative sentiment in both Arabic script and **Arabizi** (Darija written in Latin characters and numerals), then turns the saved predictions into an interactive HTML report.

**Python 3.11+ · TypeSafe Jev · Modal GPUs · traceable evaluation · statistical comparison**

## Results

The frozen holdout contains **171 reviews**: 127 Arabic-script and 44 Arabizi. All three models returned predictions for all 171 reviews in the saved comparison run of **22 September 2026**.

| Model | Accuracy (95% interval) | Macro F1 | Arabic | Arabizi |
| :--- | ---: | ---: | ---: | ---: |
| **Jev** · TypeSafe API, `jev-1.13.0` | **79.5%** (72.9–84.9) | **71.7%** | **84.3%** | **65.9%** |
| **Kev-9B** · Qwen3.5-9B on Modal | 71.9% (64.8–78.1) | 61.1% | 78.0% | 54.5% |
| **Laya** · multilingual checkpoint on Modal | 53.8% (46.3–61.1) | 50.0% | 60.6% | 34.1% |
| Always predict positive · descriptive reference | 53.8% | — | — | — |

Jev led Kev by **7.6 percentage points** on these paired reviews (exploratory 95% paired bootstrap interval: **2.3–12.9 points**). Laya matched the always-positive reference on overall accuracy. Every model scored lower on the Arabizi subset, and neutral sentiment was the weakest class.

These are agreement scores against the dataset's original annotations, **not** independently adjudicated labels or estimates of production performance. The style groups