The at-home Darija showdown: Jev vs self-hosted Laya on Moroccan Arabic sentiment

# Jev vs Laya: The At-Home Darija Showdown

Can an open-weight model you control match a hosted AI service? This reproducible
zero-shot benchmark pits TypeSafe Jev against self-hosted Laya on informal Moroccan
Darija sentiment—across Arabic script and Arabizi, with `positive`, `neutral`, and
`negative` labels. No model is trained or fine-tuned.

## Setup

```bash
uv sync
cp .env.example .env
# Put the keys in .env, or export them in the shell:
export TYPESAFE_API_KEY="..."
export HF_TOKEN="..."  # authenticates dataset and Laya model downloads
export LAYA_ENDPOINT_URL="https://...modal.run"
```

The CLI loads these variables from `.env`. The checked-in `.env.example` points to
the deployed Laya endpoint but contains no credentials.

The benchmark loads the `test` split of
[`ohidaoui/darija-reviews`](https://huggingface.co/datasets/ohidaoui/darija-reviews).
The source dataset has no row ID, so source row indices are used as stable IDs.
The first data load freezes a deterministic seed-42 split under `data/splits/`:
80% dev and 20% eval, stratified by normalized sentiment and writing style when
possible.

The dataset currently contains four raw labels. The four rows labeled `negative `
with a trailing space are explicitly normalized to `negative`. Any other unknown,
blank, or null label stops the run.

## Deploy Laya on Modal

The deployment uses the repository's multilingual checkpoint directly for both
Arabic-script Darija and Arabizi. Keeping one checkpoint makes the writing-style
comparison meaningful. The weights and source revision are pinned, and the service
runs on one L4 with scale-to-zero behavior.

Create a Modal secret named `hf-secret` containing `HF_TOKEN`, then run:

```bash
uv run modal run -m darija_eval.modal_laya::download_model
uv run modal deploy -m da