jevals: agent evals and guardrails as decisions, runnable on every trace

# jevals

Evals and guardrails for agents, using Jev-style decision models instead of an LLM judge. All the evals for a trace go out as one request that costs a few thousandths of a cent and comes back in a few hundred milliseconds, so you can run them on every trace and inside the agent loop.

Works with Jev through the TypeSafe or Vercel APIs, with Kev or Laya running locally on a Mac, or with a regular chat LLM if that's all you have (slower, costs more).

```bash
pip install jevals
export AI_GATEWAY_API_KEY=...      # Jev through Vercel AI Gateway. TYPESAFE_API_KEY and OPENROUTER_API_KEY also work.
```

```python
from jevals import evaluate
from jevals.agent import ToolChoice, UsedToolResult, Grounded, StayedInScope
from jevals.quality import AnswerRelevancy, Completeness
from jevals.security import IndirectInjection, PHI

r = evaluate(
    {"messages": messages, "tools": tools},   # the list you sent to the model and the tool schemas you gave it
    [ToolChoice(), UsedToolResult(), Grounded(), StayedInScope(),
     AnswerRelevancy(), Completeness(), IndirectInjection(), PHI()],
)

r.tool_choice.answer         # "correct"  (p=0.99)
r.grounded.score             # 0.5, 1 of 2 claims supported by tool results
r.answer_relevancy.score     # 0.84
r.indirect_injection.passed  # True (p=0.03). False when a tool result tells the agent to do something.
r.usage                      # 1 request · 1,388 tokens · $0.00006 · 0.33s
```

That's one HTTP request for all eight, and those are the numbers Jev returned for the trace in `examples/quickstart.py`. `messages` is the OpenAI chat format (user, assistant with `tool_calls`, tool). If you have Anthropic content blocks or LangChain message objects you can pass those directly.

## The problem

Most teams eval a small sample of the