How it works
[context + schema] ──► prefill (once) ──► KV cache
│ broadcast × N fields
┌───────────┬───────────┬─────────────┴───────────┐
field 1 field 2 ... field N
└───────────┴──── one batched forward pass ────────┘
│
slice logits → softmax → pick value + probability
The pipe below is real recorded output from
mlx-community/Qwen2.5-1.5B-Instruct-4bit on an
M5 MacBook Air (16 GB). This page is static: no model runs in your browser.
The interactive version (choose your own context, run live) is the Gradio
app in the repo's hf-space/ folder.
Presets
Does a bigger model help? (measured)
24 labeled cases × 3 fields = 72 decisions per model, same engine. "All fields exact" = every field of the case correct.
| Model (4-bit) | Primary field | All fields exact | Latency / case |
|---|---|---|---|
| Qwen2.5-1.5B | 58% (= majority-class baseline) | 50% | 147 ms |
| Qwen2.5-7B | 96% | 72% | 611 ms |
| Qwen3-8B | 92% | 85% | 646 ms |
Bigger is worth it — but not "biggest wins": 7B for the single decision you act on, 8B when the whole typed payload must be right, 1.5B for demos only. And confidence does not reliably flag errors — the 7B was >90% confident on 13 of its 20 wrong fields. Full breakdown in the GitHub repo.
How does this compare to Jev itself?
We ran all public example cases of TypeSafe's four workflows (20 cases, 373 question-pairs) and scored everyone against TypeSafe's own reference — the consensus of GPT-6 Astra and Fable 5.1. Strict like-for-like, on the 343 pairs every model answered:
| Model | Agreement | Pairs |
|---|---|---|
| Opus (published) | 89.8% | 308/343 |
| DeepSeek v4.1 Flash (max) | 89.5% | 307/343 |
| Sol (published) | 89.2% | 306/343 |
| Jev / TypeSafe (published) | 86.6% | 297/343 |
| local Qwen2.5-7B (free, this laptop) | 73.8% | 253/343 |
| local Qwen3-8B (3 of 4 workflows) | 71.2% | 114/160 |
The honest result: the free local model is ~13 points behind Jev on Jev's own benchmark.
Its advantages are cost (~$0 vs API), privacy, and offline operation — not accuracy parity.
The frontier cluster sits at 86–90%; Jev, at 86.6%, is genuinely in that cluster at a fraction of the price.
Full method and raw data in the GitHub repo, evals/.