Benchmark · October 14, 2026
The same 80 citizen decisions, asked 3 times each.
Every engine got the identical citizen state and the identical set of feasible actions from the JEV City simulation. 1,920 decisions across two tracks, all measured — nothing estimated from documentation.
Action + uncertainty: LLMs return the action and a probability for every eligible action (what the JEV City inspector needs). JEV answers one Choice question, whose distribution is native.
- 12× faster
- JEV median 93 ms vs fastest LLM 1,104 ms
- 2.7× cheaper
- per decision vs the cheapest LLM tested
- 240/240 valid
- JEV schema-valid answers in this track (per engine; LLM failures listed below)
Measured on 80 fixed city snapshots × 3 trials per engine. Choice-only requests measured on fixed city snapshots, not full four-question JEV City transactions. Results depend on model, provider routing and benchmark environment.
Median latency (p50)
HTTPS round-trip from the benchmark host
- TypeSafe JEV93 ms
- gpt-4o-mini1,104 ms
- gemini-2.5-flash1,467 ms
Tail latency (p95)
HTTPS round-trip from the benchmark host
- TypeSafe JEV148 ms
- gpt-4o-mini1,867 ms
- gemini-2.5-flash6,426 ms
Cost per 1,000 decisions
USD · JEV: tokens × published tariff; LLMs: OpenRouter-reported cost
- TypeSafe JEV$0.041
- gpt-4o-mini$0.110
- gemini-2.5-flash$0.314
Same answer on all 3 trials
share of snapshots · higher = more repeatable; not a quality score
- TypeSafe JEV96%
- gpt-4o-mini97%
- gemini-2.5-flash89%
- Rules baseline100%
All numbers
| Engine | Resolved model | Valid | p50 | p95 | Consistency | Cost / decision | Mean input tokens | Probabilities | Failures | Agrees with JEV |
|---|---|---|---|---|---|---|---|---|---|---|
| TypeSafe JEV | jev-1.13.0 | 240/240 | 93 ms | 148 ms | 96% | $0.000041 | 968 | model-native | 0 | — |
| gpt-4o-mini | openai/gpt-4o-mini | 236/240 | 1,104 ms | 1,867 ms | 97% | $0.000110 | 555 | model-reported | probabilities_do_not_sum_to_1 ×4 | 75% |
| gemini-2.5-flash | google/gemini-2.5-flash | 235/240 | 1,467 ms | 6,426 ms | 89% | $0.000314 | 562 | model-reported | timeout ×3, schema_violation: not JSON ×1, http_429 ×1 | 83% |
| Rules baseline | — | 240/240 | 0.002 ms* | 0.003 ms* | 100% | $0.000000 | — | heuristic | 0 | 28% |
* Rules run in-process; their timing is a local function call and is not comparable to network round-trips. “Agrees with JEV” compares modal choices per snapshot — it describes similarity, not correctness.
What this does and doesn’t show
Method
- Every engine receives the identical citizen state and the identical eligible-action set.
- JEV: POST api.typesafe.ai/v1/systemone, model jev-1.13.0, one Choice question over the eligible actions (same task as the LLMs).
- LLMs: OpenRouter chat completions, strict JSON Schema (Track A: action enum only; Track B: action enum + probability per action), temperature 0.2, require_parameters, no provider fallbacks.
- Rules: deterministic hand-written heuristic with a fixed seed per snapshot (the city itself uses seeded jitter for variety).
- Order: all (track, engine, snapshot, trial) tasks shuffled together with a fixed seed, then run with bounded concurrency; concurrency 4.
- Host: developer laptop (macOS), residential network, Node v22.23.3.
Caveats
- LLM probabilities are model-reported estimates, not calibrated probabilities; JEV probabilities are native model output.
- Schema-valid is not the same as correct. Decision quality is not evaluated by this benchmark.
- Rules latency is an in-process function call and is not comparable to network round-trips.
- Both engines get the same state JSON and the same action descriptions; the LLM additionally receives a natural-language instruction wrapper.
- Latency depends on network, region and provider load at the time of the run.
- Decision quality: not evaluated. This benchmark measures validity, speed, cost and repeatability only. We make no claim about which engine decides “better”.