Benchmark · October 14, 2026
The same 80 citizen decisions, asked 3 times each.
Every engine got the identical citizen state and the identical set of feasible actions from the JEV City simulation. 1,920 decisions across two tracks, all measured — nothing estimated from documentation.
Action selection: LLMs return only the chosen action (strict enum). JEV answers one Choice question.
- 12× faster
- JEV median 92 ms vs fastest LLM 1,116 ms
- 1.9× cheaper
- per decision vs the cheapest LLM tested
- 240/240 valid
- JEV schema-valid answers in this track (per engine; LLM failures listed below)
Measured on 80 fixed city snapshots × 3 trials per engine. Choice-only requests measured on fixed city snapshots, not full four-question JEV City transactions. Results depend on model, provider routing and benchmark environment.
Median latency (p50)
HTTPS round-trip from the benchmark host
- TypeSafe JEV92 ms
- gpt-4o-mini1,116 ms
- gemini-2.5-flash1,418 ms
Tail latency (p95)
HTTPS round-trip from the benchmark host
- TypeSafe JEV143 ms
- gpt-4o-mini1,716 ms
- gemini-2.5-flash4,057 ms
Cost per 1,000 decisions
USD · JEV: tokens × published tariff; LLMs: OpenRouter-reported cost
- TypeSafe JEV$0.041
- gpt-4o-mini$0.079
- gemini-2.5-flash$0.166
Same answer on all 3 trials
share of snapshots · higher = more repeatable; not a quality score
- TypeSafe JEV98%
- gpt-4o-mini96%
- gemini-2.5-flash91%
- Rules baseline100%
All numbers
| Engine | Resolved model | Valid | p50 | p95 | Consistency | Cost / decision | Mean input tokens | Probabilities | Failures | Agrees with JEV |
|---|---|---|---|---|---|---|---|---|---|---|
| TypeSafe JEV | jev-1.13.0 | 240/240 | 92 ms | 143 ms | 98% | $0.000041 | 968 | model-native | 0 | — |
| gpt-4o-mini | openai/gpt-4o-mini | 240/240 | 1,116 ms | 1,716 ms | 96% | $0.000079 | 495 | none (action only) | 0 | 74% |
| gemini-2.5-flash | google/gemini-2.5-flash | 238/240 | 1,418 ms | 4,057 ms | 91% | $0.000166 | 486 | none (action only) | http_429 ×2 | 74% |
| Rules baseline | — | 240/240 | 0.002 ms* | 0.008 ms* | 100% | $0.000000 | — | heuristic | 0 | 29% |
* Rules run in-process; their timing is a local function call and is not comparable to network round-trips. “Agrees with JEV” compares modal choices per snapshot — it describes similarity, not correctness.
What this does and doesn’t show
Method
- Every engine receives the identical citizen state and the identical eligible-action set.
- JEV: POST api.typesafe.ai/v1/systemone, model jev-1.13.0, one Choice question over the eligible actions (same task as the LLMs).
- LLMs: OpenRouter chat completions, strict JSON Schema (Track A: action enum only; Track B: action enum + probability per action), temperature 0.2, require_parameters, no provider fallbacks.
- Rules: deterministic hand-written heuristic with a fixed seed per snapshot (the city itself uses seeded jitter for variety).
- Order: all (track, engine, snapshot, trial) tasks shuffled together with a fixed seed, then run with bounded concurrency; concurrency 4.
- Host: developer laptop (macOS), residential network, Node v22.23.3.
Caveats
- LLM probabilities are model-reported estimates, not calibrated probabilities; JEV probabilities are native model output.
- Schema-valid is not the same as correct. Decision quality is not evaluated by this benchmark.
- Rules latency is an in-process function call and is not comparable to network round-trips.
- Both engines get the same state JSON and the same action descriptions; the LLM additionally receives a natural-language instruction wrapper.
- Latency depends on network, region and provider load at the time of the run.
- Decision quality: not evaluated. This benchmark measures validity, speed, cost and repeatability only. We make no claim about which engine decides “better”.