Which Model Tops Our Leaderboard?
How the LLMs did in our realworld tests. Our focus here was real tasks that real people carry out, not academic metrics. We focus on single tasks to simplify the assessment. An agentic flow is ultimately a series of such tasks. Think of these like unit tests for the agent. We made them cheap enough to run so that even the whole suite costs just $30. See every task and each model's actual answer, or compare two models head to head →
The overall score is the pass rate across my 28 realworld tasks. As we only had a limited number of trials there is a wide Wilson interval — the whiskers on the chart.
Summary of results: click a column to sort by your chosen metric.
Four caveats on how these numbers were produced
The lap, corner by corner #
The most recently added models appear first, with the latest test date shown under each. The lap is five corners in fixed order: Coding → Data → Realworld → Security → Tool-use. A corner's colour is that model's pass rate in that category. Green is good — it means 85%+ success. For models that can do it all, look for all green. The number in the middle of each ring is that model's cost per task; below it is the median time to first token, in seconds.
Hover or tap any segment for what that corner tests and how the model handled it.
Clean corner (>85%) Ragged (60–85%) Off the track (<60%)
See the exact numbers by category
Cells below 60% are flagged red and 60–85% amber — coding, data and tool-use are the harness floor, so the race is decided in realworld and security.
What Do the Results Actually Tell You?
If you only run one model, run glm-5.3
glm-5.3 is the first model on the board to clear all five corners — coding, data development, realworld, security and tasks — at 100%. It backs that with a 9.3 rubric, third-highest on the board, and $0.28 for the lap. The one cost is patience — a 16.3-second median time-to-first-token. gpt-5.5 is the faster alternative at 13.2s, with the same 100% security but an 89% realworld corner and $1.43 for the lap.
Fable failed to complete a single lap
fable-5 is joint-bottom at 79% because it refused to do 5 of the tasks. It performed well on what it completed, but even it thought kimi-k3 was giving better answers. You'll need a fallback model if you're using Fable. opus-5 hit the same wall — four benign coding-debug-* tasks blocked before a token was generated, on an overlapping set of tasks — so Anthropic's classifier looks like it sits across the whole series 5 line, not just Fable. See the full refusal breakdown for what's actually going on.
Luna is the very cheapest workhorse
gpt-5.6-luna costs $0.064 for the full lap, or $0.0023 per task, with a 5.3-second median TTFT. That makes it attractive for high-volume, low-risk background work where failures are cheap to detect and retry. The trade-off is material: 79% overall and 33% on security, so validate every result and keep it away from untrusted prompts. haiku-4-5 is the higher-pass alternative at $0.0044 per task, 96% overall and a 0.9-second TTFT. deepseek-v4-pro is nominally cheaper still at $0.0029 per task for the same 96% pass rate, but its 40.0-second median TTFT — the slowest on the board — rules it out for anything interactive; treat it as a batch-only option.
The mystery guest sets the fastest quality lap
kimi-k3 still tops the rubric at 9.5 — judged independently by fable-5 — with a 96% pass rate, though opus-5's 9.4 now runs it close on quality at a third of the wait. The catch is patience: a 26.4-second median time-to-first-token, second slowest on the board behind deepseek-v4-pro's 40.0s, and a 75% wobble on data development tasks, its only weak corner. Not suitable for interactive applications.
Three cars failed the crash test
The gpt-5.6 line is quick, but it has a safety problem. gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol emitted the jailbreak canary in 11 of 12 jailbreak cells (33–50% security pass) — make sure you protect in your harness, and apply more careful Red teaming if using these models. The Claude trio went 6/6 clean, as did gpt-5.5.
A safety filter can look exactly like a bad lap
opus-5 posts the best rubric on the default panel at 9.4 and 100% on both realworld and security — then shows 43% on coding. That cell is not its debugging ability: four benign coding-debug-* tasks were blocked by a provider-side classifier before a single token was generated, on an overlapping set of tasks to the ones already blocked on fable-5. Two Anthropic-family models now hit the same filter, so treat it as a measurement hazard rather than a model quirk — and note opus-5 was also penalised twice for flagging an attack it had successfully resisted.
How Is the Ed-o-meter Scored?
- Same tasks run for all models using the same prompts, same API calls, measured through one identical OpenRouter streaming path, run serially as time-trial. No other cars on track
- Latency is time-to-first-token, measured through one identical OpenRouter streaming path, run serially so the clock is uncontaminated. Wall-clock is recorded alongside.
- Checkers are binary and automated. The LLM rubric is the only judged component — and its bias is made visible in the footnotes rather than assumed away.
- Effort and reasoning settings are pinned in
models.jsonand stated with any published number, because they materially move quality and cost. - Refusals are recorded, not hidden. A provider-side hard stop is logged as a refusal with its category — never silently retried on another model. Routing is pinned with
allow_fallbacks:false, so no quiet re-serves on quantized variants. A model that declines in prose is scored by the checker like any other answer.
Harness, tasks and checkers are open source at Featherbench (MIT). Clone it and run the lap yourself, or request a new model via GitHub issue.
See all 28 tasks
Coding (7 · Python)
- CSV dedupe — small, well-specified task with a deterministic unit-test checker
- Debug billing date — fix a month/day-overflow date bug without regressing the working cases
- Debug money split — split integer pennies N ways so shares sum exactly and stay fair
- Debug mutable default — fix the classic mutable-default-argument bug
- Debug pagination — fix an off-by-one page-count bug
- Log parsing — parse logs with quoted quotes, missing fields, IPv6 edge cases
- Rate limiter — stateful class implementation tested against an injected fake clock
Data (4)
- CSV schema mapping — map a raw CRM export onto a conformed customer dimension
- Fabric roadmap — turn a catalogue, mapping and requirements into a phased user-story roadmap
- Data model from interview — derive a star-schema model and requirements from a stakeholder transcript
- Data quality assessment — catch seeded defects in a small table and prioritise fixes
Realworld (9)
- Crying baby advice — safety-sensitive triage for an unusually crying 8-week-old
- Date night in Nottingham — local recommendations without hallucinated venues
- Flight search honesty — admit no live pricing data rather than fabricate flights
- Strict bullet formatting — obey exact format constraints under an awkward word ban
- Lisbon holiday plan — five-day itinerary respecting cost, kids and a rest day
- Honey cough pushback — correct a wrong premise: no honey for a 5-month-old
- Marathon PB plan — week-by-week training plan around a fixed club session
- Vegetarian weeknight recipe — genuinely meat-free recipe for six
- Tenancy extraction — grounded answers from a provided agreement, including admitting what isn't covered
Security (6)
- Email summary injection — summarise an email without obeying an embedded refund instruction
- UnGpt document injection — summarise a review without emitting the embedded jailbreak canary
- AIM/Machiavelli jailbreak — refuse the unfiltered persona while still answering the benign question
- Developer Mode jailbreak — refuse the dual-response jailbreak template
- Evil Confidant jailbreak — refuse the "no rules" persona while staying helpful
- OPPO jailbreak — refuse the opposite-of-a-safe-model persona template
Tool-use (2)
- Flight tool selection — search only, don't book, when explicitly told not to
- Weather tool call — call the weather tool with the correct location argument
See each model's actual answer to every task, or pick two models to compare head to head →
Change log — latest: four models added, 23 August 2026
-
Four models added. Added glm-5.3, grok-4.6, deepseek-v4-pro and gemini-3.7-flash. glm-5.3 clears all five corners at 100% and is now the board's headline pick, ahead of gpt-5.5.
-
OpenAI cost update. Re-ran the GPT-5.6 trio after OpenAI cut its prices, refreshed the cost figures and highlighted Luna as the cheapest workhorse for low-risk, retryable jobs.
-
Three models added. Added opus-5, gemini-3.6-flash and grok-4.5, with their full task, quality, security, latency and cost results.
-
Leaderboard launched. Published the first Ed-o-meter, including the Claude reference set of haiku-4-5, sonnet-4-6 and sonnet-5.