LLMs are increasingly deployed to produce structured data from unstructured and semi-structured sources, parsing invoices, medical records, meeting transcripts, and converting PDFs to database rows.

For deterministic output, the next step in a workflow reads a specific key and expects a specific type. A hallucinated invoice_total or an array ordered incorrectly because of inaccurate date values silently breaks downstream systems. Yet existing benchmarks either check schema compliance alone or evaluate value correctness within a single source domain.

Top 5 at a glance

A side-by-side look at the top 5 models across all seven metrics. The structural metrics (JSON Pass, Path Recall, Structure Coverage, Type Safety) cluster near the ceiling for every model, while Value Accuracy and Perfect Response separate them.

The problem with current structured output benchmarks

Most benchmarks collapse "structured output quality" into a single number: does the response parse, and does it validate against the schema? That's necessary, not sufficient.

References to existing benchmarks: JSONSchemaBench | StructEval | DeepJSONEval | LLMStructBench

How SOB works

SOB evaluates structured output across three modalities using the same scoring harness. The goal is to isolate the extraction capability from every other ability a model has.

Three sources, one scoring pipeline

Every record is paired with a JSON Schema and a ground-truth answer that was verified against the source context through human authoring with an LLM cross-check, so a missing or hallucinated value is unambiguously wrong.

To isolate the structured-output capability from vision and ASR quality, image and audio records are converted to text-normalized context before scoring. Models see the same modality-stripped context, and the differences that remain are attributable to how they handle schemas, nesting, and value grounding under different content distributions.

Seven metrics, not one

SOB reports seven metrics per record so you can see exactly where a model fails:

Value Accuracy is the metric that matters for production. It's the share of fields a downstream system can trust without a human review step.

Scoring gates

Two gates prevent inflated scores from schema-only wins:

• Hardening gate: If JSON parse fails, downstream semantic metrics are zeroed for that record.
• Coverage gate: Value Accuracy is only credited on fields the model actually returned, with missing paths counting as wrong.

Schemas are tagged between easy, medium or hard. The final leaderboard is schema-complexity-weighted (easy = 1.0, medium = 2.0, hard = 3.0) so hard schemas contribute more to the ranking than medium ones.

The results

We ran SOB on all models at temperature 0.0, max output 2048 tokens and no reasoning/thinking wherever the provider allows it, so the score reflects pure structured output and extraction capability.

Unified leaderboard

View the full leaderboard

The top four are within 1 point of each other on overall score, but swap freely across individual metrics. Rank order is metric-specific, not absolute.

Per-metric charts

Each chart re-sorts all 21 models on that single metric, so you can see which models win each category (not just the overall average).

To expose the gaps, each chart's x-axis starts from a floor appropriate to that metric (e.g. 60% for Value Accuracy, 80% for JSON Pass). Without that, the top cluster looks identical.

Value Accuracy

The metric production systems care about. Note how tightly the top cluster sits compared to the overall leaderboard spread.

Faithfulness

How often values are grounded in context instead of hallucinated.

JSON Pass Rate

Almost every modern model clears 95%+ in the unified leaderboard. This is why a pass-rate-only benchmark can't separate them anymore.

Path Recall

Whether all required keys appear in the output.

Structure Coverage

Whether nested objects and arrays are present with the correct shape.

Type Safety

Whether leaf values respect the declared JSON Schema types (no strings where numbers are expected).

Perfect Response Rate

The fraction of records where every single leaf value is exactly right. This is the hardest metric and collapses to roughly half even for the best models.

The JSON-pass vs Value-Accuracy gap

The single most important view: most models clear 95%+ on JSON Pass, but Value Accuracy sits 15 to 30 points lower. That gap is the space where structured output benchmarks have been lying to us.

The gap column is the headline. Every model on this list passes JSON parsing 97%+ of the time, but actual leaf-value extraction drops by 17 to 26 points. Qwen3.5-35B has the tightest gap (16.8) and the highest Value Accuracy on the list, while Schematron-8B passes JSON 98.7% of the time but lands the lowest Value Accuracy at 73.1% — a 25.6 point fall.

Modalities diverge more than we expected

The same model scores very differently across text, image, and audio, even when every model gets the same text-normalized context. Audio is the hardest by far. The transcripts are long (~7,300 tokens on average) and full of overlapping speakers, so models struggle to pull out the right values.

Best Value Accuracy by modality across all valid models:

No single model wins all three. GPT-5.4 ranks 3rd on text but 9th on images. Schematron-8B ranks 19th on text but 10th on images. Gemma-4-31B ranks 11th on text but 1st on images.

Seven patterns worth internalizing

1. Valid JSON ≠ correct JSON. JSON Pass and Value Accuracy diverge by 15 to 30 points on every frontier model.
2. Structural metrics mask value errors. Path Recall / Structure Coverage / Type Safety can all read ~99% while 20 to 30% of leaf values are still wrong, and Perfect Response collapses to about half even for the top models.
3. Model size is not a predictor. Qwen3.5-35B and GLM-4.7 beat GPT-5 and Claude-Sonnet-4.6 on Value Accuracy. Phi-4 (14B) edges out GPT-5 and GPT-5-Mini on text.
4. Structured hallucinations are the hardest bug. The value is type-correct, schema-valid, and plausible, so it slips through most guardrails. On one audio record the ground truth is "target_market_age": "15 to 35 years" and a model returns "25 to 35" — invisible without field-level checks.
5. Modalities don't transfer. Text-trained structured output behavior degrades sharply when the source is a transcribed conversation. Best Value Accuracy drops from 83.0% on text to 67.2% on images to 23.7% on audio.
6. Rankings shift across modalities. GLM-4.7 leads text, Gemma-4-31B leads images, Gemini-2.5-Flash leads audio. No single model dominates all three, so a text-only leaderboard would mask the gaps.

What's next

SOB is a first step, not a finish line. We'll keep growing the benchmark along several axes:

• More datasets, including newer datasets with increasing complexity and difficulty.
• More schemas and difficulty tiers, including recursive types, unions, and large enum spaces.
• Continuous re-evaluation as new models ship, and ongoing transparent tracking of our own models against the same scoring harness so we can measure ourselves honestly.

Why we released SOB?

Our goal is to be the best general model for deterministic tasks and a key aspect of determinism is controllable and consistent output structure. The first step to making structured output better is to measure it and hold ourselves against the best.

SOB launch video