Valid JSON was never the hard part
Grammar-constrained decoding makes an unconditional promise: mask out every token that would break the schema, and the model literally cannot emit invalid JSON. No parse errors, no missing braces, no trailing comma. XGrammar, now the default structured-generation backend behind vLLM, SGLang, TensorRT-LLM, and MLC-LLM, advertises near-zero overhead doing this. Every major model provider ships some version of the same guarantee today. It’s a real promise, and it answers a question that stopped being expensive a while ago — a parse failure was never the costly kind of wrong.
BAML’s engineering team put a number on the failure mode that actually costs money. Hand a receipt with 0.46 units of bananas to gpt-5.2 under a structured-output schema that expects an integer quantity, and the model returns 1. Free-form completion on the identical input returns 0.46. No parse error either way — the structured version just validates cleanly while being wrong. That’s the blind spot: masking works by deleting probability mass from anything schema-invalid at each step, and if the true value doesn’t fit the type you declared, there’s no token left for the model to flag the mismatch with. It can’t hedge. It picks the nearest legal value and moves on, and the schema says nothing went wrong.
This isn’t a new worry; it just gets a fresh number attached every few months. An August 2024 study found JSON-mode measurably hurting GSM8K and Last Letter — reasoning tasks — while barely touching classification tasks, and the gap wasn’t explained by parse failures, which stayed near zero in both conditions. Whatever the constraint was costing, it wasn’t legibility. A February 2026 benchmark comparing plain JSON, constrained JSON, and the token-efficient TOON format across twenty-one models found the same shape of damage at the model level: Hermes-4-405B scored 92.5% one-shot accuracy generating plain JSON and 35% under constraint on the identical task. The grammar interferes with the model’s own probability distribution, pushing it down token paths it wouldn’t otherwise take. A study from that April measured the mechanism more directly on Qwen3-8B doing multi-hop reasoning: accuracy dropped from 50% to 38% under constraint, and in the failed cases the model’s attention visibly shifted toward satisfying the grammar rather than the question. Structure snowballing, they called it: a perfectly formed answer wrapped around reasoning nobody actually checked.
None of that is unanimous, and treating it as settled would be its own mistake. JSONSchemaBench ran comparable reasoning benchmarks across several constrained-decoding engines and found the opposite sign: accuracy up three to four points against unconstrained generation. Same category of task, opposite conclusion, published a year apart. The disagreement is real, and the likely explanation is unglamorous: these studies aren’t measuring the same thing. One-shot accuracy under a hard constraint, final accuracy after a repair loop, single-field extraction versus multi-step arithmetic — change any one of those and the sign of the result can flip. The technique has a fixed effect on parseability. It has a conditional one, task by task and model by model, on everything else.
Type systems settled a version of this argument decades ago, and it’s worth remembering how. A type checker guarantees a program won’t crash from a type mismatch. It has never guaranteed the arithmetic inside that program is correct — those are different axes, and nobody mistakes “compiles” for “correct” anymore, not after enough production incidents taught the difference the hard way. “Schema-valid” is walking the same path. It’s a real guarantee, worth having, about shape. It was never a stand-in for a guarantee about content.
The fix that sticks has nothing to do with the format war — TOON against JSON, YAML against whatever comes after. What matters is where the constraint gets applied. Let the model reason in whatever shape it wants first, unconstrained, and clamp the grammar down only on the final structured emission, once the thinking is already done. That’s what BAML’s own alternative amounts to: something they call schema-aligned parsing, which reads the free-form answer and coerces it into the schema afterward, so a mismatch surfaces as an error instead of disappearing into a silently rounded number. Constrain the write. Never the thought.
Related, on this site: a small converter that moves data between JSON, YAML, TOML, and TOON, useful for seeing exactly what the syntax overhead costs on a given payload.