Small LLM JSON output: the grammar fixes syntax, not answers

Can small local LLMs return reliable structured JSON? I gave six of them, from 0.8B to 4B parameters, the same job on a 2-core laptop CPU: turn 20 customer support emails into JSON tickets that match a schema. With llama.cpp’s json_schema grammar turned on, all 120 replies passed the schema. But the grammar didn’t make the answers better: 87 of 120 tickets had every checked field right with the grammar, 88 without it. And when I left the schema out of the prompt and relied on the grammar alone, only 47 of 120 were right.

So the grammar fixes syntax, not answers. The part that decides whether the fields are correct is still the prompt and the model.

This is the third post on local models on a CPU. The first showed that Ollama and llama.cpp run at the same speed with equal settings, and the second compared 10 small models on speed, memory and a sanity check. There, every model returned a trivial {"name", "age"} object correctly. This time the JSON had to carry real decisions.

How I tested

The task is a common one: read a support email and fill in a ticket. The schema has seven required fields and the usual JSON Schema features: a string, an order number that is either null or matches ^ORD-[0-9]{5}$, two enums (category and priority), an array of {product, quantity} objects with an integer minimum, and a boolean refund_requested. The category enum carries a description of what each value means.

I wrote 20 short emails with hand labels. They include the traps real mail has: a customer who says “I don’t want a refund”, an email with no order number at all, a company name next to the person’s name, a “$389” and a “version 5.2” that are not quantities. Here is one:

I ordered 2 phone cases (order ORD-77140) but my card was billed for 4. Please refund the extra two.

Lucía Fernández

Each email went to each model in three modes:

Mode	System prompt	Output constraint
prompt	Instructions plus the full JSON Schema	None
grammar	Instructions plus the full JSON Schema	response_format with json_schema
grammar-only	One line: “You turn customer support emails into JSON tickets.”	response_format with json_schema

In the grammar modes, llama-server converts the schema into a grammar and, at every step, masks out the tokens that would break it. The model can’t write a missing brace, a wrong key or a category outside the enum. The request looks like this:

payload["response_format"] = {
    "type": "json_schema",
    "json_schema": {"name": "ticket", "strict": True, "schema": schema},
}

I graded five fields against the labels: customer_name, order_id, category, refund_requested and items (same number of items, each with the right quantity). Priority has no single right answer, so it isn’t graded. Two emails accept two categories (a broken blender with a refund request is technical or billing). A ticket counts as right only if all five fields are.

Everything ran on llama.cpp b11554 with 2 threads, temperature 0 and thinking off, one server per model with three slots (one per mode, so each mode’s system prompt stays cached) and a 6 GB memory cap. The models are bartowski’s Q4_K_M quantizations: Qwen 3.5 0.8B, 2B and 4B, Llama 3.2 1B, Qwen 2.5 1.5B and Gemma 4 E2B. I left out four of the last post’s ten models (SmolLM3, Llama 3.2 3B, Phi-4 mini and Gemma 4 E4B, which needs my whole 6 GB budget) to keep the run under two hours. No reply hit the 300-token limit, and the server returned no errors.

Do small local models return valid JSON without a grammar?

Mostly yes, but rarely bare JSON. Every one of the 120 unconstrained replies contained a JSON object that parsed. The catch was the wrapper: Gemma 4 E2B put the JSON in a Markdown code fence in all 20 replies, Qwen 3.5 2B in 19 and Llama 3.2 1B in 15, even though the prompt asked for “only a JSON object”. json.loads() on the raw reply fails on those. Stripping the fence fixed every one.

Schema validity is where the unconstrained mode lost points, and only on the two smallest models:

Model	Parses as-is	Parses after removing the fence	Passes the schema
Qwen 3.5 0.8B	20/20	20/20	19/20
Llama 3.2 1B	5/20	20/20	17/20
Qwen 2.5 1.5B	20/20	20/20	20/20
Qwen 3.5 2B	1/20	20/20	20/20
Gemma 4 E2B	0/20	20/20	20/20
Qwen 3.5 4B	20/20	20/20	20/20

The four schema failures were all different. Two order numbers in the wrong format (ORD-123456789, ORD-123456), one extra "$schema" key, and one I didn’t expect: for “Please delete my account and all my personal data”, Llama 3.2 1B wrapped the whole ticket in {"$delete": {...}} and set the category to null.

With the grammar on, all 120 replies in each grammar mode parsed as-is and passed the schema. No fences, no extra keys. If your code calls json.loads() on the raw reply and needs every field present, the grammar is what makes that safe.

Does a json_schema grammar make answers more accurate?

No. With the schema in the prompt, the grammar changed almost nothing:

Model	prompt	grammar	grammar-only
Qwen 3.5 0.8B	12/20	12/20	3/20
Llama 3.2 1B	8/20	8/20	1/20
Qwen 2.5 1.5B	15/20	15/20	5/20
Qwen 3.5 2B	19/20	18/20	13/20
Gemma 4 E2B	15/20	15/20	11/20
Qwen 3.5 4B	19/20	19/20	14/20
Total	88/120	87/120	47/120

The table counts tickets with all five fields right. Comparing email by email, the prompt and grammar modes gave the same fields wrong in 115 of 120 cases. That makes sense: at temperature 0, the grammar only changes a reply when the model’s first choice would break the schema, and these models rarely wanted to.

Where it did step in, it didn’t always help. For an email with no order number, Qwen 3.5 0.8B invented ORD-123456789 without the grammar, which fails the pattern and would be caught by validation. With the grammar, the same reply became ORD-12345: valid against the schema, and just as made up. The grammar can only choose among the tokens the schema allows; it has no idea which value is true.

The misses that remained were about understanding the email, not formatting. The hardest emails were the damaged parcel (one of three mugs broken, “no refund needed”, which half the models called technical), the lost phone and two-factor code (the phone ended up as an ordered item, the category as technical or billing), and the parcel where only the router arrived (most models listed the two missing extenders and dropped the router). Some of those misses are judgment calls on my labels, such as whether the mug order is 3 mugs or 1 broken mug, or whether the router that did arrive belongs in the list; every miss is listed so you can disagree with my grading.

Do you still need the schema in the prompt?

Yes. This was the biggest effect in the test. In grammar-only mode the model still had to produce the exact keys and enum values, so every reply was valid, but it was guessing what they meant. Accuracy fell from 87 to 47 of 120, and every model lost at least 4 tickets:

• Invented order numbers. Five emails have no order number. With the schema in the prompt, models filled one in for 3 or 4 of those 30 cases; with the grammar alone, for 13. The grammar allows null, but nothing told the model that null was the answer.
• Invented refunds. refund_requested was wrong 17 times in grammar-only mode, every time a true for an email that asked for no money, against 3 times in the other two modes combined. Without the description “true only if the customer asks for money back”, small models tended to say true for any complaint.
• Wrong categories. “Do you have a store in Lisbon where I can see the sofas?” became shipping in all 6 grammar-only replies. The category descriptions in the schema were doing real work.

llama-server doesn’t add the schema to the prompt for you; it only uses it to build the grammar. If you pass response_format and nothing else, the model sees the field names in the grammar’s forced tokens and nothing more.

The cost of showing the schema is prompt length. With it, the system prompt was 500 to 580 tokens instead of 55 to 74, and the first request took 12 to 63 seconds just to read it on this CPU. After that, llama-server’s slot cache kept the system prompt, and each new email only needed 41 to 53 tokens of prompt processing. Keep the system prompt identical across requests, and put the variable text last, so the cache can do this.

Does grammar-constrained decoding slow generation?

Not measurably here. Within each model, median generation speed with and without the grammar differed by 2% or less, for example 14.45 and 14.65 tokens/s on Qwen 3.5 0.8B, 3.91 and 3.97 on Qwen 3.5 4B. Replies were 74 to 95 tokens long, so a ticket took 7 to 29 seconds per email depending on the model.

One honest caveat: the laptop wasn’t idle. Other programs used 26% to 68% of the CPU when each model started, so don’t compare speeds across models here. Llama 3.2 1B ran at only 3 to 4 tokens/s in all three modes, against 15.9 in yesterday’s benchmark, and I can’t separate the load from anything else. The comparison between modes holds because they were interleaved email by email on the same server.

What should you use?

• Pass the schema in the prompt and as a grammar. The prompt gives the meaning, the grammar guarantees the format. Together they gave valid JSON in 120 of 120 replies with no loss in accuracy.
• Write descriptions into the schema. “null when the email has none” and “true only if the customer asks for money back” are what kept models from inventing values.
• Still validate the values. Valid JSON is not correct JSON. A grammar can turn an obviously broken value into a plausible wrong one.
• Without a grammar, strip code fences before parsing. Three of the six models fenced their JSON, even when asked not to.
• Pick the model for the reading, not the format. Qwen 3.5 2B and 4B got 19 of 20 right; Llama 3.2 1B got 8. The grammar evened out the syntax and left that gap untouched.

What broke along the way

• The first smoke test was slow: the 0.8B model generated about 6 tokens/s instead of 19, because other programs were using close to three of the four logical CPUs. I couldn’t stop them, so I cut the run to six models, dropped a free-text summary field from the schema to shorten the replies, and recorded the background load before each model.
• My first launch was tied to my session’s two-hour tool limit, which the full run might have exceeded. I stopped it early, started it again as a detached process, and threw away the partial results.
• I wrote my own schema validator to keep the project dependency-free, then checked it against the jsonschema Python package. They agreed on all 360 replies.
• Caveats: 20 synthetic emails written by me, one schema, temperature 0, Q4 quantization, thinking off. Real support mail is messier. I guessed that temperature above 0 would make the unconstrained mode fail more often; a follow-up measured it: with llama.cpp’s default samplers, validity held at temperature 0.8 and 1.5, and it broke only when top_k, top_p and min_p were turned off.

Everything is in the project folder: the emails and labels, the schema, the scripts, every raw reply with the server’s timings, and the graded results. With the models from the previous post downloaded, ./run.sh reproduces it in under two hours on a laptop like this one.