Ollama structured outputs on /v1: valid JSON, worse answers

Does Ollama’s OpenAI-compatible API support structured outputs? Yes: response_format with type: json_schema works on /v1/chat/completions, and the OpenAI SDK’s parse() helper works with it. On a JSON extraction task with two small Qwen 3.5 models, it made all 80 sampled replies valid JSON as-is, against 40 of 80 without it. But it did not make them more correct: 48 of 80 had every field right with the schema, 55 of 80 without it, while json_object removed the fences and kept 55. Part of the cause is in Ollama: it re-sorts the schema’s properties alphabetically before llama.cpp builds the grammar, so a ticket that should start with the customer’s name starts with category, and lists items before order_id. With the same seeds and nothing else changed, that order alone cost Qwen 3.5 0.8B 6 of 40 right tickets and made it invent products in 8 of 10 replies to the 5 emails that had none. The 2B model did not care.

Two more things turned up. Two common ways of writing response_format wrong are accepted with HTTP 200 and silently ignored. And Ollama 0.40.14 fixed the /v1 sampler defaults I measured in October, which mattered more than any format setting: with only temperature: 1.5 sent, fully right tickets went from 24 of 80 on 0.40.13 to 50 of 80 on 0.40.16.

This is the eighth post on running local models on a laptop CPU. The JSON grammar post found that llama-server’s grammar fixed syntax without changing accuracy (87 against 88 of 120). Through Ollama the grammar costs accuracy, and this post is about why.

How I tested

The same 20 labelled support emails and ticket schema as in the earlier JSON posts: customer name, order id (or null), category, priority, products with quantities, and whether the customer asks for a refund. Qwen 3.5 0.8B and 2B (Q4_K_M), imported into Ollama with a Modelfile that has only FROM, so no model-specific sampler settings are involved. Thinking off, at most 300 output tokens, every request to /v1/chat/completions, two parallel slots, a 2-core laptop CPU with no GPU.

Each condition sends as few fields as it can, like a client that sets nothing:

Condition	Sent besides the messages
none	nothing
schema	response_format as the OpenAI SDK’s parse() sends it: {"type": "json_schema", "json_schema": {"name", "schema", "strict": true}}
object	response_format: {"type": "json_object"}
schema-only	the same json_schema, but the system prompt names the fields in one sentence instead of showing the schema
t0, t0-schema	temperature 0, without and with the schema
1.5, 1.5-schema	temperature 1.5 only, without and with the schema

Temperature 0 runs once per email, every other condition twice, each request with its own seed, and the seeds are the same on both Ollama versions. In all, 920 replies from Ollama and 160 from upstream llama-server for the key order test. “Valid” means the reply passes the schema; “as-is” means json.loads on the raw reply works, with no cleanup. “Right” means all five graded fields match my labels.

Does Ollama’s OpenAI API support response_format json_schema?

Yes, in the shape OpenAI documents. To check which shapes actually reach the grammar, I sent a schema that requires {"animal": "zebra", "code": <integer>} with a prompt that asks for a favourite colour. A reply that says zebra can only come from the grammar. Same result on both versions and both models, and every request returned HTTP 200:

response_format sent to /v1/chat/completions	Schema enforced?
OpenAI SDK client.chat.completions.parse(response_format=MyModel)	yes
{"type": "json_schema", "json_schema": {"name": ..., "schema": {...}, "strict": true}}	yes
{"type": "JSON_SCHEMA", "json_schema": {...}}	yes
{"type": "json_schema", "schema": {...}}	no, silently
{"type": "json_schema", "json_schema": {...the schema itself...}}	no, silently
{"type": "json"}	no, silently
{"type": "json_object"}	any JSON object (the probe cannot tell more)

The two silent ones are easy to write by hand. The schema has to sit at response_format.json_schema.schema; anywhere else, Ollama’s OpenAI layer (openai/openai.go) finds no schema, sends no format, and the model answers unconstrained with no error or warning. The SDK’s parse() helper puts it in the right place. Ollama’s struct for json_schema has only a schema field, so name and strict are accepted and ignored; the grammar is strict either way.

Does json_schema make the answers more accurate?

No. It makes them parse. Both models together, Ollama 0.40.16, 80 replies per row (40 for temperature 0):

Condition	Valid as-is	Valid after removing a fence	All 5 fields right
t0	21/40	39/40	33/40
t0-schema	40/40	40/40	31/40
none	40/80	76/80	55/80
schema	80/80	80/80	48/80
object	80/80	80/80	55/80
schema-only	80/80	80/80	52/80
1.5	42/80	76/80	50/80
1.5-schema	80/80	80/80	48/80

Without a format, the usual failure was not broken JSON but a Markdown fence: Qwen 3.5 2B wrapped 18 of 20 greedy replies and 34 of 40 sampled ones in ```json. Any response_format removed every fence. That is worth having, because the OpenAI SDK’s parse() and a plain json.loads both fail on a fenced reply.

The cost shows in the fields. Per field, with nothing sent against the schema (0.40.16, 80 replies each): items 68 against 60, order_id 78 against 76, the other three within two of each other. json_object kept the fence fix and lost nothing (items 69, order_id 79). It constrains less than schema (any JSON object, no enums or patterns), and it lets the model write the keys in the order it reads them from the prompt: all 80 json_object replies started with customer_name.

The misses are revealing. On the 5 emails with no order and no products, the schema replies filled items for 8 of 20 sampled replies (4 of 20 without a format) and put an order id in 4 of 20 (2 without). The most common invented id was ORD-12345, the example from the schema’s own description. At temperature 0, Qwen 3.5 0.8B turned an email about a login problem into this:

{
  "category": "technical",
  "customer_name": "Daniel Okafor",
  "items": [{ "product": "account", "quantity": 1 }],
  "order_id": "ORD-12345",
  "priority": "high",
  "refund_requested": false
}

Without a format, the same model and seed wrote "order_id": null and "items": [], and put customer_name first, as the schema declares it.

Why does Ollama return the JSON keys in alphabetical order?

Because it re-encodes the schema through a Go map before llama.cpp sees it. On the path that imported GGUF models use, Ollama passes the format to its bundled llama-server as a response_format, and on the way it decodes the schema into a map[string]any (llm/llama_server.go). Go’s JSON encoder writes map keys in sorted order, so llama.cpp receives properties sorted alphabetically and builds a grammar that forces that order. Every schema reply in this test, 440 of 440, came out sorted. Upstream llama-server with the same schema keeps the declared order (it did for 120 of 120 grammar replies in the October test). The bug is reported as ollama/ollama#18717, open since September, a regression of a 2024 fix for the same thing.

For this schema the sorted order is category, customer_name, items, order_id, priority, refund_requested. The model has to classify the email before it has written anything about it, and it has to commit to a list of products before it decides whether there is an order at all.

To see whether the order alone explains the drop, I sent the same requests to upstream llama-server b11554 twice, with the same seed and Ollama 0.40.16’s samplers (temperature 0.8, top_p 0.9): once with the schema as declared, once with its properties sorted the way Ollama sends them. Nothing else differed. “Right” counts tickets with all five fields right; the made-up columns count the 10 replies to the no-order emails, declared against sorted.

Model	Declared right	Sorted right	Made-up products	Made-up order id
Qwen 3.5 0.8B	25/40	19/40	2 vs 8	2 vs 7
Qwen 3.5 2B	33/40	35/40	0 vs 2	0 vs 0
Both	58/80	54/80	2 vs 10	2 vs 7

Every reply in both conditions was valid; only the values moved. For the small model, the sorted order is the difference: same engine, same seed, and it went from inventing products on 2 of the 10 replies to the no-order emails to 8 of them, with an order id on 7. The fields that fell were items (71 against 63 of 80 across both models), category (68 against 62) and order_id (78 against 73). The 2B model reads the email well enough to get those right in any order, and its 2-ticket gain is noise at this size. So the alphabetical order does not explain every lost ticket through Ollama (the 2B lost 3 of 40 there, also within noise), but it explains the pattern of the 0.8B’s misses: the products and order ids it made up are the ones the sorted order asks for before the model has decided whether there is an order.

This matters more for schemas where order carries meaning. The common pattern of a reasoning field before an answer field turns into answer first under alphabetical order, so the model answers before it reasons. I measured that case later: on GSM8K, Qwen 3.5 0.8B solved 20 of 50 problems reasoning-first on llama-server and 1 of 50 with the same schema through Ollama.

What changed in Ollama 0.40.14 for /v1?

Before 0.40.14, a /v1 request that left out temperature or top_p got 1.0 for both, over Ollama’s own defaults and over the model’s Modelfile. I measured that in October on 0.40.3. Ollama 0.40.14, released on 3 November, leaves missing values unset (commit c1db5f9, “openai: don’t override model sampler defaults”), so the model’s parameters apply, and a model with none gets Ollama’s 0.8 and 0.9. The server log confirms it: the same empty request logged temp = 1.000, top_p = 1.000 on 0.40.13 and temp = 0.800, top_p = 0.900 on 0.40.16.

Both models, 80 replies per row:

Condition	Ollama 0.40.13 valid	0.40.16 valid	0.40.13 right	0.40.16 right
nothing sent	70/80	76/80	47/80	55/80
json_schema	80/80	80/80	41/80	48/80
temperature 1.5 only	46/80	76/80	24/80	50/80
temperature 1.5 + json_schema	80/80	80/80	31/80	48/80

(“Valid” here is after removing a fence.) The temperature 1.5 row is the one the fix changed most, because 0.40.13 also dropped top_p to 1.0, which let the long tail of unlikely tokens through (the same failure as temperature-only sampling on llama-server in the JSON temperature test). It is also the only row where the schema helped on 0.40.13: when 34 of 80 replies fail the schema even after cleanup, any grammar is an improvement. At temperature 0 the two versions gave the same text for 36 of 40 emails, so the engine underneath did not change much between them.

Do you still need the schema in the prompt?

Not in this test, but write the fields down somewhere. Ollama’s docs say it is “ideal to also pass the JSON schema as a string in the prompt”. With a prompt that only named the fields in one sentence, the schema-only condition got 52 of 80 right, against 48 of 80 with the full schema in the prompt. That is within the noise of two seeds per email: in October, two conditions with identical settings differed by up to 5 of 40 per model. It does not match the October grammar test on llama-server, where dropping the schema from the prompt cut right answers to 47 of 120; that prompt did not name the fields at all. The cheap lesson: the grammar only controls the shape, so the model still needs to be told in words what each field means.

What broke along the way

• My run hit a time limit. The benchmark runs as a background job with a 2-hour cap, and the last block (Qwen 3.5 2B on 0.40.16) was stopped at email 16 of 20. I added a RESUME=1 mode that skips every request already in raw.jsonl and finished the missing 66 on a fresh server, with the same seeds.
• The machine was busy. Other programs used about a third of the CPU when each Ollama server started, and more during the llama-server test, where a 2B reply took up to two minutes. Nothing here depends on speed: the median reply took 12 to 17 seconds with or without a grammar.
• The OpenAI SDK has a new HTTP client. openai-python 3.49.0 ships httpx2, not httpx, so my first probe failed on import httpx and left the server running. The run script now stops the server on any exit.
• Caveats. Two models from one family, 20 synthetic emails written by me, one schema. The key order effect depends on the schema: one whose declared order is already alphabetical would see none of it. Library models whose template Ollama renders itself may take a different path; the issue says the other path keeps the order. I tested imported GGUF files only.

What should you send from the OpenAI SDK?

• Upgrade to Ollama 0.40.14 or later, or send temperature and top_p in every /v1 request. That was the biggest difference in this test.
• Use response_format in the shape parse() sends, with the schema at json_schema.schema. If a schema seems to be ignored, check that path before anything else; Ollama will not tell you.
• Order-sensitive schema? Check the key order of a reply. Until #18717 is fixed, the fields come out sorted. If the order matters, name them so the sorted order is the order you want (a_reasoning, b_answer, which kept the order and the score in a later test), or use json_object and keep the schema in the prompt, which gave the same accuracy as no format here with none of the fences.
• Keep validating. A grammar guarantees the shape, not the values. Look for the example values from your own descriptions (ORD-12345 here) in the output.

Everything is in the project folder: the benchmark, the format and key order probes, all 1,080 replies and every miss. With the two Ollama releases and the GGUF files from the 10-model post, ./run.sh, ./run_order.sh and python3 grade.py reproduce the tables.