Does temperature break JSON output? Small LLMs, measured

Does temperature break JSON output from a small local LLM? I ran three small models on a laptop CPU at temperature 0, 0.8 and 1.5, 1,080 replies in total, turning support emails into JSON tickets. With llama.cpp’s default samplers left on, temperature barely mattered for validity: 119 of 120 replies were valid at 0.8 and 113 of 120 at 1.5, with no grammar at all. What broke the output was turning the other samplers off. With temperature as the only sampler, 1.5 dropped to 70 of 120 valid, and a JSON grammar only got that back to 109.

So the usual advice, “use temperature 0 for structured output”, is safe but aims at the wrong knob. The truncation samplers (top_k, top_p, min_p) decide whether high temperature produces junk. Temperature itself mostly costs accuracy, and how much depends a lot on the model.

This is the fourth post on local models on a CPU. The last one ran the same task at temperature 0 only and found that a json_schema grammar fixes syntax but not answers. It ended with a guess that sampling would make unconstrained output fail more often. This post tests that guess.

How I tested

Same 20 hand-labelled support emails and the same ticket schema as last time: a name, an order number that is null or matches ^ORD-[0-9]{5}$, two enums, an array of {product, quantity} items and a refund flag. A reply counts as valid when it passes the schema after I strip a Markdown fence, and right when all five graded fields match my labels.

Three models, bartowski’s Q4_K_M quantizations, chosen from the last run: Qwen 3.5 0.8B (the weakest), Qwen 3.5 2B (the most accurate small one) and Gemma 4 E2B (the best speed and quality trade-off in my 10-model comparison). Each email ran in five sampling conditions:

Condition	temperature	top_k	top_p	min_p
t0	0
default 0.8	0.8	40	0.95	0.05
default 1.5	1.5	40	0.95	0.05
temperature only 0.8	0.8	0 (off)	1.0 (off)	0 (off)
temperature only 1.5	1.5	0 (off)	1.0 (off)	0 (off)

“Default” is what llama-server uses when the client sends only a temperature (0.8 is also its default temperature, and Ollama’s). “Temperature only” is what you get from a stack whose defaults turn truncation off. vLLM’s SamplingParams, for example, defaults to temperature 1.0, top_p 1, top_k 0 and min_p 0 (I did not run vLLM here).

Every condition ran in two modes: prompt (the schema is in the system prompt, nothing constrains the output) and grammar (the same prompt plus response_format with json_schema, which llama-server compiles into a grammar). Sampled conditions ran twice per email with a different seed each time, so each model has 40 replies per sampled cell. That is 360 requests per model:

CONDITIONS = [("t0", 0.0, DEFAULT, [0])]
for t in TEMPS:                      # 0.8, 1.5
    CONDITIONS.append((f"default-{t}", t, DEFAULT, [1, 2]))
    CONDITIONS.append((f"pure-{t}", t, PURE, [1, 2]))
...
seed = 42 if cond == "t0" else zlib.crc32(f"{email['id']}|{cond}|{rep}".encode())

The hardware is the same as the rest of this series: an Intel i5-7200U (2 cores, 4 threads), 19 GB of RAM, no GPU, llama.cpp b11554 with 2 threads and four server slots so four requests ran at once. Thinking was off and the limit was 300 output tokens. The whole run took 2 hours 45 minutes, and the server returned no errors.

Does temperature 0.8 break JSON output?

No. Here are all three models together:

Condition	Mode	Valid	All 5 fields right	Hit the 300-token limit
t0	prompt	59/60	47/60	0
t0	grammar	60/60	48/60	0
default 0.8	prompt	119/120	86/120	0
default 0.8	grammar	120/120	95/120	0
temperature only 0.8	prompt	112/120	84/120	0
temperature only 0.8	grammar	120/120	81/120	0
default 1.5	prompt	113/120	74/120	0
default 1.5	grammar	120/120	82/120	0
temperature only 1.5	prompt	70/120	47/120	24
temperature only 1.5	grammar	109/120	51/120	11

At 0.8 with the defaults, validity was the same as at temperature 0 (99% vs 98%) and accuracy dropped a little: 72% of tickets fully right without a grammar and 79% with one, against 78% and 80% at temperature 0. Each model has only 40 replies per cell, so a difference of three or four replies is noise. What matters is the size of the drops further down the table.

One thing that did change: Markdown fences. Qwen 3.5 0.8B never wrapped its JSON in a fence at temperature 0, but did in 4 to 12 of 40 replies per sampled condition. Gemma fenced every single reply, at every temperature. If you parse with json.loads() and nothing else, only 43 of 120 default-0.8 replies parse; with a small fence stripper in front, 119 pass the schema. You need that cleanup step at any temperature, or a grammar.

Why didn’t temperature 1.5 break it?

Because llama.cpp applies temperature last. Its default sampler chain, from common/common.h in the build I used, is:

penalties, dry, top_n_sigma,
top_k, typ_p, top_p, min_p,
xtc, temperature

min_p 0.05 drops every token whose probability is under 5% of the top token’s, and it does that on the model’s own probabilities, before temperature reshapes them. At 1.5, temperature only flattens the choice among tokens that already survived. A rare token, like a word from another language in the middle of a customer’s name, never gets a chance.

Turn the truncation off and temperature 1.5 reaches every token in the vocabulary. This is what Qwen 3.5 0.8B wrote for one email without a grammar, cut at the 300-token limit (the end of the reply, line breaks mine):

...Breath缺失Benz profundo Justice
Mở才行Childان chess緣 benutzt
romanticTiείς調節UP投。Send第一冬季tight...

Other replies started as a correct ticket and drifted halfway through a value (line breaks mine):

{"customer_name":"Chen Wei",
 "order_id":"ORD-55310",
 "category":"import_without_pay/track_
shipment Housing Combo Gim Ausbau ÜMULTAGE
RHome UW white Logibar build carb Ofer ...

Can a JSON grammar fix high temperature?

Partly. With the grammar, every reply stayed valid JSON as far as it went, because the grammar masks every token that would break the syntax. But a string field accepts almost anything, so at temperature 1.5 without truncation the model opened a string and filled it with junk until it ran out of tokens. 11 of 120 grammar replies hit the 300-token limit that way and never closed the object, so they were not valid JSON in the end:

..."product": "吻 marito Hue already 세

The 109 that did close were valid and mostly wrong: only 51 of 120 had all five fields right. A grammar guarantees the shape. It cannot make a flattened distribution pick the right words, which is the same lesson as the temperature 0 test, only louder.

Which small models handle temperature best?

The averages hide how differently the models reacted. Tickets with all five fields right, prompt mode / grammar mode:

Condition	Qwen 3.5 0.8B	Qwen 3.5 2B	Gemma 4 E2B
t0	13/20 / 13/20	18/20 / 18/20	16/20 / 17/20
default 0.8	23/40 / 24/40	33/40 / 38/40	30/40 / 33/40
temperature only 0.8	19/40 / 18/40	32/40 / 32/40	33/40 / 31/40
default 1.5	13/40 / 16/40	33/40 / 34/40	28/40 / 32/40
temperature only 1.5	3/40 / 6/40	11/40 / 13/40	33/40 / 32/40

Gemma 4 E2B did not care. It stayed between 28 and 33 of 40 in every sampled condition, including temperature 1.5 with nothing else on, and 359 of its 360 replies were valid after fence removal. My guess is that its probabilities on this task are sharp enough that even a flattened distribution keeps the right token on top; I did not measure that. Qwen 3.5 2B was fine with the defaults at both temperatures and collapsed only without truncation. Qwen 3.5 0.8B lost accuracy at every step: 65% right at temperature 0, 58% to 60% at default 0.8, 33% to 40% at default 1.5, and 8% to 15% at 1.5 with temperature only.

Invented order numbers, the failure I worried about most last time, came only from the smallest model. Qwen 3.5 2B and Gemma never put an order number on the five emails that had none. Qwen 3.5 0.8B did in 1 of 5 replies at temperature 0 and in 1 to 5 of 10 in the sampled conditions, worst at default 1.5 (5 of 10).

What broke along the way

My first run gave every email the same seed: seed 1 for the first repeat, seed 2 for the second. After the first model finished, the fence column looked suspicious. Qwen 3.5 0.8B had fenced exactly 20 of 40 replies in three conditions. Split by seed:

Condition	Seed 1	Seed 2
default 0.8	20/20 fenced	0/20 fenced
temperature only 0.8	20/20 fenced	0/20 fenced
default 1.5	20/20 fenced	0/20 fenced
temperature only 1.5	20/20 opened with an empty <think> block	0/20

The first token of every reply is a choice between { and a fence, and its odds seem to be similar for every email. With the same seed, the sampler draws the same random number for that first token, so it makes the same choice for all 20 emails. A fixed seed per request is reproducible, which is why people set one, but if you reuse it for every request, your “random” sampling is one coin flip repeated. I threw that run away (it is kept in results/shared-seed/ as evidence) and reran everything with a seed per email, condition and repeat. That cost about 45 minutes.

Limits worth knowing:

• 40 sampled replies per model and cell. The collapse at 1.5 without truncation is far outside the noise; differences of a few replies are not.
• Four requests ran at once, so batch numerics can differ slightly from one request at a time.
• repeat_penalty was off (1.0, the llama-server default). Ollama documents different defaults (top_p 0.9, min_p 0), and I tested only llama-server. Ollama launches llama-server underneath but sends its own options.
• 20 synthetic emails that I wrote, one schema, Q4 quantization, thinking off.

What temperature should you use for JSON output?

• Temperature 0 is still the safe default for extraction. For all three models it was as valid as any sampled condition and within noise of the most accurate one.
• If you need sampling (retries that give a different answer, or a client that does not let you set 0), keep the truncation samplers on. With llama-server’s defaults, temperature 0.8 cost little accuracy and almost no validity, and 1.5 only hurt the smallest model.
• Check what your stack sends. A client that sets top_p: 1 or a server whose defaults turn truncation off turns temperature 1.5 into junk, and a grammar will not stop it.
• Strip fences or use a grammar at any temperature. The fence is the most common failure, not broken JSON.
• Don’t reuse one seed for every request if you want independent samples.

Everything is in the project folder: the bench, the grader, all 1,080 replies with server timings, every miss, and the discarded shared-seed run. With the llama.cpp build and models from the 10-model post, ./run.sh and python3 grade.py reproduce the tables.