Best small LLM for a CPU-only laptop: 10 models measured

What is the best small LLM for a laptop with no GPU? I ran 10 of them, from 0.8B to 7.5B parameters, on a 2-core Intel laptop CPU. Gemma 4 E2B was the best trade-off: 7.1 tokens/s, 3.9 GB of RAM and 18 of 20 sanity questions right. If memory is tight, Llama 3.2 1B ran at 15.9 tokens/s in 1 GB but got 14 of 20. If you can wait, Gemma 4 E4B got all 20 at 3.9 tokens/s.

Lists of the best models for a CPU tend to recommend the same names (Qwen 3.5 4B, Gemma 4 E4B, Phi-4 mini, SmolLM3, Llama 3.2), usually without numbers measured on a CPU. When numbers do appear, they often come from a desktop with fast memory or don’t name the hardware. I wanted all of them on the same ordinary machine, with the same settings, and with memory measured instead of guessed: an Intel i5-7200U laptop (2 cores, 4 threads, 15 W), 19 GB of RAM, no GPU.

This is the second post on local models on a CPU. The first found that Ollama and llama.cpp run at the same speed with equal settings, so this time I used llama.cpp directly and every number applies to Ollama too.

How I tested

All 10 models are Q4_K_M GGUF files from the same quantizer (bartowski on Hugging Face), run with llama.cpp build b11554 on 2 threads, a 4,096-token context and one slot. Each model got four measurements:

1. Speed with llama-bench: prompt processing (256 tokens) and generation (64 tokens). I ran 3 rounds, and every round went through all 10 models, so any background load hit them alike.
2. First-token latency: a real chat request through llama-server, streamed. The prompt is a 300-word engineering note plus “List the three problems the team had with the hosted API”, 362 to 429 tokens depending on the tokenizer. Median of 3 after a warm-up.
3. Peak memory of llama-server, with weights read into RAM (--load-mode none) so the number is real allocation, not mapped file pages.
4. A 20-question sanity check, temperature 0, thinking off: arithmetic, dates, facts, a translation, two letter-level tasks, extraction, JSON output, sentiment, SQL and a one-line bug fix. Every question has one right answer and a grading rule.

Each server ran in a systemd scope capped at 6 GB, my memory budget for any experiment on this laptop, so a model that needs more gets killed alone:

systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=0 \
  llama-server -m "$gguf" -c 4096 -np 1 -t 2 --load-mode none --reasoning off --cache-ram 0

One condition I didn’t choose: the laptop wasn’t idle. Another program kept one of the 4 logical CPUs busy the whole night, 27 to 30% of total CPU before each speed round. I couldn’t stop it, so I measured it and interleaved the runs instead. Generation speed stayed within 16% of each model’s median across rounds. Qwen 2.5 1.5B, which I also measured the day before, generated 13.0 tokens/s here against 14.1 then, so expect numbers about 8% higher on an idle machine like this one.

Which small LLM is fastest on a CPU?

The smallest, as expected, but the order isn’t strictly by parameter count:

Model	Params	File	Prompt t/s	Generation t/s	First token	Peak RAM	Sanity check
Qwen 3.5 0.8B	0.8B	0.53 GiB	76.8	18.9	5.3 s	765 MiB	12/20
Llama 3.2 1B	1.2B	0.74 GiB	48.6	15.9	8.5 s	1,032 MiB	14/20
Qwen 2.5 1.5B	1.5B	0.91 GiB	31.1	13.0	14.3 s	1,173 MiB	15/20
Qwen 3.5 2B	1.9B	1.29 GiB	26.0	9.1	12.7 s	1,517 MiB	14/20
Gemma 4 E2B	4.7B	3.21 GiB	18.5	7.1	17.0 s	3,880 MiB	18/20
SmolLM3 3B	3.1B	1.78 GiB	14.9	6.2	33.5 s	2,269 MiB	16/20
Llama 3.2 3B	3.2B	1.87 GiB	14.6	5.8	28.2 s	2,517 MiB	16/20
Phi-4 mini	3.8B	2.31 GiB	12.1	5.3	28.9 s	3,043 MiB	15/20
Qwen 3.5 4B	4.3B	2.80 GiB	12.0	4.2	40.8 s	3,168 MiB	19/20
Gemma 4 E4B	7.5B	5.02 GiB	11.2	3.9	36.2 s	6,002 MiB	20/20

Gemma 4 E2B is the odd one. Its file holds 4.7 billion parameters and is bigger than any 3B model here, yet it generates faster than all of them. The “E” stands for effective parameters. According to Google’s model card, a large share of the weights are per-layer embedding tables that are “only used for quick lookups”, leaving 2.3B effective parameters (4.5B for E4B). You pay for the lookups in RAM (3.9 GB) instead of speed.

First-token time is where a CPU hurts most. Every model has to read the whole prompt before it writes anything, at 11 to 77 tokens per second. A 380-token question took 5 seconds on the smallest model and 36 to 41 seconds on the 4B-class ones. A chat that pastes a page of context into every request will feel slow on all of them.

Are small models accurate enough to be useful?

For some jobs, yes, even the smallest. Every model, including the 0.8B, passed all six of these: extracting an email address, returning {"name": "Maria", "age": 34} from a sentence, labeling a review as negative, naming SQL’s DISTINCT, fixing return a - b, and naming the Red Planet. If your task is extraction, classification or JSON from text, a 1B model is fine and about four times faster than the 4B-class models. (A harder follow-up changed that picture: when 20 support emails had to become schema-valid JSON tickets, Llama 3.2 1B got only 8 of 20 fully right and Qwen 3.5 2B and 4B got 19. A second follow-up raised the temperature: with llama-server’s default samplers, 113 of 120 replies were still schema-valid at 1.5.)

Where they split was arithmetic in words and anything at the letter level:

Question	Models right (of 10)	Typical wrong answer
Monday + 10 days	3	“Sunday”, “Tuesday”
Reverse “laptop”	4	“ptapol”, “tpalop”
3 boxes of 12 pencils, give away 9	5	“3”
Count the r’s in “strawberry”	5	“1”, “2”
len("hello world")	6	“5”
5 km in meters	7	“800”, “1000”, “5000000”

The 20 questions are a sanity check, not a benchmark. One or two questions of difference mean little; the pattern does. The four models under 2.5B parameters got only 9 of their 24 tries at the six arithmetic and date questions right. The two Gemma 4 models and Qwen 3.5 4B were the only ones to miss at most two.

Two grading notes. Qwen 3.5 and SmolLM3 are thinking models, and I turned thinking off (--reasoning off, plus enable_thinking: false in the request). With thinking on they may score higher, but each answer would cost hundreds of extra tokens at 4 to 19 tokens per second. And I graded number answers by the first number, since each question asked for the number alone. SmolLM3 answered the pencils question with its working and the right result (27), so I counted it as correct by hand; the override is in the repo.

How much RAM does a small LLM need?

Roughly the file size plus 0.2 to 0.9 GB on this setup, if you turn one default off. I measured peak memory twice. The first pass used llama-server’s defaults and asked 24 questions per model. The second added --cache-ram 0:

Model	Peak RAM, default	Peak RAM, --cache-ram 0
Qwen 3.5 0.8B	1,684 MiB	765 MiB
Qwen 3.5 2B	2,432 MiB	1,517 MiB
Llama 3.2 3B	2,759 MiB	2,517 MiB
Qwen 3.5 4B	5,557 MiB	3,168 MiB
Gemma 4 E4B	killed at 6 GB	6,002 MiB

Recent llama-server builds keep a prompt cache in host RAM, up to 8 GB by default, so a returning conversation doesn’t have to be processed again. Each request I sent was different, so the cache only grew. For the three Qwen 3.5 models it grew most: 0.9 to 2.4 GB after 24 requests. They use a hybrid architecture with a recurrent state next to attention, and I didn’t dig into what exactly the cache stores for them. With the cache off, Gemma 4 E4B grew only 77 MiB over the same 24 requests, so request count alone doesn’t inflate memory.

On a laptop this decides what fits. Gemma 4 E4B was killed by my 6 GB cap during the questions with the cache on, and passed all 20 with it off. If you run llama-server for a single user on a small machine, pass --cache-ram 0 or a small value, and watch memory after a few requests, not only after loading.

Why is llama.cpp so slow when my laptop is busy?

Because of the thread count. This was the surprise of the night. My first smoke test used 4 threads, the number of logical CPUs, and the smallest model was still writing its 128 test tokens after several minutes. With another program already holding one logical CPU, I ran llama-bench on Qwen 3.5 0.8B at 2, 3 and 4 threads:

Threads	Prompt t/s	Generation t/s
2	61.0	17.84
3	81.3	19.18
4	32.0	0.37

Four threads generated 52 times slower than three. llama.cpp’s worker threads wait for each other at every step, and when one of them shares a CPU with another busy program, all of them wait for it, for every token. The same collapse was reported in Ollama inside CPU-limited containers (0.27 tokens/s with 8 threads on 4 allowed CPUs, 12 with 4). On an idle laptop, the extra threads helped a little instead: 4 threads gave 6 to 14% more on this same CPU yesterday.

The safe rule: never give llama.cpp every logical CPU on a machine that does anything else. Set -t in llama-server or num_thread in Ollama to the number of physical cores (2 here, which was within 8% of the best), and go higher only on an otherwise idle machine. If a small model suddenly crawls at under 1 token per second, check threads before blaming the model.

So which small model should you run on a CPU?

• Under 1.5 GB of RAM, simple tasks: Llama 3.2 1B. 15.9 tokens/s, 8.5 s to the first token, fine for extraction, classification and JSON. Qwen 3.5 0.8B is faster still but missed the most questions (12 of 20).
• The best balance, if you have 4 GB free: Gemma 4 E2B. Half the speed of the 1B models, 18 of 20 right, and faster than every 3B model I tried.
• Quality first, and you can wait: Qwen 3.5 4B (19 of 20, 3.2 GB) or Gemma 4 E4B (20 of 20, 6 GB). Around 4 tokens per second and 36 to 41 seconds before the first word of a 380-token prompt. Fine for a background job, tiring for a chat.
• Skip on this kind of CPU: none of the 3B to 4B models beat Gemma 4 E2B on both speed and score here. SmolLM3, Llama 3.2 3B and Phi-4 mini were slower and got 15 or 16 of 20.

What broke along the way

• My first full run measured the wrong process. $! after starting the server through a shell function pointed at a wrapper subshell, so kill never reached the real server. The second model’s answers then came from the first model, which was still holding the port. I caught it because Llama 3.2 1B gave exactly the same wrong answers as Qwen 3.5 0.8B. The script now finds the server by name, refuses to start if the port is taken, and waits until it is free.
• My first grading rule for numbers took the last number in the answer. That fixed SmolLM3’s pencils answer and broke its “9.9 is larger than 9.11”, which is right. No single rule fit every verbose answer, so I went back to the first number and wrote the one manual correction to a file.
• Background load from other programs varied through the night, from 27% to 100% of the CPU. The first pass ran during the worst of it, which is why the speed table comes from the interleaved rounds, and why I record the background load before each model.
• Caveats: one laptop, one quantization, 2 threads, a 4,096-token context, thinking off. GPU offload is a different comparison.

Everything is in the project folder: the scripts, all 20 questions with their grading rules, every answer from every model, the memory tables and the raw llama-bench output. On a Linux laptop, ./setup.sh downloads the models (about 20 GiB) and ./speed_rounds.sh reproduces the speed table in about 22 minutes.