One of the best surprises of modern local AI: you do not need a GPU to run capable models. Quantized small models run on plain CPUs — including laptops with integrated graphics — and while the speeds will never match a gaming PC's GPU, the experience is far better than most people expect. Here is what actually works, and at what RAM.
Why CPU-only works at all
As covered in CPU vs GPU, token generation streams model weights from memory. CPUs can do exactly this — they just have less memory bandwidth (typically 50–90 GB/s versus hundreds for GPUs) and less efficient compute for the attention math. The result is slower but entirely functional inference.
The other prerequisite is RAM capacity. Without VRAM, RAM is where the weights live:
| Your RAM | Realistic models (Q4) | Typical speed |
|---|---|---|
| 8 GB | 1B–3B | 10–20 tok/s |
| 16 GB | 3B comfortably; 7B possible | 5–15 tok/s |
| 16–24 GB | 7B–8B comfortably | 5–12 tok/s |
| 32 GB+ | 14B at a push | 2–8 tok/s |
(Leave room for your OS and apps — a 16 GB machine cannot dedicate all 16 GB to the model.)
What the experience is actually like
Honest expectations:
- 3B models at Q4 (10–20 tok/s): genuinely pleasant. Faster than reading speed; you will barely notice it is CPU-based.
- 7B–8B models at Q4 (5–12 tok/s): usable for chat and Q&A. Answers arrive at a readable pace; long generations test patience.
- 14B+ (single digits): a "ask and go make coffee" experience. Fine for batch work; not for conversation.
For perspective, streaming at even 8 tok/s is roughly reading-speed for comprehension — the frustration is only in generation latency while you wait mid-answer.
How to set it up
You do not need to install anything exotic:
- Ollama — install, run
ollama run llama3.2, done. Detects CPU-only automatically. - LM Studio — GUI alternative with model search built in.
- llama.cpp directly for maximum control.
Choose Q4 quantized GGUF builds (see what is GGUF) — they are the whole reason CPU-only is viable. Keep context modest (4K or less) on RAM-tight machines; the KV cache competes with weights for the same memory.
When CPU-only is the right call
- Your machine has no discrete GPU (office laptop, mini PC, older desktop).
- Your tasks are short interactions: chat, Q&A, summarization.
- You want a silent, always-available assistant rather than a speed machine.
- Batch processing where latency doesn't matter.
When it's the wrong tool: coding autocomplete (needs 30+ tok/s), reasoning models with long thinking phases, or anything interactive and long-form.
The upgrade path
If you find yourself enjoying local AI on CPU, the cheapest meaningful upgrade is a used GPU with good VRAM — even an 8 GB card transforms the experience from "usable" to "fast" (see how to choose a GPU for local AI). Until then, a well-quantized 3B–7B model on a solid CPU covers more real use than most people assume.
Want to see exactly which models your CPU-and-RAM configuration can run? The hardware checker works perfectly fine with CPU-only profiles — enter your RAM and threads, leave VRAM minimal, and see the honest list.