Local AI folklore says you need a big GPU. The truth is more interesting: inference speed is mostly a memory bandwidth question, and both CPUs and GPUs generate tokens the same fundamental way — by streaming model weights from memory, over and over, for every single token. Which device wins depends almost entirely on how fast it can stream.
Why bandwidth beats compute here
Generating each token requires reading essentially every weight of the model once. For a 7B model at 4-bit, that is ~3.5 GB read per token. At 30 tokens per second, the memory subsystem must sustain over 100 GB/s just for weights.
This is why the numbers fall out the way they do:
| Hardware | Typical memory bandwidth | 7B Q4 speed (typical) |
|---|---|---|
| Flagship GPU (4090) | ~1,000 GB/s | 60–100+ tok/s |
| Mid GPU (4060) | ~270 GB/s | 30–60 tok/s |
| Apple M4 Pro | ~270 GB/s | 20–35 tok/s |
| Desktop DDR5 CPU | ~60–90 GB/s | 5–15 tok/s |
GPU advantage is not (mainly) raw compute — it is that VRAM sits next to the compute and moves data an order of magnitude faster.
What this predicts
The bandwidth model explains everything people observe in practice:
- Token speed barely changes with CPU core count. Inference is bandwidth-bound, not compute-bound; a 6-core and a 16-core CPU with the same RAM produce similar tok/s.
- GPU speed tracks bandwidth more than "AI cores." The 4060 and 4090 have similar compute-per-byte efficiency; the 4090's dominance comes mostly from 1 TB/s versus 270 GB/s.
- Offload is slow because of the bus, not the RAM. Moving layers between VRAM and RAM across PCIe each token adds round-trips that dominate the time.
- Apple Silicon's trick is one memory pool. Unified memory gives the GPU full-bandwidth access to all of RAM — that is why Macs host big models respectably despite "no VRAM."
When CPU-only is actually fine
CPU inference is not a punishment — it is a valid tier when your expectations match:
- 3B models: 10–20 tok/s on modern CPUs. Fully interactive.
- 7B at Q4: 5–12 tok/s on desktop machines. Usable for chat and short answers; slow for long generations.
- 14B+: increasingly painful; expect single digits and lots of patience.
Choose CPU-only when you have no GPU (or an unusable one), when you need silent/portable operation for small models, or when batch latency doesn't matter (e.g. overnight document processing).
The offload middle ground
Most local runtimes support a hybrid: put as many layers in VRAM as fit, run the rest on CPU. Speed lands between the two extremes, dropping sharply as the GPU share shrinks. Practical guidance:
- 90%+ of the model in VRAM: near full-GPU speed.
- 50–70% in VRAM: a few times slower than GPU-only, still usable.
- Below ~50%: usually worse than a clean choice either way — pick a smaller model or go all-CPU.
What to optimize, in order
- Get the model to fit in VRAM (right quantization — see 4-bit vs 8-bit).
- Maximize bandwidth within your budget tier.
- Only then chase higher core counts — for CPU-only machines, RAM speed (DDR5) genuinely helps.
The checker applies this to your actual profile: it distinguishes models that fit your VRAM from ones that would need offload, and estimates the speed difference. To see what the resulting tok/s numbers mean in practice, read tokens per second explained.