Once you've decided to run a model locally, the next fork in the road is the bit-width: the 4-bit build or the 8-bit build. Both exist for nearly every popular model. They use the same weights, the same architecture, and often the same runtime — yet they produce meaningfully different experiences. Here is how to choose.
The memory math first
Using the standard estimate (~0.55 GB per billion parameters at 4-bit, ~1.0 GB/B at 8-bit):
| Model | 4-bit (est.) | 8-bit (est.) | FP16 (est.) |
|---|---|---|---|
| 3B | ~2 GB | ~3.5 GB | ~6.5 GB |
| 7B | ~4.5 GB | ~7.5 GB | ~14.5 GB |
| 13B–14B | ~8–9 GB | ~15 GB | ~28 GB |
| 32B | ~19–20 GB | ~33 GB | ~65 GB |
The 4-bit build is roughly 40% smaller than the 8-bit build, and that gap decides a lot: it is the difference between a 14B model fitting a 12 GB card or requiring offload, or a 32B model fitting a 24 GB card with context room versus not fitting at all.
Quality: smaller than you fear, real at the edges
The practical differences:
- Chat, drafting, summarization: 4-bit and 8-bit are typically indistinguishable. Modern 4-bit formats like Q4_K_M and AWQ preserve quality remarkably well at this level.
- Coding: 8-bit has a measurable edge on complex logic; if your card fits a Q6/Q8 coding model, completions get more reliable.
- Math and dense reasoning: the most quantization-sensitive workload. If your use is heavy reasoning, prefer the highest bit-width that fits.
- Multilingual and unusual inputs: lower-bit builds occasionally show rougher behavior on less common languages.
A good mental model: 4-bit is a format choice for most users, not a compromise — 8-bit is the upgrade you make when memory allows, not the default you're downgrading from.
Speed: the counterintuitive part
Faster inference is not automatic at lower bits. Token generation is bound by memory bandwidth — how fast weights stream from memory — so a 4-bit model that streams half the bytes often runs faster than its 8-bit twin. On memory-limited cards the difference is dramatic: the 4-bit model may run entirely in VRAM while the 8-bit build spills into system RAM, turning a 40 tok/s experience into a 5 tok/s one.
That is the real speed story: the quantization that fits your VRAM is almost always the fast one, regardless of theoretical efficiency.
A decision procedure
- Estimate 4-bit and 8-bit memory for your model (table above, or ~0.55 / 1.0 GB per billion parameters).
- Add runtime overhead: 0.5–1 GB, plus context memory.
- If 8-bit fits your VRAM with 1+ GB headroom → use 8-bit (or Q6/Q8 GGUF). You get a small quality edge at similar or slightly lower speed.
- If 8-bit doesn't fit but 4-bit does → use 4-bit without hesitation. A fully-resident Q4 model beats an offloading Q8 in both speed and often sanity.
- If neither fits → smaller model, or CPU offload with Q4 and lowered expectations.
Two edge cases
- Very small models (1B–3B): with so little memory at stake, run them at 8-bit or FP16 — the quality gain is free and the speed difference is negligible.
- Very large models (70B): even Q4 doesn't fit most single GPUs. Here the choice collapses to "what extreme quantization still produces acceptable quality," and Q2/Q3 discussions replace the 4-vs-8 debate entirely.
The hardware checker applies this procedure across the catalogue for your machine — each recommendation arrives with the quantization that fits your specific profile. For the underlying format details, see what is quantization and what is GGUF.