Quantization is the reason a model with 7 billion parameters can run on a 6 GB laptop GPU. Without it, local AI would be a hobby for owners of 24 GB cards and nothing else. Understanding it takes about five minutes and pays off every time you choose which model file to download.
The problem: weights are enormous
A language model stores its knowledge in billions of numeric weights. At full training precision (FP16 — 16 bits per weight), each weight is 2 bytes, so a 7B model needs ~14 GB before a single token of context. That is already beyond most consumer GPUs.
Most of that precision is wasted. The neural network tolerates small representation errors the same way a JPEG tolerates them: what survives is the structure, not every exact decimal. Quantization exploits that slack by storing weights in fewer bits.
What the bit numbers mean
- FP16 / BF16 (16-bit): training precision. ~16 GB for a 7B model. Almost never worth it locally.
- 8-bit (Q8, Int8): near-lossless in practice. Roughly half the memory of FP16 — around 7–8 GB for 7B.
- 4-bit (Q4, Int4, AWQ 4-bit): the local-AI standard. ~4–5 GB for 7B, with quality close enough that most users cannot tell in everyday tasks.
- 2–3 bit (Q2/Q3): extreme compression that does degrade noticeably — a last resort for giant models.
The pattern: each halving of bits roughly halves the weight memory. Quality loss is gentle from 16→8→4 bits and steep below 4.
The formats you'll encounter
Quantization comes tied to runtime ecosystems, and the names are not interchangeable:
- GGUF — the format of llama.cpp and its ecosystem (Ollama, LM Studio). Quantization levels appear as Q4_K_M, Q5_K_S, Q8_0, etc. Works on CPU, NVIDIA, AMD and Apple hardware. See what is GGUF.
- GPTQ — a 4-bit (and 8-bit) format for GPU inference, common in the vLLM/ExLlama ecosystem. The model is quantized ahead of time per layer.
- AWQ — a 4-bit GPU format that identifies which weights matter most and protects them during quantization, often retaining better quality than plain GPTQ at the same size.
- MLX — Apple's format for Apple Silicon, optimized for unified memory.
Choosing by hardware: GGUF is the universal default; AWQ/GPTQ make sense when running GPU-only stacks like vLLM with 8+ GB of VRAM.
What you actually trade
Honest expectations at 4-bit:
- Everyday chat, writing, summarization: typically indistinguishable from 8-bit.
- Coding: usually fine; trickier logic benefits from Q5/Q6. Coding quality is somewhat more quantization-sensitive than chat.
- Math and multi-step reasoning: the most sensitive area; higher-bit builds or reasoning-mode models help here.
- Fine-tuning: quantized weights are for inference. Fine-tuning methods like QLoRA quantize the frozen base model, then train small adapter layers in higher precision — a different pipeline.
How to choose in practice
The decision is almost always "the best quantization that fits your VRAM with context headroom":
- Compute your model's approximate size at each quantization (roughly 0.55 GB per billion parameters at 4-bit, 1.0 GB/B at 8-bit, 2.0 GB/B at FP16).
- Add 0.5–1 GB of runtime overhead, plus more for long context.
- Pick the highest bit-width whose total fits your card with 1+ GB to spare.
If a Q4 build fits at 8 GB but a Q8 needs 12 GB you don't have, the Q4 build is simply the right answer — not a compromise, the correct configuration for that hardware.
CanIHost bakes this arithmetic into every recommendation: the checker reports, for each model against your profile, the quantization that makes sense and the memory it needs. For the next level of detail, see 4-bit vs 8-bit.