If you ask five people how much VRAM you need for local AI, you will get five answers — usually because each person is answering for a different model size. The honest answer is that VRAM requirements scale with three things: model size, quantization, and context length. Once you understand how those three interact, you can answer the question for any model in about ten seconds of mental math.
The short answer by model size
| Model class | 4-bit memory (est.) | Comfortable VRAM | Notes |
|---|---|---|---|
| 1B–3B | 1–2.5 GB | 4 GB | Runs on almost anything, including CPU |
| 7B–8B | 4.5–5.5 GB | 8 GB | The local-AI sweet spot |
| 12B–14B | 7–10 GB | 12–16 GB | Strong quality step-up |
| 27B–32B | 15–20 GB | 16–24 GB | Enthusiast tier |
| 70B | 24–41 GB | 48 GB / dual 24 GB | Multi-GPU or high-memory Mac |
Read the table as estimates, not guarantees: runtime overhead and context memory sit on top of the weight memory, and they vary by runtime.
The math you can actually use
Model memory at a given quantization is approximately:
memory ≈ parameter count × bits-per-weight ÷ 8
A 7B model at 4-bit weights: 7 × 4 ÷ 8 ≈ 3.5 GB of raw weights. Real builds land a bit higher because quantization formats carry small metadata and the runtime needs overhead for the KV cache and compute buffers — a practical rule is to add 0.5–2 GB on top, plus more for long context.
That is why a 7B model at Q4 is described as "about 4.5–5 GB": 3.5 GB of weights, and the rest is overhead that grows with context length.
Why quantization is the real lever
The same 7B model needs roughly 16 GB at FP16, 8 GB at 8-bit, and 4–5 GB at 4-bit. Same model, same quality ballpark for most tasks — four times less memory. This is why almost every local-AI guide, including ours, steers beginners toward 4-bit quantization first: it is the difference between a model that fits on mainstream hardware and one that does not.
The trade-off is real but smaller than most people fear. Modern 4-bit formats (GGUF Q4_K_M, GPTQ Int4, AWQ 4-bit) are tuned so that everyday chat and coding remain close to full-precision quality. The degradation shows up at the edges: unusual languages, dense math, or long chains of complex reasoning.
Context is the second wall people hit
Weights are only half the story. The KV cache — the memory that lets the model attend to everything said so far — grows with context length and can silently eat gigabytes:
- A 7B model with an 8K context might use 0.5–1 GB of KV cache.
- The same model at 32K context can need 2–4 GB depending on the runtime's attention optimizations.
This is the classic failure mode: the model fits with a 4K context, you raise the context slider to 32K for long documents, and suddenly you are out of VRAM. Plan for context from the start: buy the card for weights plus your realistic context, not weights alone.
Practical recommendations
- 4 GB laptop (shared or dedicated): 3B models at 4-bit. A 7B at Q4 is possible but will fight for memory on shared-memory machines.
- 6 GB (RTX 4050 class): 7B at Q4/Q5 with a moderate context. This is the entry point where local AI starts feeling good.
- 8 GB (RTX 4060 class): the sweet spot — 7B–8B at Q5/Q6, 12B–14B at tight quantization, long-context headroom for smaller models.
- 12 GB (RTX 4070 / 3060 12 GB class): 14B at Q4/Q5 entirely in VRAM; coding models at Q8.
- 16 GB (4080 / 4060 Ti 16 GB class): 14B at Q6/Q8, 32B at Q3.
- 24 GB (4090 / 7900 XTX class): 32B at Q4/Q5, 70B at Q2 (with compromises).
- 48 GB+ (dual GPU or Apple Silicon Max): the 70B class becomes genuinely usable.
What you don't need to worry about
- VRAM brand or generation matters less than capacity and bandwidth. A used 12 GB card with 360 GB/s out-hosts a fast 8 GB card for many workloads, because capacity decides what runs and bandwidth decides how fast.
- You don't need VRAM equal to model size in FP16. Almost nobody runs FP16 locally; the quantized ecosystem exists precisely so that mainstream cards can host capable models.
The one-line takeaway
Budget roughly 0.55–0.6 GB of VRAM per billion parameters at 4-bit, add 0.5–1 GB of overhead, and add more for context beyond 8K. If that number fits your card with a little headroom, the model runs well. When you want the answer for your exact machine, the hardware checker does this math for an entire model catalogue at once.
Want to go deeper? See VRAM vs RAM for why system memory only partly compensates, or what quantization means for the full trade-off picture.