Every model card lists two memory costs, but most guides only talk about one. Weight memory is the headline number; context memory is the quiet second cost that grows as you increase the context slider — and it is the reason models that "fit" suddenly stop fitting when you feed them long documents.
Where context memory comes from
While generating text, the model must remember every token processed so far. It does so in the KV cache: for each token, the attention layers store compressed "keys" and "values" that later tokens attend to. The cache grows linearly with context length — twice the context, twice the cache.
The size depends on the model's architecture (layer count, hidden size, attention-head design):
| Model class | KV cache at 8K context (typical, FP16 cache) | At 32K |
|---|---|---|
| 3B | ~0.4 GB | ~1.6 GB |
| 7B–8B | ~0.7–1 GB | ~3–4 GB |
| 14B | ~1.3 GB | ~5 GB |
| 32B | ~3 GB | ~11 GB |
Runtimes vary: some quantize the cache (halving these numbers), some optimize attention; treat the table as an order-of-magnitude guide. The lesson stands regardless: at long contexts, the cache stops being a footnote and becomes a model-sized cost.
The classic failure mode
Here is how people get bitten, step by step:
- They verify a 7B Q4 model fits their 8 GB card with a 4K context. ✓ (weights ~4.5 GB, cache ~0.3 GB, overhead ~0.7 GB — fits).
- They raise context to 32K for a long-document task.
- The cache grows by ~3 GB and the model no longer fits; the runtime starts offloading layers to RAM.
- Speed collapses — and it looks like "the GPU is too slow," when in fact the configuration changed.
The model did not get slower. The context got bigger. Planning for context prevents the confusion entirely.
How to plan for context
Budget weights + cache + overhead for your realistic context:
- Chat and everyday use: 4K–8K context is plenty. Cache is a minor cost; plan ~0.5–1 GB.
- Long documents, codebases, RAG: 16K–32K. Add 2–5 GB on a 7B–14B model — often the difference between fitting and not.
- Reasoning models: their thinking chains consume context fast. A 14B reasoning model may use 8K+ tokens of thought per hard question; budget context for the thinking, not just your prompt.
Practical rule: for every doubling of context beyond 8K, expect several GB of additional memory on mid-size models — unless your runtime supports quantized KV cache, in which case roughly halve it.
Practical settings
- On 6–8 GB cards: keep context at 4K–8K for 7B models; you buy speed and stability.
- On 12–16 GB cards: 8K–16K is comfortable at Q4/Q5 for 14B models; 32K possible for smaller ones.
- On 24 GB+: 32K is realistic for 14B–32B classes; watch the first minutes of a long session for slowdowns as the cache fills.
Many runtimes display current cache usage live — if generation speed decays over a long conversation, the growing cache is usually the cause, and reducing context (or enabling cache quantization) fixes it.
Where this fits in the bigger picture
Context is the third variable in the compatibility equation, after model size and quantization. Compatibility tools that ignore it produce optimistic numbers; CanIHost's recommendations treat context as a first-class input, and the checker explains, for each model, how much memory remains for context on your specific hardware.