GPU buying advice for gaming rarely translates to local AI. Frames-per-second logic leads you to the wrong card, because inference optimizes for two different things: VRAM capacity (what models can run at all) and memory bandwidth (how fast they run). Here is a framework that puts those in the right order.
Priority 1: VRAM capacity
Capacity is the gate. A model either fits or it doesn't — and if it doesn't, no amount of speed saves the experience. At 4-bit quantization, budget roughly 0.6 GB per billion parameters plus ~1 GB of overhead:
- 8 GB → 7B–8B comfortably, 12B–14B squeezed
- 12 GB → 14B at Q4/Q5 comfortably
- 16 GB → 14B at Q6/Q8, 32B at Q3
- 24 GB → 32B at Q4/Q5, 70B at Q2 (with compromises)
Decide which model class you actually want to run in a year, not just today — VRAM is the one spec you cannot upgrade later. Most satisfied local-AI users wish they had bought more capacity, not more speed.
Priority 2: memory bandwidth
Speed tracks bandwidth almost linearly (see tokens per second). Within a capacity tier, the card with more bandwidth produces tokens faster. Notable bandwidth landmarks: ~270 GB/s (mid cards like the 4060), ~500 GB/s (4070 class), ~1 TB/s (4090 class).
Practical implication: a 12 GB card with 360 GB/s (used RTX 3060) and a 12 GB card with 504 GB/s (4070) host the same models — the second one just does it roughly 40% faster.
Priority 3: ecosystem (CUDA vs ROCm vs Vulkan)
Covered fully in CUDA vs ROCm: NVIDIA is the zero-friction default; AMD offers more VRAM per euro with a modest tax in setup effort; Vulkan keeps everything portable. If you value "it just works," weigh that into the price.
Cards by budget tier
| Tier | Card | VRAM | Best for |
|---|---|---|---|
| Budget (used) | RTX 3060 12 GB | 12 | The classic value entry; 13B Q4 |
| Budget AMD | RX 7800 XT | 16 | 14B class on Linux/Vulkan |
| Mid new | RTX 4060 | 8 | 7B–8B with speed |
| Mid used | RTX 3090 | 24 | 24 GB at used prices — the value king |
| Upper mid | RTX 4070 | 12 | Balanced: capacity + real bandwidth |
| Enthusiast | RTX 4090 | 24 | Fastest consumer inference |
| Value 24 GB | RX 7900 XTX | 24 | 24 GB cheaper, ROCm ecosystem |
| Apple path | M-series (32–64 GB unified) | 32–64 | Big models, quiet, no discrete GPU |
The two cards that punch above their price for AI specifically: the used RTX 3060 12 GB (cheapest real capacity) and the used RTX 3090 (24 GB at half the price of new). The cards that don't make sense for AI: the RTX 4060 Ti 8 GB (same VRAM as 4060, more money) and any card under 6 GB.
Two honest warnings
- Don't buy for "AI tops." Marketing tensor-core numbers matter little for single-stream local inference; bandwidth and capacity dominate. A "gaming" card at the right VRAM/bandwidth beats an "AI-branded" one with less.
- Mind your platform. A 24 GB card wants a real PSU (750–850 W+) and case cooling; a Mac path avoids all of it but costs more per GB. Total-system thinking beats card-thinking.
The 10-second version
Pick the model class you want in two years, buy the cheapest card (new or used) whose VRAM hosts that class at 4-bit with headroom, prefer more bandwidth within that tier, and let the ecosystem preference break ties. Then verify: run your configuration through the checker before spending — it grades the entire model catalogue against the exact card you're considering, so you know what your money buys.