Local AI suitability
The RTX 4090 remains the reference consumer GPU for local AI. With 24 GB of VRAM and over 1 TB/s of bandwidth, it runs 32B models at high-quality quantization entirely in VRAM, supports very long contexts on 14B models, and delivers the fastest single-GPU inference in its class.
Top-tier for single-GPU setups. Everything in the 7B–32B range runs locally at usable speeds, and multimodal/vision models fit comfortably. Two 4090s (or a 4090 plus offload) extend into the 70B class at 4-bit.
Recommended model sizes
- 14B (FP16)
- 32B (Q4/Q5)
- 70B (Q2/Q3 with offload)
Recommended quantizations
- Q4_K_M / Q5_K_M (GGUF)
- GPTQ Int4
- AWQ 4-bit
Limitations
- Power draw (450 W) and heat require serious cooling and a 850 W+ PSU
- Llama 3.3 70B still needs multi-GPU or aggressive quantization on one card
How it compares
For pure local-AI capability per card, the 4090 outclasses everything in the consumer Ada lineup. Used RTX 3090s offer half the bandwidth and VRAM at a much lower price — a common budget path to 24 GB.
Compatible models
Model pages below include per-quantization memory estimates and FAQs.
FAQ
Can a 4090 run a 70B model?
Not fully at usable quality in 24 GB — a Q2 70B is around 24–26 GB and quality suffers. Practical options are two 4090s at Q3/Q4, or one card with heavy CPU offload at reduced speed.
How fast is a 14B model on a 4090?
Q4 14B models commonly reach 60–90 tok/s; Q8 lands around 30–50 tok/s. Throughput scales with memory bandwidth, which is the 4090's strength.
Is the 4090 overkill for 7B models?
For pure fit, yes — but the speed benefit is real: 7B Q6 models run well above 100 tok/s, making interactive coding assistants feel instant. Budget buyers get more value from smaller cards.
What about fine-tuning?
With QLoRA, a 4090 can fine-tune 7B–13B models. It is a popular entry point for local training experiments, though larger models require cloud GPUs.