Running a model locally used to mean wrestling with Python environments and CUDA versions. Today it is genuinely easy — the tooling went through the same revolution as local media players. This is the complete path from zero to a working local assistant, on whatever hardware you have.
Step 1: know what your hardware can host
Before downloading anything (models are 2–40 GB), know your target. Three numbers decide everything: VRAM, RAM, and CPU threads. The hardware checker turns those into a ranked list of compatible models with the right quantization — one minute of checking prevents a 5 GB download that won't load.
Quick reference for what to aim at:
- No GPU, 8–16 GB RAM → 3B models at Q4 (see CPU-only guide)
- 6–8 GB VRAM → 7B–8B at Q4/Q5
- 12–16 GB VRAM → 14B at Q4/Q5
- 24 GB VRAM → 32B at Q4/Q5
Step 2: pick your tool (one of three)
Ollama — the recommended default. Dead-simple install, command-line friendly, serves an OpenAI-compatible API for editor integrations.
# Install (Windows/macOS/Linux), then:
ollama run llama3.2 # small, runs anywhere
ollama run qwen2.5-coder # for coding help
That is genuinely the whole process — Ollama picks an appropriate quantization for your machine automatically.
LM Studio — the GUI path. Point-and-click model discovery, chat interface, per-model settings (context, GPU layers), and it shows VRAM usage live. Best if you prefer not to touch a terminal.
llama.cpp — the power-user option. The engine underneath both tools above. Direct use gives you per-layer GPU offload control, quantized KV cache, and every quantization format. Worth it once you outgrow the conveniences.
Step 3: pick your model file
With Ollama, model names map to managed downloads. With LM Studio or llama.cpp, you download GGUF files (see what is GGUF) — choose the quantization by the VRAM math: at 4-bit, ~0.55 GB per billion parameters plus ~1 GB overhead, and remember context memory (see context guide).
Sensible first models:
| Goal | Model | Why |
|---|---|---|
| General chat | Llama 3.1 8B / Qwen3 8B | Best quality per GB |
| Coding | Qwen2.5 Coder 7B | Strong local coding model |
| Low memory | Llama 3.2 3B / Qwen3 4B | Runs on almost anything |
| Hard reasoning | DeepSeek-R1-Distill 14B | Chain-of-thought on consumer GPUs |
Step 3: first run
With Ollama on a machine with 8 GB of VRAM:
ollama run llama3.1
>>> Explain quantization in one paragraph.
You'll see tokens stream in at 20–50 tok/s. That's it — a capable model, fully offline, on your machine. Everything else is configuration.
Step 4: the two settings that matter
Once the default works, only a handful of knobs change the experience:
- Context length — the memory-per-session dial. Keep 4K–8K on smaller cards; raise only with VRAM headroom.
- GPU layers (llama.cpp/LM Studio) — how much of the model lives in VRAM versus RAM. Full residency = full speed; offload degrades quickly (see CPU vs GPU).
Most other knobs (temperature, top-p) change writing style, not performance — leave defaults until you have a reason.
Step 5: connect it to things (optional)
The real payoff of local models is integrations, and Ollama makes this trivial because it speaks the OpenAI API format at http://localhost:11434:
- Editor plugins (Continue, various IDE extensions) for local coding assistance,
- Local RAG setups that search your own documents,
- Automation tools that route prompts to your machine instead of a paid API.
Nothing leaves your machine in any of these setups — that is the point.
Troubleshooting quick reference
| Symptom | Likely cause | Fix |
|---|---|---|
| Very slow generation | Model offloading to RAM | Smaller quant or model; more GPU layers |
| Out-of-memory at load | Weights + overhead > VRAM | Lower quantization; reduce context |
| Speed decays over long chats | KV cache filling | Reduce context or enable cache quantization |
| Model answers nonsense | Wrong variant (base not instruct) | Download the "-instruct" / chat build |
Where this ends
Fifteen minutes in, you have a private, offline assistant tuned to your hardware. The natural next steps: browse the model directory to see what else your machine can host, and check the guides on speed to tune the experience. When you upgrade hardware later, re-run the checker — the same machine profile unlocks a different catalogue every time VRAM crosses a tier.