Download almost any local model today and the file is named something like model-q4_k_m.gguf. GGUF is the format of llama.cpp — and by extension of Ollama, LM Studio and a large share of the local-AI world. Knowing what it is helps you read those filenames and pick the right build.
A single file with everything inside
GGUF (GGML Universal File) bundles everything a runtime needs into one self-contained file:
- the quantized weights in whatever bit-width the build uses,
- the tokenizer and vocabulary,
- architecture metadata (model type, context training length, parameter details),
- optional per-quantization calibration data.
Because everything travels together, running a model means pointing a compatible runtime at one file. No Python environment, no weights scattered across folders, no conversion steps — which is why the CPU-plus-GPU universal ecosystems standardized on it.
Reading a GGUF filename
Take a real example: qwen2.5-7b-instruct-q4_k_m.gguf
qwen2.5-7b-instruct— the model and variant (instruction-tuned, i.e. chat-ready).q4_k_m— the quantization: 4-bit, K-quant family, "medium" mix. The K-quant suffix encodes which block sizes and mixing strategies the build uses.
Common levels you'll see:
| Label | Bits (approx) | Character |
|---|---|---|
| Q2_K | ~2.5 | Extreme compression, visible quality loss |
| Q3_K_M | ~3.5 | Tight; for squeezing big models into small cards |
| Q4_K_M | ~4.5 | The standard default |
| Q5_K_M | ~5.5 | Small quality edge over Q4 when memory allows |
| Q6_K | ~6.5 | High fidelity at moderate size |
| Q8_0 | ~8.5 | Near-lossless; the "if it fits, use it" tier |
The _S/_M/_L variants trade a little quality for a little size within the same bit level; M is the usual choice.
Why it became the default
Three properties made GGUF the lingua franca of local AI:
- Hardware neutrality. The same file runs CPU-only, on NVIDIA (CUDA), AMD (ROCm/Vulkan) and Apple (Metal). One format covers a friend's gaming PC, your laptop, and a Mac.
- Per-layer offload. llama.cpp can split a GGUF model between VRAM and RAM — the graceful-degradation mechanism behind Ollama's "runs on almost anything" reputation.
- Quantization maturity. The K-quants and importance-matrix variants (IQ quants) are well-tested across thousands of community-quantized models.
GGUF vs the GPU-server formats
GGUF is not the only format — it is the most portable one:
- GPTQ / AWQ target GPU-only inference stacks (vLLM, ExLlama). They can be faster for throughput-heavy serving on NVIDIA hardware, at the cost of portability.
- MLX is Apple's optimized format for Apple Silicon, often a bit faster than GGUF on Macs.
Practical guidance: start with GGUF; move to AWQ/GPTQ only when you are specifically serving with vLLM-class stacks and want that ecosystem's optimizations.
How to pick the right file
- Match the model to your task (coder variant for code, general for chat).
- Match the quantization to your memory: use the VRAM guide — at 4-bit, budget ~0.55 GB per billion parameters plus ~1 GB overhead.
- Prefer widely-used quants (Q4_K_M, Q5_K_M, Q8_0) from reputable quantizers.
When a model is quantized well, the format disappears — you get a file, you run it, it works. When you want to know which GGUF fits your machine before downloading 5 GB, the checker maps each model to the quantization your hardware can host.