Every compatibility tool quotes tokens per second, but the number means little until you connect it to experience. Is 8 tok/s fine or terrible? Is 40 tok/s worth paying double for? The answer depends entirely on what you are doing — and on a distinction most guides skip: time to first token versus streaming speed.
What a token is
Models process text in tokens — chunks of roughly ¾ of a word in English. "Compatibility" is about two tokens. Speed is measured either in tokens per second (streaming rate) or tokens per second of generation (excluding your prompt's processing time).
The two speeds that matter
Time to first token (TTFT) — how long you wait after pressing enter. For short prompts on local hardware this is usually fast (well under a second); for long prompts (a big document pasted in), the model must first process the input, and this can take noticeably longer.
Streaming rate — tokens per second while the answer flows. This is the number compatibility tools estimate, and the one that defines how "alive" the model feels.
What speed different tasks need
| Workload | Feels good at | Notes |
|---|---|---|
| Interactive chat | 10+ tok/s | Below ~10, waiting becomes noticeable |
| Comfortable chat | 20–40 tok/s | Matches comfortable reading speed |
| Coding autocomplete | 30+ tok/s | Latency sensitivity; instant feels broken otherwise |
| Reasoning models (chain-of-thought) | 20+ tok/s | They generate thousands of "thinking" tokens |
| Batch/overnight processing | 1–5 tok/s | Patience is free when nobody is watching |
Reading speed context: people read comfortably at 200–300 words per minute — roughly 3–5 words per second, or about 4–7 tokens per second. A model streaming at 15+ tok/s is already faster than you can read. Chasing 60 tok/s matters for autocomplete feel and reasoning-model patience, not for reading comprehension.
What determines your tok/s
From CPU vs GPU: streaming speed is bound by memory bandwidth. Everything else is secondary:
- Bandwidth of the memory holding the weights — the dominant factor (VRAM, or unified RAM on Apple Silicon).
- Model size — bigger models stream more bytes per token, so 14B runs slower than 7B on the same card.
- Quantization — smaller weights stream faster; a 4-bit build is often quicker than 8-bit (see 4-bit vs 8-bit).
- Offload share — every layer living in system RAM instead of VRAM cuts speed sharply.
- Context length — longer contexts slow generation somewhat (attention grows with context), plus KV cache bandwidth.
Estimates vs reality
CanIHost's speed figures are estimates produced by llmfit from model size, quantization and your hardware profile — not measured benchmarks on your exact machine. Real-world numbers shift with:
- runtime choice (llama.cpp vs vLLM vs MLX),
- driver and power settings (especially laptops' thermal limits),
- your CPU's ability to feed a hybrid offload setup,
- the specific quantized build.
Treat the estimates as a tier: "30–60 tok/s" means a smooth interactive experience, not a guarantee of exactly 43.7. If precision matters for a purchase decision, verify with the exact runtime and model build you intend to use — our methodology page explains the distinction in full.
The practical takeaway
For chat, 10–20 tok/s is genuinely pleasant; for coding assistance, aim higher; and with reasoning models, buy bandwidth, because their thinking phases generate a lot of tokens. Check your machine's realistic figures with the hardware checker — every result includes an estimated speed line for exactly this purpose.