The three components
Weights: parameters times effective bits per weight, divided by eight. A 14B at Q4_K_M is 14.8e9 × 4.83 ÷ 8 ≈ 8.9 GB.
KV cache: 2 × layers × kv_heads × head_dim × context × dtype bytes. Grouped-query attention, which almost every modern model uses, cuts this by the ratio of query heads to key-value heads — often 4× or 8×.
Overhead: the CUDA or Metal context, compute buffers and allocator slack. Roughly half a gigabyte plus a bit more for larger models.
You never get the whole card
The OS and desktop compositor take a slice. ModelLM reserves about 0.6 GB on a headless Linux box and 1 GB where a desktop is running, and budgets 75% of unified memory on Apple Silicon to match the default wired-memory limit.
What each budget buys
Assuming Q4_K_M and a modest context window:
| VRAM | Comfortable | Tight | Fine-tuning |
|---|---|---|---|
| 8 GB | 3–7B | 9B | 3B QLoRA |
| 12 GB | 7–9B | 14B | 7B QLoRA |
| 16 GB | 9–14B | 24B | 7–8B QLoRA |
| 24 GB | 14–24B | 32B | 14B QLoRA |
| 32 GB | 24–32B | 49B | 14–24B QLoRA |
| 48 GB | 32–49B | 70B | 32B QLoRA |
| 128 GB unified | 70B+ | — | 32B+ via MLX |
ModelLM estimates. Check a specific model against your machine on its model page.
Training needs far more than inference
A model you can comfortably run is not necessarily one you can train. Fine-tuning adds the adapter weights, their gradients, two fp32 Adam moments per trainable parameter, and activation memory that scales with batch size times sequence length.
In practice, expect to need roughly 1.5–2× the inference footprint for a QLoRA run at a modest sequence length, and considerably more without gradient checkpointing.
ModelLM applies these rules to your own data.
Upload what you have and the advisor scores all four approaches against it, with the evidence behind the recommendation.
Create a model