SmolLM2 1.7B Instruct
Hugging Face
Fully open training data and recipe. A good base when you want to understand exactly what the model saw.
Runs with CPU offload
- Weights
- 960 MB
- KV cache
- 1.5 GB
- Overhead
- 480 MB
- CPU-only inference. Expect a few tokens per second at best.
- Small models remain usable on CPU for non-interactive work.
- KV cache at 8,192 tokens is 1.5 GB — shorten the context to reclaim memory.
- Completely open training pipeline
- Apache 2.0
- Very cheap to fine-tune
- 8k context
- Limited capability without fine-tuning
smollm2:1.7bRunning SmolLM2 1.7B Instruct on CPU only — 32 GB DDR5
| Quantization | Quality | Weights | KV cache | Total | ~tok/s | Fit |
|---|---|---|---|---|---|---|
| F16 | lossless | 3.2 GB | 1.5 GB | 5.2 GB | 19 | Runs with CPU offload |
| Q8_0 | near-lossless | 1.7 GB | 1.5 GB | 3.7 GB | 36 | Runs with CPU offload |
| Q6_K | near-lossless | 1.3 GB | 1.5 GB | 3.3 GB | 46 | Runs with CPU offload |
| Q5_K_M | high | 1.1 GB | 1.5 GB | 3.1 GB | 53 | Runs with CPU offload |
| Q4_K_MPick | balanced | 960 MB | 1.5 GB | 2.9 GB | 63 | Runs with CPU offload |
| Q3_K_M | degraded | 770 MB | 1.5 GB | 2.8 GB | 77 | Runs with CPU offload |
KV cache is sized at 8,192 tokens. Longer contexts cost proportionally more — the recommendation above reserves room for a working context.
Local fine-tuning on this machine
- Base weights
- 890 MB
- Optimizer
- 270 MB
- Activations
- 410 MB
- Peak
- 2.8 GB
- Local fine-tuning needs a GPU or Apple Silicon. CPU training is not practical.
- Base weights
- 3.2 GB
- Optimizer
- 270 MB
- Activations
- 410 MB
- Peak
- 5.1 GB
- Local fine-tuning needs a GPU or Apple Silicon. CPU training is not practical.
Where this model runs
VRAM 32 GB · F16 · 5.2 GB
VRAM 24 GB · F16 · 5.2 GB
VRAM 16 GB · F16 · 5.2 GB
VRAM 16 GB · F16 · 5.2 GB
VRAM 24 GB · F16 · 5.2 GB
VRAM 12 GB · F16 · 5.2 GB
VRAM 12 GB · F16 · 5.2 GB
VRAM 16 GB · F16 · 5.2 GB
VRAM 48 GB · F16 · 5.2 GB
VRAM 80 GB · F16 · 5.2 GB
Unified 128 GB · F16 · 5.2 GB
Unified 48 GB · F16 · 5.2 GB
Unified 24 GB · F16 · 5.2 GB
Unified 192 GB · F16 · 5.2 GB
Unified 16 GB · F16 · 5.2 GB
VRAM 24 GB · F16 · 5.2 GB
VRAM 16 GB · F16 · 5.2 GB
VRAM 0 MB · Q4_K_M · 2.9 GB
VRAM 0 MB · Q4_K_M · 2.9 GB
Qwen2.5 7B Instruct
7.6B · Apache 2.0
The default starting point for local work on 8–12 GB cards. Strong instruction following and reliable tool-call formatting for its size.
Qwen2.5 Coder 7B Instruct
7.6B · Apache 2.0
The practical local copilot. Supports fill-in-the-middle, so it works as an inline completion model rather than only a chat assistant.
Llama 3.1 8B Instruct
8B · Llama 3.1 Community License
The most widely supported open model there is. If a tool, adapter or tutorial exists, it was written for this one first.
Llama 3.2 3B Instruct
3.2B · Llama 3.2 Community License
Small enough for a laptop CPU or a 6 GB card, and still coherent. A sensible target for edge deployment.
Catalogue figures come from each model’s published card. ModelLM has not independently measured them.