The names are not the bit widths
A GGUF quantization stores weights in blocks, each with its own scale and sometimes a minimum. Those per-block extras are real memory, so the effective cost per weight is always above the nominal bit width. Q4_K_M is about 4.83 bits per weight, not 4.
ModelLM uses effective bits per weight everywhere, which is why its size estimates land close to the actual file sizes rather than 20% under.
| Format | Bits/weight | Quality | Use when |
|---|---|---|---|
| F16 | 16.0 | Reference | Memory is abundant, or you are training |
| Q8_0 | 8.5 | Near-lossless | You have headroom and want certainty |
| Q6_K | 6.56 | Near-lossless | A sensible step down from Q8_0 |
| Q5_K_M | 5.67 | High | Q4 fits easily and you want more quality |
| Q4_K_M | 4.83 | Balanced | The default local trade-off |
| Q3_K_M | 3.91 | Degraded | Only to make a model fit at all |
| Q2_K | 3.35 | Degraded | Last resort; expect visible damage |
Bigger model, lower precision usually wins
Given a fixed memory budget, a larger model at Q4_K_M generally beats a smaller model at Q8_0. Capability scales with parameter count faster than quantization damages it — up to a point. Below Q4 that stops being true, and a smaller model at a decent quantization is the better choice.
Reasoning degrades first
Quantization damage is not evenly distributed. Multi-step reasoning and code generation suffer noticeably before conversational quality does, so a Q3 model can sound fine and still fail at the task you need it for.
Do not forget the KV cache
Weights are not the whole story. The key-value cache grows linearly with context length and can rival the weights on a long-context model. A 14B at Q4_K_M is about 9 GB of weights — but add a 32k context and you are carrying another 6 GB.
This is why ModelLM sizes its recommendation against a working context rather than a token or two, and why it will suggest a shorter context before it suggests a degraded quantization.
ModelLM applies these rules to your own data.
Upload what you have and the advisor scores all four approaches against it, with the evidence behind the recommendation.
Create a model