What LoRA actually does
Full fine-tuning updates every weight, which means holding the weights, their gradients and two optimizer moments per parameter. For a 14B model that is well over 200 GB — datacentre territory.
LoRA freezes the base model and trains a small pair of low-rank matrices alongside each target layer. Only those matrices need gradients and optimizer state, which cuts trainable parameters by two to three orders of magnitude. A 14B LoRA might train 40 million parameters instead of 14 billion.
What QLoRA adds
QLoRA keeps the LoRA idea and additionally quantizes the frozen base to 4-bit NF4, dequantizing on the fly during the forward pass. The adapter still trains in bf16, so gradient quality is preserved; what you lose is a little of the base model's fidelity.
The saving is large: the frozen weights drop to roughly a quarter of their half-precision size, which is usually the difference between a run that fits and one that does not.
| LoRA | QLoRA | Full | |
|---|---|---|---|
| Base weights held as | bf16 | 4-bit NF4 | bf16 |
| Trainable parameters | ~0.1–1% | ~0.1–1% | 100% |
| 14B peak memory (2k seq) | ~34 GB | ~14 GB | ~230 GB |
| Quality vs full FT | Very close | Close | Reference |
| Speed | Faster | Slower per step | Slowest |
Memory figures are ModelLM estimates from model geometry, not measurements.
Choosing a rank
Rank is the adapter's capacity. Rank 8 is enough for a narrow style change. Rank 16 to 32 suits most domain adaptations. Above 64 you are usually overfitting rather than learning, unless the dataset is genuinely large.
Keep alpha at twice the rank. That convention keeps the effective update scale constant as you change rank, so you can adjust one without re-tuning the learning rate.
Target modules matter more than rank
Applying LoRA to the attention projections only (q, k, v, o) is cheaper but consistently weaker than including the MLP projections (gate, up, down). If you have the memory, target all seven.
Sequence length is the real memory lever
Activation memory scales with batch size times sequence length. Halving the sequence length roughly halves activation memory, and is almost always the first thing to cut when a run will not fit. Gradient checkpointing trades compute for a further large reduction and is on by default in ModelLM.
ModelLM applies these rules to your own data.
Upload what you have and the advisor scores all four approaches against it, with the evidence behind the recommendation.
Create a model