Tech AI News - 테카이

LoRA — if it's fine-tuning, why isn't the whole model retrained?

· 5 min read

This post was translated from the Korean original by AI.한국어 원문 읽기 →

Hearing "we fine-tuned it with LoRA" sounds like the model was retrained from scratch. In reality, the original weights stay frozen, and only two tiny matrices inserted into each layer get newly trained. Miss that distinction and you're left stuck on questions like "parameters dropped by thousands of times, so why didn't GPU memory drop by nearly as much?"


Background — why this term matters now

In articles and tool documentation about fine-tuning (further training a pretrained model for a specific task), the name LoRA (Low-Rank Adaptation) shows up as if it were a given. As Hugging Face's PEFT (Parameter-Efficient Fine-Tuning) library became the standard tool for applying LoRA, cases of adapting large language models for specific uses on a single consumer GPU have multiplied (Hugging Face LLM Course, Hugging Face PEFT official docs).

The problem is that the word "fine-tuning" itself gives the impression that the entire model gets retrained. LoRA is closer to the opposite. The original weights (the numerical values the model learned) are left untouched, and only small correction matrices inserted into each layer get trained.

Key data & current state

MetricValueSource · date
Reduction in trainable parameters for GPT-3 (175B)~10,000xHu et al., "LoRA: Low-Rank Adaptation of Large Language Models" paper, 2021
Reduction in GPU memory requirement for GPT-3 (175B)~3xSame paper, 2021
Estimated trainable parameters for GPT-3~18 millionIBM Think, January 2025
Estimated GPU memory savings for GPT-3~2/3IBM Think, January 2025
Recommended LoRA rank (r) range4–32 (depending on task difficulty)Hugging Face LLM Course

Interpretation: The gap between the drop in parameters (on the order of ten thousand times) and the drop in memory (around 3x) shows the core of this technique. Even though the trainable parameters shrink dramatically, the full original weights still have to sit in memory for inference and the forward pass. What actually shrinks is the memory needed only for training — the optimizer state and gradients.

In-depth analysis

1) What exactly is LoRA

Ordinary fine-tuning trains the entire change (ΔW) to be added to a weight matrix W, at the same size as the original. LoRA approximates this ΔW as the product of two smaller matrices, A and B. The number that sets the width of these two matrices is the rank (usually written r) — the smaller r is, the fewer numbers (parameters) there are to train (Hugging Face PEFT official docs).

During training, the original weight W stays frozen while only A and B get updated through backpropagation. At initialization, B starts at zero so that the product of the two matrices is zero — a device meant to leave the original model's behavior untouched at the start of training (PEFT official docs). Once training finishes, A and B can be merged into the original W to form a single set of weights, so inference incurs no extra latency (PEFT official docs).

2) A common misconception — why it reads that way

The most common misconception is thinking, because of the word "fine-tuning," that the entire model's parameters get retrained. In reality only a tiny fraction of the original gets trained — for GPT-3, the trainable portion shrinks to about one ten-thousandth of the total (Hu et al., 2021 paper).

The second misconception is assuming GPU memory usage drops by the same ratio as the parameter count. As shown in the key data above, memory savings land around 3x at most. That's because the original weights still have to sit in memory for inference and gradient computation to be possible.

The third misconception is thinking the trained LoRA adapter (the bundle of A and B matrices) alone is enough to run the model. The adapter only holds the "difference" from the original weights, so unless it's merged, it only makes sense loaded together with the base model.

3) The criteria that actually matter in practice

The first value to decide when applying LoRA is the rank r. The Hugging Face LLM Course suggests typically choosing somewhere between 4 and 32 depending on task difficulty. A lower r reduces trainable parameters and storage but may hurt expressiveness; a higher r increases expressiveness at the cost of more compute.

Another value to set alongside it is the scaling factor lora_alpha, which the PEFT official docs recommend defaulting to twice the rank. Which layers to apply LoRA to (target_modules) is also an important choice — in transformers, it's typically applied only to the attention projection layers to preserve parameter efficiency. Widening the scope to something like "all-linear" increases adaptability but also increases memory usage accordingly (PEFT official docs).

4) When it's useful and when it falls short

LoRA is especially advantageous in environments with limited GPU memory or storage, or in situations where you need to serve multiple task- or customer-specific adapters by swapping them on top of a single base model. Since the adapter files themselves are small, it's easy to store and deploy many of them at once.

Conversely, for tasks that need to inject a large amount of entirely new knowledge into the model, or that require fundamentally changing how the model behaves, the expressiveness limits of a low-rank approximation can become a bottleneck. The official docs only describe LoRA's performance as "comparable to" full fine-tuning, not as "always equal to" it (PEFT official docs). Analysis: the simpler the task and the smaller the dataset, the smaller LoRA's performance loss tends to be, but that explanation relies on the premise that the model is already over-parameterized, so it doesn't necessarily generalize to every task.

In practice — 5 principles for reading LoRA correctly

  1. Don't be fooled by the word "fine-tuning." The original weights stay frozen — what actually gets trained is just two small matrices slotted into each layer.
  2. Don't read the drop in parameters and the drop in memory as the same number. Trainable parameters can shrink by thousands to tens of thousands of times, while GPU memory savings usually land around 3x, because the original weights still have to sit in memory.
  3. The rank (r) is the knob between performance and cost. Smaller is lighter but may lose expressiveness; more complex tasks may need a larger r.
  4. The adapter alone doesn't make the model work. Unless it's been merged, it must be loaded together with the base model at deployment.
  5. If the task requires changing the model's fundamental knowledge or behavior, LoRA alone may not be enough. Consider a larger rank, a wider set of target_modules, or full fine-tuning instead.

References

  • #AI Terms
  • #LoRA
  • #Fine-tuning
  • #PEFT
  • #Low-Rank Adaptation