Efficiency and cost reduction

Today's generative models can contain billions of parameters. Training, adapting, and serving them requires memory to store their weights and activations, and computing power to carry out the operations.

There is no single way to reduce this cost. In this chapter we will look at three complementary strategies:

1. Distillation: training a smaller model to reproduce the behavior of a more capable one.
2. Quantization: representing weights and, in some cases, activations with fewer bits.
3. LoRA: adapting a pretrained model by updating only a small number of parameters.

Knowledge distillation

Knowledge distillation^{[hinton2015distilling]} is a compression technique in which a large model, the teacher, supervises the training of a smaller model, the student.

The goal is to approximate the function the teacher has learned. If the transfer works, the student retains a significant part of its quality but with less memory, lower latency, and a lower inference cost. This approach is useful for producing lightweight models that can be deployed on mobile devices, edge devices, or services with heavy request volumes.

Hard labels and soft targets

In a conventional classification problem, a hard label indicates only the correct class. For an image of a cat, for example, the target vector might be:

$$y = [1, 0, 0]$$

for the classes cat, leopard, and dog. The teacher, on the other hand, can produce a distribution such as $[0.80, 0.15, 0.05]$. These probabilities contain additional information: they show that, for the teacher, a cat is more similar to a leopard than to a dog. These relationships between classes are usually referred to as dark knowledge.

Soft targets are obtained by applying softmax to the logits (the model's unnormalized outputs) with a temperature $\tau$:

$$p_i^{(\tau)} = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}$$

When $\tau=1$, the usual softmax is obtained. A higher temperature flattens the distribution and makes small probabilities visible, better exposing the similarities learned by the teacher.

Loss function

Let $p_T^{(\tau)}$ and $p_S^{(\tau)}$ be the smoothed distributions of the teacher and the student. A common formulation combines two objectives:

$$\mathcal{L}_{\text{total}} = \alpha\, \tau^2 D_{\mathrm{KL}}\!\left(p_T^{(\tau)} \,\|\, p_S^{(\tau)}\right) + (1-\alpha)\, \mathrm{CE}\!\left(y, p_S^{(1)}\right)$$

1. The distillation loss forces the student to approximate the teacher's distribution. It can be expressed using Kullback–Leibler divergence or cross-entropy.
2. The supervised loss compares the student's prediction with the true label $y$.

The hyperparameter $\alpha$ controls the relative weight of both terms. The factor $\tau^2$ compensates for the change in gradient scale produced by the temperature. It is not always essential to have human labels: unlabeled data or synthetic examples can also be used, training solely with the teacher's supervision.

Distillation in generative models

In an LLM, knowledge can be distilled at different levels:

• Logits: the student approximates the probability distribution over the next token. This approach requires being able to query the teacher's outputs and properly align vocabularies.
• Sequences: the teacher generates responses that become training data for the student. This is especially practical when only generated text is accessible.
• Intermediate representations: besides the output, hidden states or attention maps are aligned when the architectures allow it.

This is an example of sequence-level distillation: DeepSeek-R1 generates and filters solutions, and smaller models based on Qwen and Llama learn from those samples. They are not directly copying its logits.

Distillation has an upfront cost, the teacher must be run and the student must be trained, and it can transmit its errors or biases. Its usefulness depends on the accumulated savings during deployment offsetting that cost, and on the loss of quality being acceptable.

Quantization

Quantization is a technique for reducing computational and memory costs during inference by representing weights and activations with low-precision data types, such as 8-bit integers (int8) instead of the usual floating-point float32 or float16.

Reducing the number of bits means that the resulting model requires less storage in memory, consumes less energy, and operations such as matrix multiplication can be performed much faster.

The most widely used standard method is uniform affine quantization, which maps real values onto a grid of integers through a linear transformation:

• Affine (asymmetric): allows the real value 0.0 to be aligned with an arbitrary integer $z_0$ (zero point), which avoids distortions when the data is not symmetrically centered around zero (as happens after ReLU-type activations).
• Uniform: the distance between two consecutive integers always represents the same fixed interval in the real-valued space (the scale factor s).

Mapping a real value $x$ to an integer $q$ is done as follows:

$q = \operatorname{clip}\!\left(\operatorname{round}\!\left(\frac{x}{s}\right) + z_0,\ q_{\min},\ q_{\max}\right)$

During inference, an approximation of $\hat{x}$ is reconstructed:

$\hat{x} = s(q - z_0)$

Example - INT8 (range [0, 255])

Suppose the weights of a layer range between $x_{\min} = -1.0$ and $x_{\max} = 3.0$:

1. Scale (s): $s = \frac{x_{\max} - x_{\min}}{q_{\max} - q_{\min}} = \frac{3.0 - (-1.0)}{255 - 0} \approx 0.0157$
2. Zero point ($z_0$): $z_0 = \operatorname{round}\left(- \frac{x_{\min}}{s}\right) = \operatorname{round}\left(\frac{1.0}{0.0157}\right) = 64$
3. Quantize a value ($x = 1.2$): $q = \operatorname{clip}\!\left(\operatorname{round}\!\left(\frac{1.2}{0.0157}\right) + 64,\ 0,\ 255\right) = \operatorname{clip}(76 + 64,\ 0,\ 255) = 140$
4. Reconstruct ($\hat{x}$): $\hat{x} = 0.0157 \cdot (140 - 64) = 1.1932$

The discrepancy between 1.2 and 1.1932 is the quantization error, generated by the rounding operation. To reduce this margin of error, the granularity at which $s$ and $z_0$ are computed is adjusted:

• Per tensor: all values in a layer share a single pair $(s, z_0)$. It is very memory-efficient, but vulnerable to outliers.
• Per channel: each output channel has its own scale, balancing precision and computational cost.
• Per group: weights are divided into small blocks (e.g. 32 to 128 elements) with an independent scale, a technique fundamental to 4-bit LLMs (such as AWQ or GPTQ).

Which part of the model gets quantized?

1. Weights only (weight-only): considerably reduces memory and is common in LLMs. Activations are kept at higher precision.
2. Weights and activations: can speed up computation further, but activations are harder to quantize because they change with each input and can contain outliers.
3. KV cache: in autoregressive models with long contexts, quantizing the cache reduces the memory that grows with the number of tokens processed.

The theoretical storage savings are easy to estimate. For example, for a 7B model:

PTQ and QAT

There are two main points at which quantization can be introduced:

• Post-Training Quantization (PTQ): an already trained model is quantized. It is fast and may only require a small calibration set to estimate scales. Methods such as GPTQ^{[frantar2022gptq]} correct part of the error when quantizing LLMs weight by weight or block by block.
• Quantization-Aware Training (QAT): quantization operations are simulated during training so the model learns to tolerate the error^{[jacob2017quantization]}. It tends to better preserve quality at aggressive precisions, but it requires retraining.

Quantizing always introduces a trade-off between size, speed, and quality. Layers do not all have the same sensitivity, and it is sometimes worth keeping some of them at higher precision. Moreover, a model that is four times smaller will not necessarily be four times faster: latency can be dominated by sequential generation, data transfer, or the lack of native support for the chosen format.

Low-Rank Adaptation (LoRA)

When training large generative models, adjusting all the parameters can be very costly, especially when the only goal is to adapt the model to a specific task.

In full fine-tuning, every weight matrix $W_0$ must have all of its parameters updated:

$$W_{\text{updated}} = W_0 + \Delta W$$

If $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, training $\Delta W$ requires up to $d_{\text{out}}d_{\text{in}}$ parameters.

Low-Rank Adaptation (LoRA)^[hu2021lora] is an efficient fine-tuning method, or PEFT (Parameter-Efficient Fine-Tuning), that seeks to reduce this burden using a technique called low-rank decomposition.

The central idea is to freeze the original weight matrix $W_0$ (some or all of the model's matrices) and, instead, decompose the update $\Delta W$ into the product of two smaller matrices:

$$\Delta W = \frac{\alpha}{r}BA, \qquad A \in \mathbb{R}^{r \times d_{\text{in}}}, \qquad B \in \mathbb{R}^{d_{\text{out}} \times r}$$

where $\alpha$ is a hyperparameter that scales the contribution of the low-rank update (often set relative to $r$, hence the ratio $\alpha/r$).

This way, instead of training $d_{\text{out}}d_{\text{in}}$ values, only $r(d_{\text{in}}+d_{\text{out}})$ need to be trained. This significantly reduces the number of parameters that need to be adjusted, making the process more efficient.

Example. Suppose we want to adapt a square weight matrix $W_0$ of size $4096 \times 4096$ (a typical LLM layer), with $16\,777\,216$ parameters. Full fine-tuning of that matrix would update the $16.8$ M parameters. With $r=8$, LoRA trains:

$$8(4096+4096)=65\,536$$

This represents approximately $0.39\%$ of that matrix's parameters.

There is a trade-off in choosing the rank. A rank that is too small may fail to capture the target task well. A very large rank improves capacity, but also increases cost and the risk of overfitting.

In general, there are two options:

• Merge the weights: compute the product $\Delta W = \frac{\alpha}{r}BA$ and add it directly to the original matrix: $W_{\text{final}} = W_0 + \Delta W$. The model is saved with $W_{\text{final}}$ as a single standard file. This eliminates any latency or compute overhead at inference, but modularity is lost.
• Keep the adapter decoupled: only the file with the weights $A$ and $B$ is saved. At inference, the base model $W_0$ is loaded into memory and the input travels through two paths in parallel. On one hand, the base path, where $h_{\text{base}} = Wx$ is computed, and on the other, the LoRA path, where the adjustment is computed as $B \cdot (A \cdot x)$. Both paths are then summed to obtain the final response.

To reduce memory usage during inference, LoRA can be combined with quantization. A popular combination is QLoRA^{[dettmers2023qlora]}: it keeps the base model frozen and quantized, for example to 4 bits, and propagates the gradient through higher-precision LoRA adapters. This reduces both the memory of the base model and the memory associated with training.

Contributions

• Efficiency: allows models to be adapted without an exhaustive retraining, saving time and computational resources.
• Simplicity: for each task, only an adapter needs to be saved instead of a full copy of the model.

How to choose a technique

These techniques can be chained together. For example, one can distill a large model, adapt the student with LoRA, and quantize the result for deployment.