Lazy loaded image7.1 Quantization

7.1 Quantization

7.1.1 Introduction

🧸
Model quantization refers to the process of approximating continuous-valued floating-point weights such as float32 and float16 into a finite set of discrete values such as int8 and int4 with minimal loss of inference precision.
By representing floating-point data with fewer bits, model quantization can reduce model size, thereby lowering memory consumption during inference, and can also increase inference speed on processors that perform low-precision operations faster.
Common data types
  • FP32: 32-bit floating-point, the most common high-precision representation
  • FP16: 16-bit floating-point, smaller numeric range than FP32 but lower memory usage
  • BF16: 16-bit truncated FP32, with an extended exponent range and wider numeric range; commonly used in deep learning
  • INT8: 8-bit integer, only 1/4 the bit-width of FP32, mapping model parameters to corresponding numeric range
  • INT4: 4-bit integer, further reduces bit-width; suitable for extremely resource-constrained scenarios
  • Binary Network: 1-bit binary network where parameters can only be 0 or 1; extremely efficient but incurs significant accuracy loss
In industry, the most commonly used quantization is 8-bit. Quantization below 8 bits is referred to as low-bit quantization. 1-bit is the compression extreme, reducing the model size by 32×, and during inference, efficient XNOR and BitCount bitwise operations can be used for acceleration.

What to Quantize

  • Weight: Quantizing weights is the most common practice. It helps reduce the model size, memory usage, and storage space.
  • Activation: In practice, activations often take up a large portion of memory usage. Quantizing activations not only reduces memory usage significantly but also, when combined with weight quantization, can fully leverage integer computation for performance gains.
      • Activations refer to the output values produced by neurons after receiving input signals and being processed by activation functions.
  • KV Cache: Quantizing the KV cache, which stores previous tokens information for inference, is equally important.
  • Gradients: Compared to the items above, gradient quantization is slightly less common as it’s mainly used during training. During the training of deep learning models, gradients are typically floating-point numbers. Quantizing them can help reduce communication overhead in distributed training and also lower the cost of backward computation.

Quantization Format

Depending on whether the original data range of the quantized data is uniform, quantization methods can be divided into linear quantization and nonlinear quantization. The weights and activations of deep neural networks are often non-uniformly distributed. In theory, using nonlinear quantization results in less precision loss, but it also comes with higher computational complexity during inference. Therefore, linear quantization is commonly used:
notion image
Here, is the floating-point number before quantization, is the integer after quantization, round indicates rounding operation, clip represents clipping operation, is the interval of data quantization, and is the data offset.
When is 0, it is symmetric quantization; when is not 0, it is asymmetric quantization. Symmetric quantization can avoid computing during inference, reducing the computational complexity of inference; asymmetric quantization can determine the maximum and minimum values based on the actual data distribution, which allows better utilization of the quantized data distribution and results in lower loss due to quantization.
notion image
Based on the application scope of the quantization parameters and (i.e., the unit over which they are shared in the network), quantization methods can be classified as follows:
  • Per-layer quantization (per-tensor): Covers the largest scope, simplest approach, using one set of quantization parameters per network layer.
      • notion image
  • Per-channel quantization (per-token & per-channel): Uses each quantization channel within a network layer as a unit, with each channel having its own set of quantization parameters. This offers finer granularity, higher precision, and increased computational complexity. Specifically, per-token refers to activations , with each row having its own quantization scale; per-channel refers to weights , with each column having its own quantization scale. When used together, this is also called vector-wise quantization.
notion image
  • Per-group quantization: Uses groups as the unit, where each group (e.g., rows of activations or columns of weights) shares a set of and . Its granularity lies between per-tensor and vector-wise. When groupsize = 1, per-group quantization is equivalent to per-layer quantization; when groupsize equals the number of convolution channels, it's equivalent to vector-wise quantization.
Additionally, activations and weights can be quantized with different granularities. For activations, there’s also dynamic quantization (where the quantization parameters are computed on-the-fly during inference), whereas static quantization precomputes these parameters ahead of time and applies them during inference.
The LLaMA 3 technical report introduces examples of tensor-wise and row-wise FP8 quantization.
notion image
<ins/>

Quantization Categories

Based on the stage at which quantization is applied, it can be further classified as:
  • Quantization-Aware Training (QAT)
  • Quantization-Aware Fine-tuning (QAF)
  • Post Training Quantization (PTQ)

7.1.2 Quantization-Aware Training (QAT)

First, the model is pre-trained normally, and then "fake quantization nodes" are inserted into the model, meaning weights and activations are quantized and then de-quantized. This introduces quantization error, allowing the model to sense the quantization during training. In this way, quantization is part of the optimization of the training loss (not really included in the loss), thereby also indirectly minimizing the quantization error and letting the model perform well even under quantization.
This approach is suitable for scenarios requiring high accuracy after model compression. By incorporating quantization directly into the training process, the model can better adapt to low-precision representation, enhancing the LLM’s ability to handle accuracy loss caused by quantization.
QAT Method:
  • LLM-QAT: Achieves data-free distillation to quantized model (student) by utilizing the outputs of a pre-trained model (teacher).
      • notion image
      LLM-QAT quantizes not only weights and activations, but also the KV cache. This strategy aims to enhance model robustness under quantization while supporting longer sequences. With this approach, LLM-QAT can compress a LLaMA model down to a model with only 4-bit precision.
      notion image
<ins/>

7.1.3 Quantization-Aware Fine-tuning (QAF)

During the fine-tuning process, LLMs are quantized with the primary goal of ensuring that the quantized LLMs retain strong performance even at lower bit widths. By integrating quantization-aware fine-tuning, the balance between model compression and performance preservation can be effectively achieved.
QAF Method:
  • PEQA: Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization.
      • A novel quantization-aware fine-tuning technique that facilitates further model compression and accelerated inference. It follows a two-stage process.
    • In the first stage, all forward MLPs are quantized using low-bit integer matrices and additional quantization scale vectors to reduce computational overhead.
    • In the second stage, only quantization scale vectors are fine-tuned on downstream tasks, ending up with multiple scale vectors, each for a different task.
      • This approach significantly reduces the model size, thereby decreasing inference latency on edge devices and overall memory usage. At the same time, it enables fast fine-tuning and efficient task switching during inference.
      notion image
  • QLoRA: also a type of QAF method based on LoRA.
    • 4-bit NormalFloat (NF4) Quantization
        • QLoRA introduces a 4-bit quantization data type called NF4, specifically designed for weights with a normal distribution. Compared to traditional integer or floating-point quantization methods, NF4 retains more information and can maintain model performance at lower bit widths.
    • Double Quantization
        • To further reduce memory usage, QLoRA also quantizes the scaling factors used during quantization. This double quantization strategy significantly reduces the memory needed to store these factors, enabling more compact model deployment.
    • Paged Optimizer
        • QLoRA uses a paged optimizer to manage memory usage during training, especially when handling long sequences or large batches of data. By paging model parameters and optimizer states in and out of memory, it avoids memory spikes and enables training large models with low GPU memory.
        QLoRA makes it possible to fine-tune a 65B LLM on a single 48GB GPU.
      notion image
<ins/>

7.1.4 Post Training Quantization (PTQ)

🔍
QAT introduces fake quantization nodes, which significantly increases the computational cost during fine-tuning, especially for extremely large-scale LLMs. Current research on LLM quantization mostly focuses on post-training quantization (i.e. quantization after training), such as LLM.int8(), SmoothQuant, GPTQ. For weight quantization, it can be calculated in advance before inference, making it easy to complete. But when it comes to activations (i.e., outputs from each layer), they are unknown beforehand and depend on actual inference input.
After LLM training is completed, its parameters can be quantized with just a small amount of calibration data, making it suitable for scenarios with high deployment efficiency demands and limited resources. The main goal is to reduce LLM storage and computation complexity, without needing to modify or retrain the LLM architecture. PTQ’s main advantage lies in its simplicity and efficiency, but there may be some accuracy loss during the quantization process.
PTQ Methods:
  • LLM.int8(): Some outlier values exist in activation , and their absolute values are significantly larger. These outliers are concentrated in a small number of features, referred to as outlier features.
    • In the diagram below, the yellow group represents outliers. Whether using per-token or per-channel quantization, these outliers greatly affect the quantization result. The solution in LLM.int8() is: since only a small number of features contain outliers, isolate those features and compute them separately, then quantize the rest.
    • Specifically, a mixed-precision quantization approach is used, first splitting the matrix, quantizing the majority of weights and activations to 8-bit using vector-wise method, and keeping dimensions of the outlier features in 16-bit for higher-precision matrix multiplication.
      • notion image
  • SmoothQuant: In response to outliers in activations, SmoothQuant offers a different solution compared to LLM.int8(). Since activation quantization is generally harder than weight quantization, SmoothQuant introduces a smoothing coefficient to balance the difficulty between the two, making quantization more manageable for both sides:
      • notion image
      After smoothing, the activations and weights can be quantized. Weights use per-tensor quantization, while activations adopt different versions of quantization with varying granularities.
      In the formula above, when the outliers in is large, then the will be large as well, which can scale down the and scale up the . For most models, α = 0.5 is a good start to evenly split the quantization difficulty
      notion image
  • GPTQ: Both LLM.int8() and SmoothQuant are based on round-to-nearest (RTN) quantization, i.e., rounding to the nearest quantization level. In contrast, GPTQ frames quantization as an optimization problem, aiming to find the optimal quantized weights layer by layer. For a given block, all parameters within it are quantized in sequence. After each parameter is quantized, the rest within the block are adjusted accordingly to compensate for the accuracy loss caused by quantization. Additionally, GPTQ requires a calibration dataset.
      • notion image
  • AWQ: For LLMs, not all weights contribute equally to performance. By retaining just 1% of the most significant weights, AWQ can greatly reduce quantization error. Building on this, AWQ introduces an activation-aware method that evaluates the importance of each weight based on its corresponding activation magnitude, this plays a key role when dealing with important features. Specifically, it uses a channel-wise scaling technique to determine the optimal scaling factor, thereby minimizing quantization error across all weights during quantization.
      • notion image
Prev
6.6 PEFT
Next
7.2 Pruning
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant