Lazy loaded image6.1 GPU Usage Analysis

🚗 LLM, officially called Large Language Models, have seen their parameter counts grow massively from 0.5B, 1B, 400B, to even 600B. To handle the training and inference of models with such enormous parameter sizes, some works have made optimizations and improvements in this area.

6.1 GPU Usage Analysis

6.1.1 Model GPU Usage

When training or inferencing a model, GPU memory allocation is partly used by the AI framework and partly reserved for the system. The total GPU memory usage can be checked via commands. For example, on NVIDIA GPUs, you can use the nvidia-smi command to display the memory usage of each process:
📌
The GPU memory usage includes: Model parameters (parameter), Optimizer states (optimizer_state), Activations (activation), Gradients (gradient), Input data (input), Temporary variables (temporary), Autograd information (autograd_detail), and Unknown data (unknown). From the user's perspective, these can be categorized into:
  • Estimatable Data: Model parameters (parameter), Optimizer states (optimizer_state), Activations (activation), Gradients (gradient), Input data (input)
  • Unamed Data: Temporary variables (temporary), Unknown data (unknown)
  • Other (Framework-specific): Autograd information (autograd_detail)
🤔
Why is there sometimes a large gap between the estimated GPU memory usage and the actual measured usage?
One possible reason is that the amount of unknown data is large. In other words, the proportion of estimatable data in GPU memory is relatively small, while the proportion of unestimatable data is large. This can cause a significant difference between the estimated and actual values (with errors can exceed 30%).
For example, you might estimate GPU memory usage to be 50 GB, but actual measurements show it reaching 75 GB, where the exceeded amount might include, for example, certain temporary buffers, intermediate results, and internally allocated data by underlying frameworks (such as CUDA or cuDNN). These are usually dynamic and can vary, making their size difficult to estimate in advance.
<ins/>

6.1.2 GPU Usage During Training

📌
Memory Consumption During Training (Estimatable Portion) mainly includes: Model parameters + Optimizer status + Gradient values + Activation values
Based on the value changes, memory consumption can be classified into static and dynamic allocations. During training, GPU memory for model parameters and optimizer status generally remain unchanged and are categorized as static; the allocation for activation values and gradient values changes during training, so they are categorized as dynamic.
Static Allocation Analysis
  • Model Memory
      • The GPU memory occupied by a model mainly depends on the number of parameters and the data type. Common data types include fp32, fp16/bf16, as well as int8, fp8, etc.
      The memory usage can be calculated as ModelMem = TypeSize * Params based on different data types, using the formulas below (unit: GB):
    • fp32 = 4 * params / (1024 * 1024 * 1024)
    • fp16/bf16 = 2 * params / (1024 * 1024 * 1024)
    • fp8/int8 = 1 * params / (1024 * 1024 * 1024)
      • Regarding model checkpoint size calculation :
      When saving a checkpoint, only the parameters are stored, not the optimizer states (if we don’t want to retrain or fine-tune the model). For example, with a 1B (1 billion) parameter model using fp32 storage, the disk size would be:
      Memory = 4 * 1 * 10⁹ / (1024 * 1024 * 1024) ≈ 3.725GB
      Since 1TB typically equals 3.725GB of storage space, for easier estimation we often consider 1B ≈ 4GB, which can simplify storage planning. For instance, Llama-13B requires around 52GB of storage. The DeepSeek-R1 has 671B parameters, but its file size on the Huggingface is around 688 GB, which is because it’s stored in fp8.
      Note: If mixed precision training is used, the saved checkpoint is still stored as fp32 unless otherwise specified.
  • Optimizer States
      • In LLM training, a common optimizer is Adam. Each parameter typically needs a Momentum and a Variance (Second moment) state. In mixed-precision training, Adam also maintains a model parameter copy.
      The Adam optimizer state memory can be calculated as (unit: GB):
      OptMem = (4 + 4 + 4) * Params / (1024 * 1024 * 1024)
      Here, (4 + 4 + 4) refers to:
    • Model copy: 4 Bytes
    • Momentum: 4 Bytes
    • Variance: 4 Bytes
      • If 8-bit optimization is applied, the calculation changes to:
      8BitOptMem = (4 + 1 + 1) * Params / (1024 * 1024 * 1024)
      Where:
    • Model copy: 4 Bytes
    • Momentum: 1 Byte
    • Variance: 1 Byte
Dynamic Allocation Analysis
  • Activation Memory
      • The size of activation memory is related to model parameters, recomputation, and parallelism strategies. Here we refer to the formula provided by Megatron to explain the calculation of activation memory usage (case w/o parallelism).
      Activation memory consumption (Unit: GB):
      where:
    • s: sequence length, the number of tokens
    • b: microbatch size
    • h: hidden dimension size
    • a: number of attention heads
    • L: number of transformer layers
    • γ: scaling factor, for unit of GB, γ = 1 / (1024 * 1024 * 1024)
  • Gradient Memory
      • The memory required for gradients is consistent with the model parameters, depending on the data type used. The calculation formulas are as follows (unit: GB):
    • For fp32 model parameters:
        • fp32 = 4 * params / (1024 * 1024 * 1024)
    • For fp16 or bf16 models:
        • fp16/bf16 = 2 * params / (1024 * 1024 * 1024)
<ins/>

6.1.3 GPU Usage During Inference

💡
The memory composition for inference is simpler compared to training. There is a straightforward estimation formula for total memory usage:
When is the 1.2× estimate inaccurate?
  • When the sequence length is very long (e.g., long-context inference with tens of thousands of tokens), the KV cache grows significantly and memory usage can far exceed 1.2×.
  • When the batch size is large, the increase in intermediate activations can cause inference memory to expand.
  • When techniques like offloading (to cpu) are enabled during inference, memory usage can also vary.

6.1.4 Memory Optimization

🧸
Due to the exponential growth in model parameters, which far exceeds the physical memory capacity of a single GPU, LLM training inevitably requires memory optimization. Memory optimization focuses on reducing the computational overhead of the model or expanding the physical memory. But because the physical memory is hard to expand, the main ways to obtain extra memory are:
  • Time-for-space tradeoff: e.g., recomputation
  • Data shifting: e.g., multi-device parallelism, offload
Among them, time-for-space usually incurs computation overhead and latency; data shifting mainly incurs I/O overhead and may cause delays, possibly affecting throughput.
🌲
The memory optimization process generally starts from the model algorithm layer down to the system layer. A suggested optimization path could be:
Multi-device parallelism -> Operator/Data type -> Framework optimization -> Memory management -> Low-level API
  1. Multi-device parallelism:
      2. At this stage, this one is most frequently used and generally does not affect computational accuracy. Techniques like TP/PP/DP/Zero and recomputation can be used to lower memory consumption.
      Drawback: these methods may increase bandwidth overhead.
  1. Operator optimization:
      2. Select operators that consume less memory but with the same precision.
      Drawback: developing new operators is time-consuming.
  1. Data type optimization:
      2. Use lower precision to replace higher precision, such as replacing fp32 with fp16, or even using int8/int4.
      Drawback: this usually affects training convergence speed/stability/performance.
  1. Framework optimization:
      2. Within frameworks (like PyTorch), some data are intermediate results produced by the framework. These can be cleaned up to save memory.
      Drawback: relatively high development cost.
  1. Memory management:
      2. Reduce memory fragments introduced by frameworks’ memory management
      Drawback: currently few available methods.
  1. Low-level API:
      2. In GPU driver libs or CUDA ops libs, the way APIs consume memory differs. By using better low-level APIs that consume less memory, memory consumption can be reduced, such as FlashAttention. Some default operations will introduce additional system memory consumption; you may have to pay attention to if there is a higher version of the API that’s been improved.
Prev
5.5 Enterprise-level RAG Practice (II)
Next
6.2 Latency Analysis for Inference
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant