Lazy loaded image6.2 Latency Analysis for Inference

6.2 Latency Analysis for Inference

6.2.1 Inference Latency

Inference Mechanism

  • Traditional inference method:
      • Sequential token generation, no parallelism possible
  • Two ways to accelerate:
    • Matrix-vector multiplication: a large matrix (e.g., 8192x8192) multiplied by a vector to get another vector
    • Attention computation, using KV-cache for inference
  • Bottleneck analysis: main sources of floating-point operations
    • Matrix-vector multiplication requires one multiply and one addition operation per matrix element (2 FLOPs)
    • Attention (with KV-cache) requires one multiply and one addition operation for each element in Key and Value

Latency Estimation

Here is a reference for the data in bytes and parameter numbers; just don’t get confused when doing the calculations.
Category
Abbreviation
Full Form
Value
Data
b
bit
1 bit
ㅤ
B
byte
8 bits
ㅤ
KB
kilobyte
1,024 bytes
ㅤ
MB
megabyte
1,024 KB = 1,048,576 bytes
ㅤ
GB
gigabyte
1,024 MB = 1,073,741,824 bytes
ㅤ
TB
terabyte
1,024 GB = 1,099,511,627,776 bytes
Parameter Number
K
thousand
1,000
ㅤ
M
million
1,000,000
ㅤ
B
billion
1,000,000,000
ㅤ
T
trillion
1,000,000,000,000
  • Calculate the amount of data for computing one token
    • Total Parameters Data: If the model (let’s say a 7B LLM) uses FP16 as the matrix element type, then generating one token requires loading the following amount of data to the Arithmetic Logic Unit (ALU):
        • Although the matrices during token generation can be reused theoretically, the device cache usually only has a few tens of MB, and matrices cannot be saved in the on-chip cache. Therefore, it’s easy to conclude that the speed won't be faster than the bandwidth. Attention calculations need to retrieve the corresponding KV cache of previous tokens during each token computation, so the amount of data fetched depends on how many previous tokens the model looks at when generating a new token. This includes:
      • System Prompt (typically the system's hidden instructions and may not be visible to users)
      • User Prompt
      • Previous model outputs and possibly multiple rounds of conversation history
    • KV-cache Data: Still, suppose we have a 7B-sized LLM (let’s say it has 32 layers), and its KV-cache stores each Key and Value as eight 128-dimensional vectors at every layer (i.e., 8 heads, and each is a 128-length vector). So in total (add both K and V), each token corresponds to elements; if the KV-cache uses FP16, then for P tokens, we need to read roughly of data.
        • For example, if token number is 1000, reading from the KV-cache requires around 130MB of data. Compared to the 14GB of matrix computation, this 130MB is almost negligible.
  • Compute Latency:
      • For example, on an NVIDIA RTX 4090 (1008 GB/s), reading 14GB takes about ~13.9ms, so we can expect the tokens in the very beginning, each token roughly takes 13.9ms to read (ignoring the minor impact of KV-cache here).
      If using 8bit precision, it would require reading 7GB, which would take ~7.0ms. This setting is the theoretical minimum time to generate a token.
🍒
Normal autoregressive model prediction time can be roughly approximated by: , where b is the initial cost of generating the first token (prefilling stage), and k is the continuous cost for each token afterward (generating/decoding/sampling stage), which grows with the number of tokens.
More specifically, b depends on factors like the model size and prompt length, potentially being more than ten times k or even more. This is related to the KV-cache mechanism.
This explains why everyone praises the effectiveness of CoT (Chain of Thought), but at the same time, everyone is not willing to use it (from the perspective of users, and of course, at the current stage of o1/o3/o4 and R1, they need a lot of CoT data for improvement). Also, we can observe that the model's generation speed and the number of tokens generated are positively correlated, and CoT often introduces a lot more generated tokens.
<ins/>

Tokens Per Second (TPS) Calculation

💬
How to calculate TPS?
When deploying LLMs, the number of tokens generated per second, TPS, is a key metric to measure inference performance:
The total latency consists of two stages:
  • Time To First Token (TTFT): The delay from input to generating the first token, mainly influenced by the prompt length and model parameters, corresponding to the Prefilling stage.
  • Time Per Output Token (TPOT): The average time needed to generate each subsequent token, corresponding to the Decoding stage.
The total latency can be expressed as:
Thus, TPS can be expressed as:
📌
TPS Estimation Method
  • Determine model parameters: X (unit: B)
  • Calculate FLOPs for the Prefilling stage:
      • Matrix multiplication typically requires two operations (multiplication & addition) per element, hence 2 FLOPs, which holds the same below
  • Calculate FLOPs for the Decoding stage:
  • Estimate TTFT and TPOT:
      • Divide the above FLOPs by the actual GPU compute power (considering utilization rate, e.g., A100 312 TFLOPs, 60% utilization, actual compute is 312 × 60% = 187 TFLOPs) to obtain TTFT and TPOT.
  • Calculate TPS:
      • Use the estimated TTFT and TPOT to calculate TPS.
Note: Actual TPS calculations may vary based on model implementation, hardware architecture (such as GPU parallel computation capabilities), and optimization techniques (such as Flash Attention).
<ins/>

6.2.2 First Token Latency Optimization

First Token Latency

⚙
In the LLM inference process, generating the first token is a computation-intensive task. This stage is referred to as the prefilling phase or context phase.
The time taken to generate the first token is related to the amount of computation required to process and input the prompt into the model, and is directly proportional to the prompt length.
For example, when the prompt length is long, we can consider techniques like FlashAttention; the time to generate the first token is roughly linearly related to the prompt length.
Note that High Bandwidth Memory (HBM) is typically found only on certain professional-grade GPUs, such as the NVIDIA A100, H100, etc., which are specifically designed for AI training and inference tasks.
But consumer-grade GPUs like GeForce RTX 4090 use GDDR6X memory, which is still a high-performance type of memory, but it is not HBM.
The memory bandwidth plays a crucial role during the inference of subsequent tokens. Unlike the first token, where computational FLOPs dominate the bottleneck, generating subsequent tokens typically becomes memory bandwidth-bound.

System Prompt Optimization

🍒
System Prompt Caching
The basic idea is to perform a one-time computation for the System Prompt and cache its corresponding Key and Value (for example, store them in GPU memory).
When the LLM encounters the same (or even partially the same) System Prompt again during inference, it can directly use the cached Key and Value, thus avoiding redundant computation of the System Prompt.
First Form: Prefix Sharing, suitable for scenarios where the Prompt = System Prompt + User Prompt, with the System Prompt acting as the Prefix.
Second Form: Prompt Cache, a more advanced method, involves caching the Key and Value corresponding to the entire input Prompt, not just limited to a shared prefix.
🍒
Specifically, for multi-turn conversation scenarios and AI Agent applications based on LLMs, the second method mentioned above (i.e. Prompt Cache) can support Session Prompt Cache.
In a multi-turn conversation session, the input Prompt to the LLM carries the full dialogue history, involving a lot of redundant computation. By using Session Prompt Cache, unnecessary repeated computation can be significantly reduced, saving GPU resources and improving response speed and user experience during conversations.
Prev
6.1 GPU Usage Analysis
Next
6.3 FlashAttention
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant