AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
6.4 PagedAttention
6.4 PagedAttention
6.4.1 Mechanism
Leverage the concept of virtual memory and make good use of fragmented memory.
The performance bottleneck in large language model inference mainly comes from memory.
- First, a large amount of memory is needed to cache the
KandVduring the auto-regressive decoding process. For example, in LLaMA-13B, a single sequence input can require 1.7GB of memory.
- Second, memory usage is dynamic and depends on the input sequence length. Due to fragmentation and pre-allocation overhead, current systems waste 60%-80% of memory.
PagedAttention is inspired by virtual memory and paging at the operating system level. It allows caching the
KV in non-contiguous memory blocks while ensuring logically contiguous access.Specifically, PagedAttention splits each sequence's KV cache into multiple blocks, each block contains a fixed-length of tokens, allowing the attention computation to efficiently locate and retrieve the relevant blocks.
Each fixed-length block can be viewed as a page in virtual memory; tokens can be seen as bytes, and sequences as lines of text. This way, one can use a block table to map logically continuous blocks to physically non-contiguous memory blocks, and the physical blocks can be allocated based on generated tokens.
6.4.2 Inference Workflows
- For single request
- For multiple requests
After all sequences are split into blocks, only the final block may potentially waste memory (in practice, the memory waste is less than 4%). The benefits of high memory utilization are obvious: the system can input more sequences per batch, improving GPU utilization and significantly increasing throughput.
Another advantage of PagedAttention is efficient memory sharing. For example, during parallel sampling, a single prompt may need to generate multiple output sequences. In such cases, both computation and memory for that prompt can be shared across the output sequences.
Block tables naturally enable memory sharing. Similar to how processes share physical pages, in PagedAttention, sequences in different logical blocks can point to the same physical blocks, enabling shared blocks. To ensure safe sharing, PagedAttention tracks reference counts to physical blocks and implements a Copy-on-Write mechanism. Memory sharing reduces memory usage by 55%, significantly lowering the memory footprint of sampling algorithms while boosting throughput by up to 2.2×.
Copy-on-Write: When multiple processes or tasks share the same block of memory (such as a piece of data), they share the memory in a read-only manner. Only when one task needs to modify that memory does the system actually create a copy for it, to prevent affecting other tasks.
<ins/>
6.4.3 Usages in Prefilling and Decoding
A request is sent to the model, asking it to continue a prompt and produce three different responses. This is called parallel sampling.
- Prefilling stage: The vLLM receives sample A1 and sample A2. Based on the prompt content, it allocates logical and physical blocks for them. In this stage, A1 and A2 share block0 and block1 (because they have the same initial prompt).
- Decoding stage: When the tokens generated by the two samples start to differ, copy-on-write is triggered — a new memory block is created in physical memory. At this point, only the logical block1 of A2 is mapped to physical block1, and its ref count is reduced by 1; physical block3 is only mapped to the logical block1 of A1, and its ref count is set to 1.
Below is the model inference of prefilling and decoding stages; it works just as it was, all virtual pages and mappings are managed by the PagedAttention, which is integrated into vLLM.
Prev
6.3 FlashAttention
Next
6.5 Packing for Efficient Training
Loading...