✨ Awesome-Anything

Awesome-AI-Researchs

Awesome-GitHub-Repos

Awesome-AI-Examples

🖼️ Digital Image Processing

DIP I Image Demosaicing and Histogram Manipulation

DIP II Image Denoising

DIP III Edge Detection

DIP IV Digital Half-toning

DIP V Geometric Image Modification

🍃 LLM Components

1.1 Tokenization

1.2 Embedding

1.3 Attention

1.4 FFN, Residual Addition, and LN

1.5 Positional Encoding

1.6 Architecture & Decoding Policy

🌱 LLM Pre-training

2.1 Pre-training Definition

2.2 Pre-training Data

2.3 Pre-training Workflow

2.4 Continued Pre-training (CPT)

2.5 Pre-training Evaluation

☘️ LLM Post-Training

3.1 Supervised Fine-Tuning (SFT)

3.2 Reinforcement Learning

3.3 RLHF: Reinforcement Learning from Human Feedback

3.4 Optimizing PPO-Based Algorithms

3.5 Direct Preference Optimization (DPO)

🍀 LLM Popular Models

🪴 LLM Applications

5.1 Prompt Engineering

5.2 LLM-based Agents

5.3 Retrieval-Augmented Generation (RAG)

5.4 Enterprise-level RAG Practice (I)

5.5 Enterprise-level RAG Practice (II)

🌿 LLM Optimization

6.1 GPU Usage Analysis

6.2 Latency Analysis for Inference

6.3 FlashAttention

6.4 PagedAttention

6.5 Packing for Efficient Training

6.6 PEFT

🌾 LLM Compression

7.1 Quantization

7.2 Pruning

7.3 Knowledge Distillation

🌵 LLM Hands-on Practice

8.1 nanoGPT: Training a GPT from Scratch

8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example

8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis

8.4 OpenRLHF: Reward Model Training and Code Analysis

8.5 OpenRLHF: PPO Tutorial and Code Analysis

🌴 LLM Must-read Papers

9.1 Technical Reports Explained

9.2 MedLLM

9.3 o1 Techinical Paper

9.4 Prompting

9.5 LLM-as-Judge

🌳 LLM Q&A

10.1 Why GRPO training loss starts at 0

10.2 VLM-R1 Reproduction Results

10.3 KL(P||Q) or KL(Q||P)?

10.4 Left Padding or Right Padding?

10.5 What is QK-Norm?

10.6 Why LLM training rarely adopts Dropout?

10.7 Dropout and DropPath

🐝 VLM Image Encoders

11.0 Introduction

11.1 Vision Transformer

📝 MISC.

🤔 Large Language Models Pass the Turing Test.

OpenAI ChatGPT Progression

⏳ Claude Sonnet 4 Test with Sandtris

🔍 What DeepSeek-R1's Surge Teaches Us About SEO

💰 CivitAI’s Payment Issue

🎓 arXivSub, an Academia Assistant

6.6 PEFT

🦋

Parameter-Efficient Fine-Tuning (PEFT) only fine-tunes a small portion of or a small amount of additional model parameters, keeping most of the pre-trained parameters fixed. This greatly reduces computational and storage costs, while the most advanced PEFT techniques can achieve performance comparable to full fine-tuning.

Advantages:

Significantly reduces memory usage and hardware resource requirements

Faster training and shorter iteration times

Lower storage costs, as shared weight parameters can be reused across different tasks

May even lead to better model performance by alleviating issues related to overfitting

However, full fine-tuning is still used if compute resources are not a concern

6.6.1 Prompt Tuning

Concept

💡

Prompt tuning is actually a relatively broad concept, referring to fine-tuning a model by adjusting prompt templates. Different tasks can define their own prompts. However, instead of adding prompts directly into the input layer, prompt tokens are inserted. By fixing pre-trained model parameters, prompt tuning only allows each task to prepend k learnable tokens to the input text, without modifying the model’s encoding layers or adding task-specific output layers.

For the modeling side, take T5 as an example: considering all tasks as text-to-text tasks, it is represented as . Prompt Tuning adds a sequence of special tokens before the input , and the language model generates , i.e., . The indicates pre-trained model parameters, which are frozen during training; are task-specific prompt parameters, which are optimized during training.

For the training side, Prompt Tuning introduces the Prompt Ensembling method, where multiple prompts are trained for the same task. They possess their own trainable parameters and share the same pre-trained LLM.

🧠 Model Tuning VS Prompt Tuning

Model Tuning: Requires training a separate model for each downstream task.

Prompt Tuning: Only requires storing a small amount of task-specific prompts for each task, and can perform mixed-task inference using the original shared pre-trained model by routing different task inputs to corresponding fine-tuned prompts.

Hard Prompt and Soft Prompt

🔆

Prompt Tuning can be divided into Hard Prompt and Soft Prompt:

Hard Prompt (Discrete) refers to directly concatenating manually designed tokens with the original text. These tokens are explicit and remain fixed throughout training. Here, remaining fixed means that the embeddings of these tokens are not updated during training, and no additional parameters are introduced.

Manually Constructed Template
Heuristic-based Template: Templates built using rules or heuristic methods
Generation: Templates generated based on training data for the task (usually with a few-shot setup)

Soft Prompt (Continuous) refers to letting the model continuously optimize the prompt parameters during training based on specific contexts and tasks. The core idea is that fixed discrete prompts cannot effectively participate in the model's training loop and are prone to underfitting, whereas turning the prompt into trainable parameters allows the model to search for suitable pseudo tokens in the embedding space during training, helping the model to better adapt and generalize.

Word Embedding: Templates represented by learnable dense vectors (word embeddings) instead of explicit text. These embeddings are fine-tuned during training.
Pseudo Token: Templates not represented by actual words’ embeddings, but as additional trainable parameters, i.e., introducing pseudo-tokens.

Prefix tuning, P-tuning, and P-tuning v2 all fall under continuous prompt construction, where manually designed tokens from hard prompts are replaced by virtual tokens that can be optimized.

<ins/>

6.6.2 P-Tuning

Concept

⭐

P-Tuning converts prompts into learnable embedding layers, and uses MLP+LSTM to process embeddings. By using a small number of continuous embedding parameters as prompts, it allows GPT to better adapt to natural language understanding (NLU) tasks.

Note: P-tuning is limited to the embedding layer, meaning it only adds at the input layer, not at every layer. Additionally, the position of the virtual tokens doesn't have to be at the beginning—the insertion position can be chosen.

An example: The capital of Britain is [MASK]. Here, Britain is the input context , and [MASK] is the target to be predicted. For a continuous prompt, the template can be expressed as:

where represents a prompt token (continuous embedding tokens). After passing through the embedding layer, it becomes:

where are the continuous learnable embeddings, while and are the fixed embeddings of the input and target token respectively. Besides, we can also combine the discrete and continuous prompts to provide more context and achieve better performance. In this case, the embedding would be like:

This paper designs a prompt encoder, which consists of a Bi-LSTM and a two-layer feed-forward neural network. This encoder encodes the prompt embedding sequence before passing it into the language model. The specific formula is as follows:

Contributions

Considering the contextual relationships between tokens: For example, capital and Britain have a sequential relationship, but the bidirectional attention mechanism for the prompts can’t fully capture this. Therefore, a prompt encoder is introduced, where the LSTM can naturally capture the sequential relationships.

Specifying the context: If the model can’t accurately generate the answer based on pure pseudo tokens during training, it’s hard to optimize the model. Therefore, it’s better to use a few words from the input sentence that semantically represent the target labels (e.g., capital and Britain).

Reparameterization: P-tuning uses a prompt encoder to process these pseudo embeddings to encourage output embeddings to be optimized to farther locations in the embedding space. After training, the embeddings in the embedding table can be replaced with the new features from the prompt encoder, so we can discard the encoder. In other words, the prompt encoder is used during training, but not during inference.

Hybrid Prompt: Combines continuous prompts with discrete token prompts. The added capital can be seen as a single discrete token too, but we can add more custom prompts for specific tasks.

Fewer trainable parameters: P-tuning only updates the parameters of the virtual token part in the embedding layer.

🤔

Why should embeddings first pass through MLP or LSTM before inputting to GPT?

After being trained, GPT/LM’s vocabulary is already highly dispersed. If virtual tokens with random embeddings are chosen, it’s easy to fall into local optima. So, the authors use a prompt encoder (i.e., an LSTM + MLP) to map these virtual embeddings to a better representation for better results. Also, this RNN (e.g., LSTM) module can capture sequential relationships in the prompts.

<ins/>

6.6.3 Prefix Tuning

Essentially, prefix tuning replaces the real tokens in traditional manually designed prompts with differentiable virtual tokens. It involves constructing a segment of task-relevant continuous virtual tokens before the input token as the prefix, but this prefix is composed of freely tunable virtual tokens instead of real tokens. Then, during training, only the parameters of the prefix part are updated, while the rest of the parameters in the LLM remain fixed.

💡

Details

Constructing different prefixes for different model architectures

For autoregressive architectures:

Prefix is added in front of the sentence, resulting in z = [PREFIX; x; y]. A suitable prefix can guide LM generation (e.g., GPT-3's in-context learning).

For encoder-decoder architectures:

Both the encoder and decoder add prefixes. The result is z = [PREFIX; x; PREFIX; y]. The encoder-side prefix guides the encoding of part of the input, and the decoder-side prefix guides the generation of subsequent tokens.

Encoding methods for virtual tokens

To avoid unstable training and performance degradation caused by directly training the prefix parameters, an MLP structure is added to process the prefix.

In addition, experiments have shown that only tuning the embedding layer leads to limited expressiveness. Therefore, additional MLP is added to map the soft prompts at every layer.

Comparison between Prompt Tuning and Prefix Tuning

Prompt Tuning: Adds trainable embeddings to the input embedding

Prefix Tuning: Adds trainable embeddings to all transformer blocks

<ins/>

6.6.4 P-Tuning V2

💡

Problems with Prompt Tuning and P-Tuning Methods

Lack of model parameter scalability and task generalization

Poor scalability: The Prompt Tuning paper shows that when the model has over 10 billion parameters, prompt tuning can compete with full fine-tuning. However, for smaller models (100M–1B), the performance of prompt tuning and full fine-tuning differs significantly, greatly limiting its applicability.
Limited generalization: Although Prompt Tuning and P-Tuning have shown advantages in some NLU benchmark tests, their effectiveness on sequence labeling tasks is still unproven.

Lack of deep optimization for input prompts:

In P-Tuning, prompt tokens are inserted directly into the input embedding sequence and passed through the first layer of the transformer.

The number of learnable tokens is limited due to sequence length constraints.
The input embedding has limited direct impact on the model's predictions.

P-Tuning V2 method proposes inserting prompts at every transformer layer, similar to prefix tuning, rather than just at the input layer. This brings several benefits:

More learnable parameters (P-Tuning v2 increases learnable parameters from 0.01% to 0.1%–3%), while still far fewer than full fine-tuning

Deeply inserted prompts can have a stronger impact on model prediction at different layers

⭐

Details of P-Tuning V2

This method is similar to Prefix Tuning. You can think of it as adapting the Prefix Tuning technique, originally designed for text generation, to NLU tasks, and then applying a few improvements:

Removal of the reparameterization encoder:

Earlier methods relied on encoders to improve training speed and robustness (e.g., MLPs in Prefix Tuning, LSTMs in P-Tuning). In P-Tuning v2, the authors discovered that these encoders contribute little, especially for smaller models, and can lead to performance degradation.

Using different prompt lengths for different tasks:

The optimal prompt length depends on the task. Hyperparameter searching of prompt lengths is essential. This aligns with findings from Prefix Tuning, confirming that prompt length tuning is important to achieving optimal results.

Introducing multi-task learning:

Prompts are first pre-trained on a variety of tasks and then fine-tuned for specific downstream tasks.

On one hand, this enhances model robustness by pre-training on more data; on the other hand, it allows the continuous prompts to benefit from broader knowledge across various tasks and datasets during pretraining.

Replacing verbalizer with a classifier:

The verbalizer has been a core component of prompt tuning methods. It establishes a mapping between task-specific class labels and semantically meaningful words (e.g., “great”, “terrible”) in the vocabulary, allowing language models to perform classification by predicting words at the [MASK] position. While the use of a verbalizer may be necessary in few-shot settings, it is not essential in fully supervised scenarios with large-scale data. In fact, verbalizers can limit the generality of prompt tuning by requiring predefined label words.

Therefore, P-tuning v2 adopts the traditional CLS-based classification approach, applying a randomly initialized classification head directly on top of token representations.

In practical implementation, both Prefix Tuning and P-Tuning v2 inject prompts using past_key_values inside the attention operations for every transformer layer.

Summary of Soft Prompt Methods:

Method	Processing of Virtual Tokens	Parameters Tuned	Suitable Downstream Tasks
P-tuning	MLP + LSTM or MLP	Only the virtual token part in the embedding layer	Adapts GPT to NLU tasks
Prefix tuning	MLP	Prefix encoder & additional parameters in each layer	Mainly for NLG tasks
P-tuning v2	—	additional parameters in each layer	NLG/NLU tasks

<ins/>

6.6.5 Adapter Tuning

💡

Core Idea: Adapt to downstream tasks by introducing some additional modules

Adapter tuning: Inserts extra Adapter modules into each transformer layer and then fine-tunes downstream datasets. Both training and inference computation increase since Adapters are added linearly within the transformer architecture (i.e. 10 layers means 10 adapters).

LLaMA-adapter:

Adapter tuning, Prefix tuning, and P-tuning V2 face a common issue: a large number of randomly initialized parameters may damage the pre-trained semantic knowledge, leading to unstable fine-tuning and performance loss. LLaMA-adapter addresses this problem with two modifications to enhance stability:

Introduces zero-initialized attention and gate mechanisms

Modifies only the deepest few Transformer layers

6.6.6 LoRA & its Variants

LoRA

🥝

Low-Rank Adaptation (LoRA)

LoRA is a method for efficiently fine-tuning large models. By using LoRA, the number of trainable parameters is only a tiny fraction of the full model’s parameters. GPU memory usage can be reduced by more than 2/3 without impacting inference speed (an advantage over Adapters).

During fine-tuning, with the full weight matrix , we optimize it with formula , where the update is a low-rank decomposition and are low-rank matrices. This approach is effective when the low-rank update space approximates the space spanned by the full-rank update. This allows for reducing the number of trainable parameters and resource consumption, while maintaining comparable performance.

LoRA Implementation: LoRA achieves low-rank adaptation by decomposing the weight update into low-rank matrices. During training, these low-rank matrices are only learning processing features for downstream tasks.

Intrinsic Dimension: LLMs demonstrate strong few-shot capabilities. Studies show that even with a small number of training samples (a few thousand), we can finetune a model with billions of parameters. This is because the pre-trained model has a small intrinsic dimension, i.e., the intrinsic dimension refers to parameters in a very low dimension, we can just fine-tune them to achieve performance comparable to full fine-tuning.
Low Rank: Based on this idea, LoRA assumes that during training, the number of parameters that actually need to be adjusted lies within a low-dimensional subspace. Therefore, we can approximate the updates using low-rank matrices.

During training, the original weights are frozen, and only the parameters in matrices and are trained. The forward pass is thus transformed into:

Which parameter matrices should LoRA be applied to?

Placing all fine-tuning parameters into just one of the attention parameter matrices doesn’t yield best results. Distributing the fine-tuning parameters evenly across and produces the best performance.
Even with a rank as low as 4 and 2, it's possible to learn a good enough .

What is the optimal rank for LoRA?

When the rank is as small as 1 or 2, LORA still shows decent performance, suggesting that the updated parameter matrix ΔW might lie within an extremely small subspace.

LoRA matrix initialization: Simply put, the down-sampling matrix A is initialized with random Gaussian distribution, while the up-sampling matrix B is initialized to zeros.

📌

This initialization ensures that the initial matrix is still a zero matrix at the start of training, and thus has no effect on the pre-trained parameters.

Why can B be initialized to all zeros?

If is initialized to all zeros, then at the beginning, , which means the model's weights haven't been changed yet, so it's still the original model. As training proceeds, will gradually be updated, and eventually learn the desired weight adjustments. The gradient of is:

Therefore, initializing to zeros doesn't hinder training, since it can still receive non-zero gradients.

Why can't A be initialized to all zeros?

If is initialized to all zeros, then , and the gradient of :

will be zero because is also initialized to zero, so won't be updated. And won’t be updated as well as the gradient of shown before, which relies on non-zero . This would prevent the model from learning meaningful adjustments through training.

This paper The Impact of Initialization on LoRA Finetuning Dynamics conducted experiments comparing the two initialization methods for A and B:

Init[A] offers better feature learning efficiency but introduces some training instability

Init[B] provides stable training and suboptimal feature learning

Init[A] generally performs better overall, due to superior feature learning. A bit of training instability is considered acceptable.

LoRA Code Analysis

Weight Initialization: non-zeros for and zeros for

Forward: There is an additional scaling factor and optional dropout. Typically, scaling = alpha / r for improving training stability especially in early stages, where alpha is a hyperparameter and the default is 1.

<ins/>

QLoRA

⭐

Based on the LoRA finetuning method, techniques such as 4-bit NormalFloat, Double Quantization, and Paged Optimizers are introduced to reduce memory usage while maintaining performance. By adding adapters to each network layer, QLoRA avoids nearly all accuracy degradation observed in prior work. This approach reduces the memory requirement for 65B parameter models from over 780GB to under 48GB, making it possible to finetune the largest publicly available models on a single GPU.

Also mentioned in [7.1.3 Quantization-Aware Fine-tuning (QAF)]

LoRA+

🔆

To assign different learning rates to matrices A and B, a more effective method for training LoRA is introduced. The specific update process is as follows. In the paper, the learning rate for B is 6 times that of A:

LoRA+ Code

Mainly on customizing optimizer parameters

VeRA

Vector-based Random Matrix Adaptation (VeRA) introduces a method to significantly reduce the number of LoRA parameters. Instead of training the full matrices A and B, shared randomly initialized matrices are used (i.e., all layers share the same weights for A and B). Two new vectors, d and b, are added, and during finetuning, only these vectors d and b are trained.

LoRA-FA

📌

Building on LoRA, LoRA with Frozen-A (LoRA-FA) follows a similar idea to VeRA, where matrix A is frozen after initialization and can therefore be treated as a random projection. Matrix B is initialized to zero (as in standard LoRA) and then trained. This approach cuts the number of trainable parameters in half, while achieving performance comparable to standard LoRA.

<ins/>

AdaLoRA

💡

Adaptive LoRA (AdaLoRA):

AdaLoRA is an improvement over standard LoRA, which adaptively adjusts the rank of LoRA matrices to more efficiently allocate limited trainable parameters.

Core Concept:

Not all model layers are equally important. AdaLoRA uses singular values to assess the importance of each layer.

Higher singular values indicate greater impact on the model, so those layers are assigned higher ranks.

It leverages Singular Value Decomposition (SVD) for efficiency, avoiding the high cost of directly computing singular values.

AdaLoRA also considers loss sensitivity—how much changing a parameter affects the loss function—to determine importance.

Differences from Standard LoRA:

Standard LoRA: All adapter layers have the same rank.

AdaLoRA: Ranks vary by layer—more important layers get higher ranks, less important ones get lower.

Total parameter count remains the same, but parameters are distributed more intelligently.

Advantage: With the same parameter budget, AdaLoRA prioritizes key layers, often leading to better performance than standard LoRA in practical tasks.

The figure below illustrates how AdaLoRA assigns ranks to a given model. As shown, it allocates higher ranks to the layers near the end of the model, indicating that adjusting these layers has a greater impact on improving performance for the target task.

AdaLoRA Code Example

Definition of a single AdaLoRA layer, where the self.lora_E is the singular values

Note that AdaLoRA does not determine each layer’s rank r before training. Instead, It dynamically adjusts the rank during training using a mechanism that:
Starts with an initial rank that is slightly higher than the final target.
Iteratively prunes singular values based on an importance score, which reflects the contribution of each parameter triplet to model performance.
This is why the self.lora_E is initialized with a fixed rank r, and AdaLoRA masks out the singular values based on importance scores. “This preserves the possibility of future recovery and stabilizes the training” mentioned in the paper.

Specifically, schedule_threshold(self, step) calculates the current total rank curr_rank to be used based on the current training step. It follows a cubic decay curve from the initial rank down to the final target rank:

This acts as a global scheduler, determining how many ranks should be retained at any given step. Then, mask_to_target_rank(self, model, curr_rank) is the most critical step:

First, calculate the sensitivity (importance) of each LoRA triplet (i.e., the singular value lora_E and its corresponding direction) using

Then use torch.kthvalue(...) to find the global masking threshold.

Finally, zero out the unimportant elements in lora_E:

<ins/>

DoRA

📌

DoRA can be broken down into two steps:

First, decompose the pre-trained weight matrix into two parts:

one is the magnitude vector (m),
and the other is the direction matrix (V).

Then, apply LoRA to the direction matrix V, while the magnitude vector m is also trained independently.

DoRA Code Example

X-LoRA

⭐

By combining LoRAs pre-trained on multiple different domains and introducing a trainable scaling factor, each LoRA's contribution can be dynamically adjusted.

X-LoRA Code Example

6.5 Packing for Efficient Training

7.1 Quantization

Catalog

Last update: 2025-05-22

🧬 AI Learning Blog 🧬

Anticipated updates:

Must-read papers, and Hands-on practice in general
Digital Image Processing/Mathematics
Vision Language Model