✨ Awesome-Anything

Awesome-AI-Researchs

Awesome-GitHub-Repos

Awesome-AI-Examples

🖼️ Digital Image Processing

DIP I Image Demosaicing and Histogram Manipulation

DIP II Image Denoising

DIP III Edge Detection

DIP IV Digital Half-toning

DIP V Geometric Image Modification

🍃 LLM Components

1.1 Tokenization

1.2 Embedding

1.3 Attention

1.4 FFN, Residual Addition, and LN

1.5 Positional Encoding

1.6 Architecture & Decoding Policy

🌱 LLM Pre-training

2.1 Pre-training Definition

2.2 Pre-training Data

2.3 Pre-training Workflow

2.4 Continued Pre-training (CPT)

2.5 Pre-training Evaluation

☘️ LLM Post-Training

3.1 Supervised Fine-Tuning (SFT)

3.2 Reinforcement Learning

3.3 RLHF: Reinforcement Learning from Human Feedback

3.4 Optimizing PPO-Based Algorithms

3.5 Direct Preference Optimization (DPO)

🍀 LLM Popular Models

🪴 LLM Applications

5.1 Prompt Engineering

5.2 LLM-based Agents

5.3 Retrieval-Augmented Generation (RAG)

5.4 Enterprise-level RAG Practice (I)

5.5 Enterprise-level RAG Practice (II)

🌿 LLM Optimization

6.1 GPU Usage Analysis

6.2 Latency Analysis for Inference

6.3 FlashAttention

6.4 PagedAttention

6.5 Packing for Efficient Training

6.6 PEFT

🌾 LLM Compression

7.1 Quantization

7.2 Pruning

7.3 Knowledge Distillation

🌵 LLM Hands-on Practice

8.1 nanoGPT: Training a GPT from Scratch

8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example

8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis

8.4 OpenRLHF: Reward Model Training and Code Analysis

8.5 OpenRLHF: PPO Tutorial and Code Analysis

🌴 LLM Must-read Papers

9.1 Technical Reports Explained

9.2 MedLLM

9.3 o1 Techinical Paper

9.4 Prompting

9.5 LLM-as-Judge

🌳 LLM Q&A

10.1 Why GRPO training loss starts at 0

10.2 VLM-R1 Reproduction Results

10.3 KL(P||Q) or KL(Q||P)?

10.4 Left Padding or Right Padding?

10.5 What is QK-Norm?

10.6 Why LLM training rarely adopts Dropout?

10.7 Dropout and DropPath

🐝 VLM Image Encoders

11.0 Introduction

11.1 Vision Transformer

📝 MISC.

🤔 Large Language Models Pass the Turing Test.

OpenAI ChatGPT Progression

⏳ Claude Sonnet 4 Test with Sandtris

🔍 What DeepSeek-R1's Surge Teaches Us About SEO

💰 CivitAI’s Payment Issue

🎓 arXivSub, an Academia Assistant

1.5 Positional Encoding

1.5.1 Introduction

RNN structures contain sequential information, but Transformer completely losses sequence information because of the parallel mechanism of self-attention, making them insensitive to the order of words in a sequence. However, “He owes me 1 million” and “I owe him 1 million” have completely different meanings, so to address this issue of lacking sequential awareness, the authors of Transformer proposed a clever solution: Positional Encoding.

The Attention mechanism is directionless, and the positional information and the relationships between tokens are very important. For humans, it's easy to know the position of tokens. For example: in the sequence of “a1, a2, a3, …”

Absolute positional information, like a1 is the first token, a2 is the second token…

Relative positional information, like a2 is one token after a1, and a4 is two tokens after a2…

Different positional distances, like a1 and a3 are two tokens apart, and a1 and a4 are three tokens apart…

Therefore, we need a kind of positional representation that satisfies:

It can represent the absolute position of a token in the sequence

In situations with different sequence lengths, the relative position/distance between tokens in different sequences should remain consistent

It can allow the model to generalize to sentence lengths it hasn't seen during training (extrapolation)

Therefore, we need a bounded and continuous function, and the simplest option is the sine function. Setting the frequency appropriately low to ensure that different positional encodings rarely repeat. So we define:

1.5.2 Absolute Positional Encoding

Transformers PE

In Transformers, the PE is added to the token embedding. Because right now sine and cosine are alternated, you can obtain representations for other positions through linear transformation matrices, which allows you to expect the representation to contain relative position information. Also, due to the regularity in how trigonometric functions are generated, you can expect the model to have extrapolation capability.

Visualization: The figure below is a positional encoding visualization for a sequence of length 100 () and encoding dimension 512 (). You can observe that due to the nature of sine/cosine functions, every value oscillates between -1 and 1. Also, visually, you’ll notice the right side is mostly yellow. This is because later positions use lower frequencies and longer wavelengths.

Transformer PE limitation

Due to the directionless nature of the dot product of positional encodings

i.e., the dot product between two positional encodings depends only on the absolute value of , and therefore cannot represent directional information. When the input embeddings are fed into the attention mechanism, this can lead to ambiguous recognition of distances, meaning the ability to represent relative positions through positional encoding can be disrupted by matrix projections.

As a result, later improvements in BERT adopted learnable positional encodings.

📌

The Directionless Nature of the Dot Product of Sinusoidal Positional Encoding

In the original Transformer, each position is encoded as a vector with sine and cosine functions of different frequencies. Except for the previous expanded version, it can also be written in the below form:

Where:

is the embedding dimension,

is the dimension index.

To show:

We define:

Then the PE components are:

Now for each frequency , we take the dot product of and :

Note that:

Then we have . Thus, the total dot product becomes:

This is symmetric with respect to :

So the dot product between PEs is invariant of the sign of . Q.E.D.

Let’s visualize the symmetry of the PE dot product (in terms of the )

BERT’s Learnable Positional Encoding

Directly treats positional encoding as trainable parameters. For example, if the maximum length is 512 and the encoding dimension is 768, a matrix of size 512×768 is initialized to represent positional embeddings, and it is updated during training. For such learned absolute positional encodings, a common belief is that the drawback is the lack of extrapolation capability—i.e., if the model is trained with a maximum sequence length of 512, it may not be able to handle sequences longer than 512 later on. Of course, positions beyond 512 can be randomly initialized and then fine-tuned further, but it is just hard to do in a zero-shot behavior.

RNN Positional Encoding

Uses recursive absolute positional encoding. Input is fed into an RNN layer, allowing the RNN to learn the position of each token. The advantage is extrapolation and flexibility, while the disadvantage is losing the parallel processing characteristics of the Transformer.

<ins/>

1.5.3 Relative Positional Encoding

Relative Position Representation (RPR)

💡

Why RPR?

RPR helps the model focus on how far tokens are from each other, rather than where they are, making attention more flexible and better at generalizing across sequence lengths, e.g., longer context.

To understand the RPR, we can first decompose the query and key vectors in the self-attention formula into embedding and sinusoidal PE.

Let be the input embedding for position . Note that this RPR is for self-attention, which is why the input embeddings of Q and K are the same. Let be the learned projection matrices for query Q, key K, and value V, respectively.

The standard self-attention (with absolute position encodings ) becomes:

To incorporate relative positional information, The position embedding part () for the Q is discarded; the original positional term is replaced with a relative position embedding , and similarly in the output term with . The modified attention becomes:

The relative position vectors and are learned embeddings that depend only on the relative distance , and are clipped within a fixed window size . Formally:

This clipping ensures that the number of learnable relative position embeddings is bounded, making it efficient and generalizable to sequences of unseen lengths.

For example, during training, if we set a clipping distance of , then all relative distances greater than 2048 — such as 2476 or 3412 — will be clipped to 2048. This means the model will treat any tokens that are more than 2048 positions apart as if they are exactly 2048 apart. Effectively, the model learns that such distant tokens are just "too far" and assigns them the same relative positional embedding. During inference, we can let the model to process longer sentence without providing unrecognizable (out-of-distribution) position embeddings.

XLNET Positional Encoding

From the Transformer-XL paper, the self-attention formula is expanded as follows:

Transformer-XL simplifies this by directly replacing the with a relative position vector , where the is a sinusoid encoding matrix without learnable parameters; is replaced with two trainable vectors :

Because the encodings and are not in the same distribution, the is replaced with another matrix , and also and are merged into the two vectors , resulting in the final formula:

In addition, the position embedding for Value V is just discarded. So

Starting from here, many subsequent works apply relative positional encoding only to the Attention score matrix, not to value V, for simplicity and efficiency.

T5 Positional Encoding

The T5 model uses a simpler form of relative positional encoding, still originating from previous expansion formats as shown below.

If we break down the meaning of each component, we can interpret them as combinations of four types of attention: "Input-Input", "Input-Position", "Position-Input", and "Position-Position". The authors think that the input content and position information should be independent (interpreted separately), then they should not interact too much. Therefore, "Input-Position" and "Position-Input" attention components can be removed.

Besides, the term actually only depends on the label pair , so we can directly train a learnable parameter for it, simplifying the to:

Clearly, this method only adds a learnable bias term on top of the Attention matrix. A similar idea is also found in a paper presented at ICLR 2021 titled "Rethinking Positional Encoding in Language Pre-training", which proposed TUPE positional encoding.

T5 processes relative positions with a "bucketing" method, where the relative position is mapped to a bucket position , establishing a mapping relationship of:

This mapping design is actually quite intuitive: positions that are relatively close need to be more precise, so each of them is assigned a unique position code (like positions -7, -6, …, 0, …, 6, 7). As for farther positions, there’s no need to distinguish them as clearly, so they can share a position code. The farther the distance, the larger the shared range can be, until it reaches the specified range and then clips.

Note that the uni-directional is only for the negative part because the current token can only see its past tokens (not disclosing futures!)

Bucketing Visualization

The _relative_position_bucket function is from Huggingface

DeBERTa Positional Encoding

Similarly, starting from the formula , T5 removes items 2 and 3, keeping only item 4 and replacing it with a learnable relative positional encoding; DeBERTa, on the other hand, does the opposite: it removes item 4, keeps items 2 and 3, and replaces them with relative positional encoding:

where the is also a learnable embedding and the is the predefined maximum relative distance.

DeBERTa offers a new perspective on utilizing relative and absolute positional encodings. It points out that although most NLP tasks benefit from relative positional information, in some cases, absolute positional information is more helpful.

The entire model is divided into two parts. In the case of the MLM pretraining using the Base version of DeBERTa, it has a total of 13 layers: the first 11 layers only include relative positional encoding (referred to as the encoder), and the last 2 layers adopt absolute positional information (referred to as the decoder, also called EMD – Enhanced Mask Decoder). For downstream fine-tuning tasks, the 11-layer encoder and 1-layer decoder are used.

Note that the names "encoder" and "decoder" here are not in their traditional sense, i.e. only specific to the DeBERTa here, so don’t get confused

ㅤ	Relative PE	Absolute PE
BERT	❌	✅ Learnable
RoBERTa	❌	✅ Learnable
DeBERTa	✅ Learnable	✅ Learnable

<ins/>

1.5.4 RoPE and ALiBi

RoPE

Rotary Position Embedding (RoPE), through injecting absolute position information, achieves relative position encoding, combining the advantages of both absolute and relative position encodings. The main idea is that the q and k vectors incorporate absolute position information, and then the updated q and k vectors will bring relative position information by doing inner product during attention computation. RoPE is widely used in the current large language model ecosystem.

Let’s start with recapping the common absolute and relative positional encodings.

Absolute positional encoding

The encoding process can be formulated as:

Relative positional encoding

So and are learnable lookups (embeddings) in the shape of .

Rotary position embedding

The authors try to formulate the relative relationship between tokens like below, where the dot product between tokens injected with absolute position information inherently contains relative position information.

And they found a solution:

Above is the actual method but we will prove it and give a more practical implementation. Note that the indicates the real part of the complex number; represents the conjugate of ; and the is a pre-defined constant.

📌

Proof: assuming a 2D case (i.e. d = 2)

We start with the :

Now we transform the 2D vector and the expotential into complex format:

Therefore, we have:

And similarly,

Now let’s dot product them together:

We denote the above equation as . Next, we derive the to show that it’s equal to :

Obviously, and equals, Q.E.D., i.e.,

📌

Generalize to d > 2 cases

Recall that for d=2 case,

Since the inner product satisfies linear additivity, any even-dimensional RoPE can be represented as a concatenation of two-dimensional cases, i.e.

After expanding the matrix multiplication, we can reformat this equation as:

Where the indicates the Hadamard product (element-wise product); and the is pre-defined as:

RoPE Code

In below code, the Hadamard product is implemented in a complex number multiplication way. For example, to get and , we use

Where the real part and the complex part are exactly what we want, then we use torch.view_as_real to separate them.

<ins/>

ALiBi

Attention with Linear Biases (ALiBi)'s modification is very simple — before the Softmax, the Attention computation is changed from to

where is a hyperparameter, with each head having a different value.

From this definition, it's clear that ALiBi is quite similar to local attention — both subtract a non-negative matrix before Softmax, though the exact matrix differs. ALiBi can be seen as a “smoothed” version of local attention.

Matrix effectively subtracted by local attention

ALiBi’s bias matrix penalizes the attention score based on the relative distance between q and k — the greater the distance, the larger the penalty. In other words, the farther apart two tokens are, the less they contribute to each other.

The length extrapolation: This refers to the mismatch between training and inference context lengths.

During inference, positions not seen during training may appear (whether absolute or relative).

During inference, the number of tokens processed by attention mechanisms may far exceed the number seen during training.

Note that: The length extrapolation ability of relative position encoding is generally better than that of absolute position encoding; function-based relative position encodings like RoPE tend to extrapolate better than learnable relative position encodings (look-up embeddings).

1.5.5 Length Extrapolation

Representation of Numbers & Extrapolation

Number Representations: Assume we have an integer N less than 1000 (not including 1000) that needs to be used as a conditional input to a model. Which representation would be better?

Floating Point:

Simply input a float (1D space) between 0 and 999. This approach comes with issues when crossing large scales from 0 to 999, and it is not easy to optimize for gradient-based methods. Scaling it down to between 0 and 1 doesn't help much either, because the relative distance becomes 0.001, making it hard for the model and optimizer to distinguish the numbers effectively.

Decimal Representation:

Convert the number N into a 3D vector [a, b, c], where a, b, and c represent the hundreds, tens, and units digits, respectively. This helps avoid issues related to large crossings in magnitude. You can even go further by using binary representation, or even one-hot encoding. The cost is the increasing of vector dimension.

Direct Extrapolation:

If we train the model directly with the 3D vector (Decimal representation) as input, we cannot input a number range beyond 999 later on. A common solution is to reserve several dimensions in advance during training, so that during inference, we can directly use the reserved dimensions for larger numbers. This is direct extrapolation. However, direct extrapolation is difficult because the model has not seen similar examples in training, or the extrapolated values are far from what it has learned. Therefore, direct extrapolation often leads to severe performance degradation in the model.

Linear Interpolation & Representation Transformation

Linear Interpolation: Changing extrapolation to interpolation means compressing the range from 2000 to 1000.

For example, by dividing by 2, 1749 becomes 874.5, then it is transformed into a three-dimensional vector [8, 7, 4.5]. Now, the new [7, 4, 9] corresponds to 1498, [3, 7, 4.5] corresponds to 749. From a relative value perspective, the distance between adjacent numbers used to be 1, but now it’s 0.5, so the last dimension is more crowded.

Interpolation requires fine-tuning, so the model can adapt to the more compressed mapping relationships.

When the number range increases, the differences between adjacent numbers are also shrinking. Such crowdedness is only in the unit's place, but interpolation methods will result in different distributions in each dimension, making the model more difficult to adapt.

Representation Conversion: No need to increase dimensions, and relative distances are preserved

Three-digit decimal encoding can represent 0–999, while base-16 (hexadecimal) can represent 0–4095, which is sufficient to cover the target value of 1749 in three dimensions. And each dimension's value changes from 0–9 to 0–15.

A well-trained model has already learned that 875 > 874, and under base-16 encoding, 875 > 874 still holds true, so the comparison rule is preserved.

After each dimension exceeds 9 (i.e., moves from 10–15), since models usually have some generalization ability, a slight shift in each dimension is not an issue. The conversion might still work without fine-tuning the original model.

Positional Interpolation

Language models are usually pre-trained with a fixed context length. How can we use a relatively small amount of data for fine-tuning to extend the context length beyond the pre-training limit? Positional interpolation.

📌

Problem with RoPE: direct extrapolation tends to produce large Attention Scores

Impact of high frequencies:

In RoPE encoding, high-frequency components (i.e., those with higher sine and cosine frequencies) are more sensitive to positional changes. This means that as position increases, these high-frequency components change more rapidly, which can lead to sudden spikes in the Attention Score between the query and key tokens at those positions.

Key idea: Instead of extrapolating, directly reduce the position index so that the maximum position index matches the original maximum index.As shown below, if you directly use a position like [2048, 4096], since the model has not seen that part of the position before, performance degradation is likely to occur. Instead, compress the range [0, 4096] to [0, 2048], so that the original position 1 becomes 0.5, and 4096 becomes 2048. This is called positional interpolation, similar to linear interpolation mentioned before, where unseen positions are mapped to seen ones.

Interpolation Formula:

For absolute position , the scaling factor is , where the is the original context length such as 2048, and the is the extended context length such as 4096. So the computation of Q and K becomes

💡

Whether Fine-tuning After Positional Interpolation Helps?

Without fine-tuning, the model still exhibits a certain language modeling ability, such as expanding the context window up to 8192 tokens. The perplexity is still under 20 (a relatively low value), but a direct extrapolation leads to a perplexity of around 1000.

After fine-tuning, perplexity improves significantly. At 200 training steps, the model begins to outperform its original perplexity when the context window is 2048.

📌

Perplexity is a commonly used evaluation metric in natural language processing, used to measure the quality of a language model. A language model's perplexity on a test dataset is calculated using the following formula. The lower the perplexity, the better the model is at predicting the next word, and the more accurate the predictions:

The Problem of Positional Encoding (PI):

The sine function has a period . In RoPE, each dimension corresponds to a different frequency (e.g., ), with

Where is the position, is the embedding dimension.

Then the period can be calculated as:

From this formula, it can be seen that each dimension corresponds to a function with an increasing period.

If interpolation is done on the absolute position , all dimensions will be affected equally. However, since high-frequency dimensions have lower periods, interpolation causes these dimensions to become very crowded (e.g., originally one period includes 10 values, the interpolation increases it to 20 values). This is also similar to the linear interpolation before.

<ins/>

NTK-aware Interpolation

NTK-aware interpolation: The core idea is extrapolation for high frequencies and interpolation for low frequencies. Unlike PI which equally scales all dimensions, NTK-aware interpolation reduces interpolation in high-frequency regions and increases interpolation in low-frequency regions, thereby spreading the crowdedness across multiple dimensions:

We first define a general formula of interpolation:

Recall that in PI, the , where and , i.e.,

In the NTK-aware interpolation,

The fundamental difference from the RoPE is that the base is multiplied by a constant related to the expansion ratio . The is a scaling value similar to the one in PI, which means if we want to extrapolate context length of 2048 to 4096, we can set . Besides, the is usually large, so the is slightly larger than 1, which scales up the .

Note that, by fixing , , and , the base increases, then the decreases

In the high-frequency region, i.e., the is small, the is near to 0. In this case, the didn’t affect the as much, so we are basically minimally decreasing and also minimally doing interpolation. (This can be seen as extrapolation after interpolation)

In the low-frequency region, i.e., the is large, the is near to -1. In this case, the base is scaled up, and the decreases more significantly, so we are doing more interpolation than the high-frequency region.

In practice, we can set the slightly higher than what we need.

Comparison with the positional interpolation (PI)

Where the scale indicates PI and indicates the NTK-aware interpolation; the scale=1 is the baseline without any interpolation. Clearly, the PI can reduce the PPL in a longer range but still increase the PPL in a lower context range than the baseline. As for the NTK-aware interpolation, it also slightly increases the PPL than baseline, but compared to PI, it significantly reduces the downgrade and largely increases the context window length.

Visualize the rotation angle

In the left plot, you can observe the NTK-aware interpolation didn’t decrease the rotation as much as PI, which means not much interpolation is done; the right plot is a more zoomed-in version in the lower frequency region, and you can observe that the two methods are similarly doing interpolation in such lower frequency region.

<ins/>

NTK-by-parts Interpolation

This method improves over the NTK-aware interpolation by considering the relationship between the wavelength (tokens number for a period) and context length

Wavelength: For dimension and the complete rotation , the required tokens is

For some dimensions, the maximum context length seen during training may be smaller than the wavelength, i.e., . This means the embedding in some dimensions might rotate less than a cycle, so the embeddings in these dimensions are not evenly distributed. When the wavelength is very long, the embeddings in those dimensions are unique, so they can be considered to preserve absolute positional information; when the wavelength is short, the embeddings will complete multiple rotations in a sequence, which results in those embeddings only reflecting relative positional information.

Problem: When positional interpolation or NTK-aware interpolation is used, all tokens become closer to each other, although the NTK-aware interpolation is much better than PI in high-frequency regions. This affects the model's ability to understand relative positional relationships between nearby and small-range embeddings, leading to performance degradation due to the model losing its sense of token order and local relations.

Solution:

To solve these, the NTK-by-parts interpolation decides not to interpolate the high-frequency dimensions at all. Define a ratio:

which measures the relationships between the context length and the wavelength for dimension . Then introduce threshold parameters , , and define a ramp interpolation function.

Where the optimal and varies across different LLMs; a good start for LLaMA models can be and . The final NTK-by-parts formula becomes:

Where the is defined the same as in PI.

Note that: a more complex variant can be made by combining both PI and NTK-aware interpolation into this NTK-by-parts interpolation, where the hyperparameters number will increase, needing more efforts to find the optimal ones. An example can be found in here.

Visualize and compare with previous interpolation methods

You can observe that the NTK-by-parts are like the original RoPE (no interpolation) in high-frequency regions and like PI in low-frequency regions; in the middle, it’s a mix of both.

Dynamic NTK Interpolation

This method just slightly changes the computation of the scaling factor , and combined with the NTK-aware interpolation, resulting in dynamic NTK interpolation. If you’d like, this dynamic can also be applied to PI or NTK-by-parts interpolation.

Problem: During inference, the target sequence length could range from 1 to the maximum context length, or even beyond the maximum context window. This is the nature of autoregressive generation. Previous methods used a fixed scaling factor, which can lead to performance degradation when the sequence length is smaller than , and potentially cause abrupt degradation when the sequence length becomes large enough (this can be seen in the previously mentioned NTK-aware method)

Dynamic scaling

where is the current sequence length. When the is smaller than the context window , we just let the to be 1, which means we do not do any interpolation. When the current context length is longer than , we gradually increase the

You can observe that the dynamic version preserves the performance for a context window shorter than 2200. Although the dynamic NTK performs a bit worse than the NTK-aware interpolation at around 3500, it largely delays the abrupt degradation of the NTK-aware method.

<ins/>

Yet another RoPE extensioN (YaRN)

This method is simple and empirical. No matter what the data sample is or the token position, introducing a temperature before the softmax operation on the logits can uniformly affect PPL. The computation of the attention weights can be modified as follows:

Practically, the RoPE embedding is proportionally scaled down, making both the query and key scaled by , equivalent to the above equation. Then, combining this temperature scaling with the NTK-by-parts method yields YaRN. With YaRN, the inference and training phase requires no additional overhead because the RoPE embeddings are pre-generated and can be reused.

For LLaMA1 and LLaMA2, the recommended temperature is as below, where the is the one used in NTK-by-parts method.

Visualization of the extrapolation progression

1.4 FFN, Residual Addition, and LN

1.6 Architecture & Decoding Policy

Last update: 2025-04-19

🧬 AI Learning Blog 🧬

Anticipated updates:

Must-read papers, and Hands-on practice in general
Digital Image Processing/Mathematics
Vision Language Model