4.5 GLM Series | AI Roadmap

4.5 GLM Series

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Link: https://arxiv.org/abs/2103.10360

Model Architecture: GLM-1 uses a prefix-decoder architecture based on Transformer. (In essence, it is a Transformer decoder, but with a special masking mechanism that enables bidirectional attention in the prefix sequence and unidirectional attention in the later sequence.)

Several modifications are made:

Reorder the LayerNorm by placing it before the attention module and within the residual connection.
A single-layer linear head is used for output token prediction
ReLU → GeLU activation function

Autoencoding perspective: Randomly remove contiguous tokens from the input text.
Autoregressive generation perspective: Reconstruct tokens sequentially. When predicting the missing tokens in an autoregressive manner, the model can access both the corrupted input and previously predicted spans.

span shuffling + 2D positional encoding technique:

By controlling the number and length of missing spans, the autoregressive blank infilling objective can serve both conditional and unconditional generation pretraining purposes.
The input x is split into two parts: Part A is the corrupted text with masked spans (corrupt). Part B is the masked spans. Suppose the original input is: , and two sampled spans are and , then the corrupted text becomes: , which forms Part A. The corresponding segments from Part B are then shuffled, and each span is prepended with [S] as input and appended with [E] as output.
2D Positional Encoding: GLM uses 2D positional encodings. The first position ID marks the token's position within Part A. The second position ID indicates the relative position within a span. Both are projected and added into the token embedding.
GLM uses a custom attention mask:

Tokens in Part A can attend to each other but cannot see any token in Part B (prefix part uses bidirectional attention)
Tokens in Part B can attend to all of Part A and the past tokens in Part B, but not future tokens in Part B (masked attention / causal attention for Part B)

The model automatically learns both a bidirectional encoder (Part A) and a unidirectional decoder (Part B)

Multi-task Pretraining: Jointly optimizes long-text generation and blank infilling objectives.

Doc-level (Document level): Randomly sample a span within 50%–100% of the original input length. The objective focuses on generating long text.
Sentence-level: Masked spans must be complete sentences. Multiple spans are sampled to cover 15% of the original tokens (same as BERT). This objective is suitable for seq2seq (sequence-to-sequence) tasks, where the predicted output is usually a complete sentence or paragraph.

<ins/>

GitHub: https://github.com/THUDM/ChatGLM2-6B

Fully Decoder-only architecture, using RoPE (Rotary Position Embedding), replacing GELU with SwiGLU, and applying Pre-RMSNorm
MHA → MQA (Multi-Head Attention replaced by Multi-Query Attention) for more efficient inference
Supports longer context windows (based on FlashAttention, from 2K to 32K), with up to 8K context windows during dialogue training

ChatGLM2-6B uses GLM’s hybrid objective function, pre-trained on 1.4T Chinese-English tokens and trained with human preference alignment

📌

Why prefix-decoder to casual-decoder?

Multi-round dialogue using a prefix-decoder requires constructing multiple samples for training, while decoder-only models with a causal mask can directly use the entire multi-round dialogue data as one sample. This method achieves the same effect, and it weights multiple dialogue rounds differently, which made 2D positional encoding unnecessary and thus removed.

In the case of a 3-round dialogue (Q1A1, Q2A2, Q3A3), PrefixLM requires creating three training samples:

Decoder-only models leverage the nature of causal masks to implement multi-round dialogue training in a single sample:

The LLaMA architecture showcases the strong autoregressive generation capability of decoder-only models, so GLM-2 follows.

GitHub: https://github.com/THUDM/ChatGLM3

Model Architecture: ChatGLM2 and ChatGLM3 share the same architecture, which is different from the original ChatGLM. Compared to ChatGLM, the changes in ChatGLM2 and ChatGLM3 include:

Vocabulary size reduced from 150,528 (in ChatGLM) to 65,024 (This makes ChatGLM2/3 significantly faster to load than the original ChatGLM)
Positional encoding: Changed from one embedding per GLMBlock to a single shared global embedding
Feedforward network after self-attention differs: