Lazy loaded image1.6 Architecture & Decoding Policy

1.6 Architecture & Decoding Policy

1.6.1 LLM Architectures

Encoder, Decoder, and Hybrid

https://arxiv.org/pdf/2303.18223
LLM architectures can be divided into four types: Decoder-only, Encoder-only, Encoder-Decoder, and Prefix LM (can be seen as a special case of Encoder-Decoder).
  • Decoder-only: Uses unidirectional attention from left to right, typically autoregressive language models. The pre-training and training for downstream applications are exactly the same, leading to excellent text generation results. It has high training efficiency and strong zero-shot capabilities.
      • Representative models: GPT, LLaMA, BLOOM, OPT
  • Encoder-only: Uses bidirectional attention. Mainly used for language understanding tasks, extracting representations, and a pre-training language model.
      • Representative model: BERT
  • Encoder-Decoder: Uses bidirectional attention for input and unidirectional attention for output. Performs well in tasks requiring deep understanding (e.g., question answering), but training is less efficient, and text generation is less expressive.
      • Representative models: T5, Flan-T5, BART
  • Prefix Decoder: Uses bidirectional attention for the prefix and unidirectional attention for output. A hybrid approach that combines the architecture of a decoder but uses the mask as in encoder-encoder.
      • Representative models: GLM, U-PaLM

Mixture of Experts (MoE) Architecture

Representative models: Deepseek V2/V3, Mistral
Typical LLMs are dense models, in contrast to MoE models. Each MoE layer is usually composed of N expert networks , paired with a gating network . This gating network is usually a linear network with a softmax activation function, whose purpose is to guide the input to the appropriate expert networks.
The placement of MoE layers is in the Transformer modules, replacing FFN, and is usually placed after the Self-Attention (SA) layer. This placement is reasonable because as the model scales up, the computational cost of FFN increases significantly. For example, in a model like PaLM with up to 540B parameters, 90% of the parameters are in the FFN layer.
The gating network (also known as the router or switch) is the foundation of all MoE frameworks. It coordinates which experts to use for computation and how to combine expert outputs. Based on how it processes inputs, gating can be divided into three types: sparse, dense, and soft. Among them, the sparse gating mechanism activates only a few experts, while dense activates all, and soft includes differentiable methods like token merging and expert fusion. More examples can be found in the [4.8 MOE Series].
notion image
<ins/>

1.6.2 Decoding/Sampling Policy

Several decoding strategies for large language models to output tokens

Greedy Search

Greedily selecting the token with the highest probability each time is simple and efficient, but it may lead to overly monotonous and repetitive text.
notion image

Beam Search

Maintain a set of candidate sequences of size . At each step, choose the candidate sequences for the next token prediction, leading to new candidate sequences. Then still choose top candidate sequences to keep, where some candidates may be from the same old beam and some old beams may not generate sequences that are in the new top . This method balances the quality and diversity of the generated output because it can pick up sequences with lower initial probability, but it may lead to the potentially optimal solution being discarded.
notion image
  • Beam Search Code
      • Note that this implementation is a Breadth First Search (BFS) version, and there’s also Depth First Search (DFS) version out there. The outcomes of the two versions differ slightly from each other.
<ins/>

Top-K Sampling

Sampling from the top-k ranked tokens allows other less frequent or lower probability tokens a chance to be selected. Still, each time, only one token is selected, but instead of directly choosing the one with the highest probability, we sample one from the top K tokens according to their corresponding probabilities. It’s totally OK if you want to sample 2 or 3 each time from the top K, with which you are basically combining this method with the beam search.
In many cases, this kind of sampling brings randomness that helps improve generation quality. However, this may lead to the generation of gibberish when the distribution is sharp because some “not likely” tokens may be selected, and may limit the model's creativity when the distribution is relatively flat because more than one candidate token is similarly good to each other but we only sample one.
notion image
Above is an example of the .

Top-P Sampling

This one is similar to the Top-K sampling. But instead of sampling from the top-K tokens, we sample from the smallest set of tokens whose cumulative probability exceeds a predefined threshold .
notion image
Above is an example of the .

Temperature Sampling

Adding a temperature parameter to Softmax is used to adjust the flatness of the token probability distribution output by large language models. The higher the temperature, the flatter the probability distribution, and the more random the output. The lower the temperature, the more extreme the distribution, and the more stable the output. This method can be combined with other sampling methods to adjust the sampling distribution.
Where:
  • is the probability of the token
  • is the logit (unnormalized score) for the token
  • is the temperature parameter
    • When : this is the standard softmax
    • When : the probability distribution becomes flatter, making the output more random
    • When : the distribution becomes sharper, making the output more deterministic
You may notice some implementation like ChatOpenAI(model="gpt-4o-mini", temperature=0), but obviously the temperature cannot be 0. In this case, the temperature=0 indicates greedy decoding.

Best-of-N

By letting an LLM output N responses, and then using a Verifier, ORM, or PRM (PRM can provide step-by-step rewards and act as the overall response scorer) to assign a score to each response, the highest-scoring one is then selected as the final answer.
notion image

Majority Vote

Majority voting: output multiple responses, and select the one with the most agreement as the final answer. This is basically an ensemble of outputs and is commonly used in mathematics or fact questions.
💡
Example:
You ask a language model the question five times independently: "What is the capital of France?". Then the model outputs: Paris, Paris, Lyon, Paris, Marseille. With a majority vote, you will get Paris as the final answer.

Self-consistency

First introduced in the paper Self-Consistency Improves Chain of Thought Reasoning in Language Models, it involves prompting the large language model to output multiple CoT reasoning paths to obtain multiple answers, and then selecting the final answer through a majority voting approach, similar to majority voting. More details can be found at [5.1 Prompt Engineering].
Prev
1.5 Positional Encoding
Next
2.1 Pre-training Definition
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant