✨ Awesome-Anything

Awesome-AI-Researchs

Awesome-GitHub-Repos

Awesome-AI-Examples

🖼️ Digital Image Processing

DIP I Image Demosaicing and Histogram Manipulation

DIP II Image Denoising

DIP III Edge Detection

DIP IV Digital Half-toning

DIP V Geometric Image Modification

🍃 LLM Components

1.1 Tokenization

1.2 Embedding

1.3 Attention

1.4 FFN, Residual Addition, and LN

1.5 Positional Encoding

1.6 Architecture & Decoding Policy

🌱 LLM Pre-training

2.1 Pre-training Definition

2.2 Pre-training Data

2.3 Pre-training Workflow

2.4 Continued Pre-training (CPT)

2.5 Pre-training Evaluation

☘️ LLM Post-Training

3.1 Supervised Fine-Tuning (SFT)

3.2 Reinforcement Learning

3.3 RLHF: Reinforcement Learning from Human Feedback

3.4 Optimizing PPO-Based Algorithms

3.5 Direct Preference Optimization (DPO)

🍀 LLM Popular Models

🪴 LLM Applications

5.1 Prompt Engineering

5.2 LLM-based Agents

5.3 Retrieval-Augmented Generation (RAG)

5.4 Enterprise-level RAG Practice (I)

5.5 Enterprise-level RAG Practice (II)

🌿 LLM Optimization

6.1 GPU Usage Analysis

6.2 Latency Analysis for Inference

6.3 FlashAttention

6.4 PagedAttention

6.5 Packing for Efficient Training

6.6 PEFT

🌾 LLM Compression

7.1 Quantization

7.2 Pruning

7.3 Knowledge Distillation

🌵 LLM Hands-on Practice

8.1 nanoGPT: Training a GPT from Scratch

8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example

8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis

8.4 OpenRLHF: Reward Model Training and Code Analysis

8.5 OpenRLHF: PPO Tutorial and Code Analysis

🌴 LLM Must-read Papers

9.1 Technical Reports Explained

9.2 MedLLM

9.3 o1 Techinical Paper

9.4 Prompting

9.5 LLM-as-Judge

🌳 LLM Q&A

10.1 Why GRPO training loss starts at 0

10.2 VLM-R1 Reproduction Results

10.3 KL(P||Q) or KL(Q||P)?

10.4 Left Padding or Right Padding?

10.5 What is QK-Norm?

10.6 Why LLM training rarely adopts Dropout?

10.7 Dropout and DropPath

🐝 VLM Image Encoders

11.0 Introduction

11.1 Vision Transformer

📝 MISC.

🤔 Large Language Models Pass the Turing Test.

OpenAI ChatGPT Progression

⏳ Claude Sonnet 4 Test with Sandtris

🔍 What DeepSeek-R1's Surge Teaches Us About SEO

💰 CivitAI’s Payment Issue

🎓 arXivSub, an Academia Assistant

🔗9.1.5 Llama 4

Blog: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

The Llama 4 series is the first open-weight native multimodal MoE (Mixture of Experts) model series, including three model sizes: Scout, Maverick, and Behemoth. Among them, Behemoth is still in training and not yet available. The three names are quite meaningful:

Scout can mean a child soldier/scout

Maverick refers to an NBA team

Behemoth is a giant beast in the Bible.

Model Parameters and Structure

⭐

The Llama 4 series models are all based on the auto-regressive MoE architecture, and use Early Fusion (explain later) to support native multimodal capabilities. Among them, Llama 4 Scout can run on a single H100 GPU (with int4 quantization). Scout and Maverick are both derived from Behemoth’s distilled versions. Llama 4 Behemoth has a total parameter count of 2 trillion (2T), with 288 billion (288B) active parameters. It is still in training and not yet available.

Below is a table of relevant metrics for the two available models:

Category	Llama 4 Scout (17B x 16 Experts)	Llama 4 Maverick (17B x 128 Experts)
Training Data	Includes publicly shared Instagram and Facebook posts under Meta, as well as interactions with Meta AI	Same
Active Parameters	17B	17B
Total Parameters	109B	400B
Input Modalities	Multilingual text and images	Multilingual text and images
Output Modalities	Multilingual text and code	Multilingual text and code
Context Length	10M	1M
Token Count	~40T	~22T
Knowledge Cutoff	August 2024	August 2024

Where “Token Count” refers to the total number of training tokens.

❓

Early Fusion

Early fusion in Llama 4 means that text and image tokens are combined and processed together from the very beginning inside the same transformer layers. This allows the model to:

Learn deep connections between text and visuals at every layer

Train jointly on unlabeled text, images, and videos

Avoid using separate pathways for each modality

It’s a more integrated and efficient way to handle multiple types of data compared to older “late fusion” approaches.

Configuration files settings comparison:

Llama-4-Scout-17B-16E

Llama-4-Maverick-17B-128E

⭐

Based on the config.json file, both Scout and Maverick have:

48 layers

40 attention heads

8 kv_heads

head_dim of 128

Hidden size of 5120, and use SiLU as the activation function.

The MoE implementation uses an interleaved structure between MoE layers and MLP layers. The intermediate size of the MoE layers is 8192, and the MLP's intermediate size is 16384.

The interleaving frequency is controlled by the parameter interleave_moe_layer_step.

For Scout: interleave_moe_layer_step = 1, meaning MoE and MLP layers alternate.

For Maverick: interleave_moe_layer_step = 2, meaning two MLPs are followed by one MoE.

The MoE layer is designed in two parts:

Several Routed Experts

One Shared Experts

The Router activates only one expert at a time, with each MoE layer having an intermediate size of 8192 (shared expert) + 8192 (one activate routed expert) = 16384, which is consistent with the MLP layers. Though Scout and Maverick differ in the number of experts, the number of activated parameters remains the same.

The MoE architecture has higher computational efficiency during training and inference. Under fixed FLOPs assumptions, it achieves comparable quality to dense models.

Pre-training

📌

The Llama 4 series adopts a native multimodal design, using Early Fusion technology to seamlessly integrate text and visual tokens into a unified model backbone (i.e., treating both modalities as a single token sequence). Early Fusion enables large-scale pretraining using a vast amount of unlabeled text, image, and video data. The visual encoder is a modified version of MetaCLIP, and it is trained separately from the frozen Llama model.

Pretraining also adopted FP8 precision for efficient model training. When training the Llama 4 Behemoth model on 32K GPUs, it achieved 390 TFLOPs per GPU.

The total pretraining dataset size reached 30 trillion tokens, more than twice that of Llama 3, covering various types of text, image, and video datasets.

Mid-training (continued training) is applied using new techniques (including specialized datasets for long-context extension) to enhance core capabilities, giving Llama 4 Scout a best-in-class context length of up to 10M tokens.

Configurations for vision encoder part for Llama-4-Scout-17B-16E:

Llama 4 Maverick: Post-training to Enhance Performance

✨

When conducting post-training on the Llama 4 Maverick model, the biggest challenge is balancing multimodal input, reasoning, and conversational ability.

Meta adopted a curriculum strategy, with a specific post-training pipeline:

Lightweight SFT → Online RL → Lightweight DPO

A key lesson learned: SFT and DPO may over-constrain the model, limiting exploration in the Online RL phase and resulting in reduced accuracy, particularly in reasoning, coding, and mathematics.

To address this issue, for each stage::

Lightweight SFT: Meta used Lthe lama model as a judge, removed over 50% of the data labeled as “easy”, and performed lightweight SFT on the remaining harder dataset.

Online RL: A continuous online RL loop is used, alternating model training with filtering to retain only medium-to-hard prompts.

lightweight DPO: The lightweight DPO stage was used to handle corner cases and further refine response quality.

Llama 4 Scout: iRoPE to Enable Long Context Window

Llama 4 calls it the iRoPE architecture, where “i” for interleaved attention layers aimed at supporting infinite context, and “RoPE” for the rotary position embeddings (Check Section 1.5.4 RoPE and ALiBi for more information) used in most layers. This technique mainly consists of two key components:

📌

Interleaved Attention Layers Without Positional Embeddings

The implementation details of alternating attention layers still need to be clarified in a future technical report.

Reference paper: The Impact of Positional Encoding on Length Generalization in Transformers

The reference paper extensively discusses the issue of positional encoding. One of the experiments in the paper shows that removing positional encoding can lead to better generalization at longer sequence lengths.

No positional encoding (NoPE) outperforms all other positional encodings at length generalization of decoder-only Transformers (GPT-style) trained from scratch and evaluated on a battery of reasoning-like downstream tasks. This figure shows aggregate ranking of positional encoding methods across 10 tasks.

📌

Temperature Scaling for Attention during Inference

This refers to adjusting the temperature coefficient used in the Softmax operation of Attention during inference.

Reference paper: Scalable-Softmax Is Superior for Attention

It points out that Softmax suffers from attention fading, whereas the input vector dimensionality increases, the maximum value decreases, causing the attention signal to weaken. To address this, the paper proposes Scalable-Softmax (SSMax). The main idea is to replace the Softmax with the following equation:

Where:

is the input prompt length, which increases as the context size grows,

is the input vector, and usually performs Softmax

is a scalar scaling parameter.

Comparison of Softmax and SSMax showing how SSMax prevents attention fading. As input size increases, Softmax output peaks diminish, while SSMax maintains a maximum near 1. The scaling parameter of SSMax is set to 0.43.

Performance of Llama 4

Appendix - Code Analysis

Meta’s Llama 4 model was released with open weights but without a detailed technical report or official source code. Instead, the most accessible way to use the model is through the Hugging Face Transformers library, which provides an implementation based on the released weights.

It’s important to note that this implementation was developed by Hugging Face, instead of Meta. While not official, it serves as a practical and widely adopted reference for understanding Llama 4’s architecture and usage.

In this section, we’ll analyze key components of Hugging Face’s Llama 4 implementation to better understand the model’s design and behavior in practice. Since the Transformers library evolves rapidly, our analysis is based on Transformers v4.52.0.

To simplify understanding, all code snippets in this section have been streamlined. Auxiliary functions, error handling, and unrelated logic have been removed to focus on the core implementation details.

Image and Text Input Preprocess

Let’s begin by examining the chat template used by the Llama 4 model, which can be found in the tokenizer_config.json of the meta-llama/Llama-4-Scout-17B-16E model, under the chat_template key.

To focus on the essential logic, we simplify the template by removing most of the tool call handling and auxiliary content. Special output tokens are marked in red, and generated text is in green.

A sample output formatted with this template might look like:

From the template, we observe that user image inputs are represented using a special <|image|> token. Let’s now break down how Llama 4 processes these image inputs step-by-step.

Step 1: Example. Image Input with Llama 4

The following example, adapted from the Llama-4-Scout-17B-16E-Instruct page, demonstrates image input usage:

Step 2: Loading the Processor Configuration

The AutoProcessor loads processor_config.json, which defines:

This configuration tells Transformers to use the Llama4Processor class, defined in processing_llama4.py:

So, Llama4Processor internally loads:

AutoImageProcessor: Uses preprocessor_config.json

AutoTokenizer: Uses tokenizer_config.json

This tokenizer defines special image tokens:

Step 3: Token Replacement with apply_chat_template

The processor.apply_chat_template method eventually invokes the __call__ method of Llama4Processor, which replaces each <|image|> token with a series of structured patch placeholders:

The _prompt_split_image method formats tokens like:

Step 4: Image Processing with image_processor

The image processor is defined in image_processing_llama4_fast.py. It performs:

Compute valid image resolutions based on patch count.
Determine the best canvas to resize without distortion.
Resize and pad the image.
Normalize and rescale.
Split image into tiles for patch-level encoding.
Optionally add a global tile.
Return pixel values and aspect ratios.

Transformer Structure: Attention and the Implementation of MoE

In Hugging Face’s implementation of the Llama 4 transformer architecture:

Grouped Query Attention (GQA) is used for efficiency, striking a balance between performance and memory usage. For background, see 1.3.5 MHA to MLA: trade-off between efficiency and performance - GQA.

The Transformer architecture is split into two major components:

A text decoder (used during generation), which adopts a Mixture of Experts (MoE) structure.

An image encoder, which uses a standard MLP layer instead of MoE.

Text Decoder

The text decoder receives only token embeddings as input and outputs generated tokens. The diagram below (simplified from Llama4ForCausalLM) omits auxiliary structures such as normalization and positional embeddings:

Image Encoder

The image encoder is structured as follows:

It includes a pixel shuffle operation to resize image embeddings while preserving information. This function adjusts channel dimensions to perform spatial reshaping:

Finally, the model uses a Llama4MultiModalProjector to project the image encoder’s output into the same embedding space as the text decoder. This projection is implemented as a simple MLP layer.

The model structure is defined in modeling_llama4.py. While functional, this implementation is not optimized for performance or large-scale deployment. The MoE (Mixture of Experts) structure used in the decoder is explored in more detail in:

1.6.1 LLM Architectures - MoE Architecture

4.7.4 DeepSeek-V3

4.8 MoE Series

Here’s a simplified version of the MoE forward pass:

In this implementation, the input is duplicated N times (where N = number of experts), and expert selection is simulated using masking. While this mimics MoE behavior, it is not a true sparse MoE, where all experts are evaluated, regardless of whether they are selected, leading to wasted computation.

MoE Design Considerations: A fully optimized MoE architecture involves several advanced techniques:

Sparse dispatch vs. dense fallback:

Dense fallback (used in Hugging Face): simple to implement and compatible with CPU/GPU, but inefficient due to redundant computation.
True sparse dispatch:

Gather only tokens routed to a given expert.
Compute expert-specific operations on that small batch.
Scatter results back to their original positions.

Multi-GPU load balancing and communication:

Experts are typically sharded across GPUs using expert parallelism.
Requires all-to-all communication to send token embeddings to the correct expert device, and back again.
Needs load balancing to prevent token congestion on any single GPU.

Without such optimizations, overall throughput suffers due to GPU imbalance — some devices sit idle while others are overloaded.

In practical deployment, vLLM is a preferred runtime for serving MoE models. It offers first-class support for expert parallelism:

Experts are sharded across GPUs.

Only the top-k tokens are routed to each expert device.

True sparse computation and communication make it both efficient and scalable.

Overall Model Structure

The complete data flow and integration of image and text inputs in Llama 4 are handled in the modeling_llama4.py file.

The overall architecture can be summarized in the following diagram:

Input Text is processed by apply_chat_template, which injects placeholders (e.g., <|image|>) into the token sequence.

Input Images are processed by Llama4ImageProcessorFast, which resizes and tiles them based on patch size constraints.

The Llama4Processor combines both: it fills in image placeholders <|patch|> with patch-level markers and tokenizes the entire input.

Image patches are fed into the image encoder, producing image embeddings.

These embeddings replace the <|patch|> placeholder during decoding.

The text decoder, a MoE-based transformer, consumes the combined text and image inputs and generates the final output tokens.

This architecture supports seamless multimodal input handling and shows how Hugging Face adapted the Llama 4 model to support image reasoning by bridging vision and language streams efficiently.

CivitAI’s Payment Issue

Awesome-AI-Tutorials

Catalog

Last update: 2025-07-27

🧬 AI Learning Blog 🧬

Anticipated updates:

Must-read papers, and Hands-on practice in general
Digital Image Processing/Mathematics
Vision Language Model