AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.9 Seed 1.6
Seed 1.6 is a general-purpose model series that integrates multimodal capabilities. It supports adaptive deep thinking, multimodal understanding, graphical interface operations, and enables deep reasoning over a 256 K tokens context. Seed 1.6 primarily explores the Adaptive CoT technique, allowing the model to automatically trigger its chain-of-thought process based on problem difficulty, achieving a balance between model and reasoning performance.
Currently, there is no technical report, and only the official website’s introduction link. Further details will be added once available.
Reference: Section Seed-Thinking-v1.5
Pre-training
Pre-training with Integrated Multimodal Capabilities
Seed 1.6’s continued pre-training integrates multimodal capabilities, supporting both text and vision input. It is divided into three main stages:
- Pure Text Pre-training. In this stage, the training data consists mainly of web pages, books, papers, code, etc. Various rule-based and model-based data cleaning, filtering, deduplication, and sampling strategies are applied to improve the quality and knowledge density of the pre-training data.
- Multimodal Mixed Continual Training (MMCT). This stage further boosts the knowledge and reasoning density of the text data by increasing the share of academic, code, and reasoning-type content. At the same time, visual data is included and trained with high-quality text data.
- Long-context Continual Training (LongCT). In this stage, long context data of varying lengths is used to gradually raise the model’s maximum sequence length from 32 K tokens to 256 K tokens.
Below are the evaluation metrics for the Seed 1.6 Base LLM:
Post-training: Enhanced Reasoning with Adaptive CoT
In the post-training stage, two variants were developed: Seed1.6-Thinking and Seed1.6 (Adaptive CoT).
- Seed 1.6-Thinking focuses on integrating all VLM capabilities and then using longer CoT processes to achieve high reasoning performance.
- Seed 1.6 (Adaptive CoT) employs a dynamic reasoning technique to compress the CoT length while preserving output quality, striking a balance between efficiency and effectiveness.
Multimodal Integrated Reasoning
During training, Seed1.6-Thinking employs multi-stage RFT (Reinforcement Fine-Tuning) + RL iterative optimization, and each round of RL starts from the previous round of RFT. For the data generated by RFT in each round, a multi-dimensional reward model was used to select the best answer. Based on Seed1.5-Thinking (Section Seed-Thinking-v1.5), the training scale was expanded, adding a large amount of high-quality training data (including Math, Code, Puzzle, and Non-reasoning data), which enhances the model’s reasoning depth on complex problems. It also integrates VLM into the multi-dimensional capability of the model to provide clear visual understanding capabilities.
To further strengthen the model’s reasoning ability, Seed1.6-Thinking also employs parallel decoding, which can use more reasoning tokens before answering. This is also a method to scale up the model’s performance without retraining. This study found that for relatively difficult tasks, parallel decoding significantly improves model performance. For example, on the high-difficulty benchmark Beyond AIME, Seed1.6-Thinking achieved an 8-point improvement. Improvements can also be observed in code task benchmarks.
Performance improvement for including parallel decoding.
Dynamic Reasoning Ability
Deep reasoning models achieve high performance in difficult math and coding tasks because the long CoT greatly improves the model’s reasoning ability. However, it also introduces overthinking problem. Deep reasoning models output long CoT for both text and visual tasks. During the CoT scaling process, the model generates too many useless “tokens”, which do not improve the accuracy but increase the reasoning burden.
Seed1.6 proposed Adaptive CoT (AdaCoT), providing three reasoning modes: Full CoT, No CoT, and Adaptive CoT. These modes greatly reduce CoT length while maintaining performance. To achieve dynamic reasoning ability, Seed 1.6 introduced a new reward function during RL training, which penalizes overlong reasoning and rewards appropriate reasoning. With this multi-mode "dynamic reasoning ability", the model can use the following reasoning modes during inference via prompt settings:
- Full Reasoning (FullCoT): Performs reasoning for all prompts before answering. Performance matches Seed 1.6-Thinking, with shorter CoT lengths.
- No Reasoning (NoCoT): Does not perform reasoning for any prompt, answers directly, with higher efficiency.
- Adaptive Reasoning (AdaCoT): A combination of the above two modes. The model automatically chooses whether to reason based on the specific prompt.
The table below shows the performance of the dynamic reasoning model on public test sets:
On tasks of simple or middle difficulty, Seed1.6-AdaCoT exhibited different CoT trigger rates:
- On MMLU, the CoT trigger rate is 37%
- On MMLU-Pro, it reached 70%, showing a correlation with task difficulty.
In these tasks, the model saves token usage without performance loss.
On harder tasks like AIME and BeyondAIME, Seed1.6-AdaCoT’s CoT trigger rate reached 90–100%, and its performance matched Seed1.6-FullCoT, showing that adaptive reasoning preserves the reasoning benefits of long CoT. At the same time, this study found that the AdaCoT approach is also valid across various multimodal scenarios.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...