AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.10 WebSailor
A key step toward Artificial General Intelligence (AGI): Alibaba Tongyi lab releases the first open-source WebSailor, whose performance surpasses DeepSeek R1 and Grok-3 in the information-seeking task, and is the first open-source web intelligence agent to challenge the BrowseComp benchmark.
Background
In Internet information‐seeking tasks, even very powerful LLMs can sometimes become trapped in an “information fog”: when the question is simple and the path clear, the model will often use its memory or one or two searches to find the answer. But when faced with highly uncertain, clue‐ambiguous questions, it is hard to get the correct answer.
For example, if we ask a straightforward question
What is the population of a certain city?, a search engine returns the answer in one search. But if the question is designed to be extremely complex, the relevant clues are scattered across multiple web pages and described vaguely, then neither humans nor AI can easily find a starting point to search. For example:This music is closely tied to a certain South American capital. Its lyricist was awarded a local honorary title in the early 21st century, and its composer once studied at a famous art academy in western Colombia. What is the name of this music?
Such questions demand reading many pages, extracting and assembling clues, and gradually getting the answer. This exceeds both human limits of memory and attention and the capabilities of common open-source models. Currently, open source agents scored almost zero on complex benchmarks, such as BrowseComp.
The challenge of BrowseComp is that it fragments the answer clues into extremely fine pieces and scatters them across ambiguous information sources of different types and eras, creating a huge “information-fog network.” Agents must therefore scour the vast internet for data, filter out irrelevant noise from massive content, and then link all the fragments together via rigorous multi-step reasoning and cross-validation. For example, the next question requires the agent to trace a spy network spanning continents and involving multiple generations:
There is a spy, A, who speaks multiple languages and is mentioned in a 2023 book review about the period of his activities. His spouse is spy B. B operated on different continents and was exposed twice: the first time in a certain European country, where he was arrested by the police along with two Americans, and their spy network was broken up. The second time in Asia, he inadvertently exposed his identity to a group of foreign officials. In addition, spy B was also the handler of another spy, C, whose grandson in 2014 wrote an article about the public release of related intelligence-agency archives. These two spies, A and B, also had a child, who was later arrested as a dissident, pleaded not guilty in court, and ultimately served a prison term shorter than the original sentence. Spy A also assisted a journalist by providing background material for a novel. Question: What is this journalist’s nationality?
In contrast, closed-source systems have shown astonishing performance. OpenAI’s DeepResearch has achieved performance surpassing that of humans in extremely complex information-seeking challenges. These systems can achieve results on BrowseComp that even human experts find hard to match. The key is the ability to reason under uncertainty. However, these closed-source solutions are black boxes. There is a performance gap between open-source models and the leading closed-source ones. This study proposes WebSailor, which uses a whole set of novel post-training methods to greatly improve an open-source model’s performance on complex web-page reasoning tasks.
BrowseComp Benchmark
BrowseComp, proposed by OpenAI, includes 1266 challenging problems that require browsing a large number of websites to solve. It is both challenging (only 29.2% problems are solved by a human trainer within 2 hours of searching, with 86.4% accuracy for the solved questions) and easy to verify. This dataset also embeds a unique “canary” string as a watermark, which is a hidden identifier in data to detect and filter it from training corpora, helping prevent this test data leakage during training.
The performance reported in this paper shows that Deep Research greatly outperforms all other GPT series models.
Calibration Error
The Expected Calibration Error (ECE) mentioned in the BrowseComp paper required the model to provide a confidence score of the answer:
And ECE measures the gap between the confidence given by the model and the real accuracy. For example, if the model believes that the confidence is 50%, and the accuracy of the answer is also 50%, then we say ECE is equal to 0. To be specific:
- Bin the N predictions into confidence bins (e.g., 0–10%, 10–20%, …, 90–100%).
- In each bin b, compute
- The average confidence
- The empirical accuracy
- Weight the absolute gap by the bin’s sample fraction , and sum over bins:
denotes the set of all examples whose self-reported confidence falls into the confidence bin .
The table above shows that models with browsing tools like GPT-4o w / browsing and Deep Research have higher calibration errors, which suggests that models are over-confident in the answer from web browsing, even if it is wrong.
Method - Introduction
WebSailor’s technical approach introduces the entire process from data to training:
- Create SailorFog-QA dataset: It generates large-scale synthetic complex task data with high uncertainty.
- Generate Reasoning Path Data and Cold-Start Training: Open-source reasoning models are used to obtain rollout data (multi-turn tool-call traces, until the model finishes the question) and reconstruct the reasoning process. Followed by RFT cold-start training based on Qwen-2.5-72b-instruct and Qwen-2.5-32b-instruct.
- RL Training by DUPO: Finally, it used RL algorithms DUPO to further enhance the model’s decision-making capabilities.
Through this series of methods, WebSailor has significantly improved open-source models’ performance in complex information-seeking tasks, fully surpassing all previous open-source solutions on benchmarks such as BrowseComp-en/zh. We will introduce the three methods in the following parts
1. SailorFog-QA dataset - Large-Scale High-Uncertainty Tasks
The three levels in information-seeking tasks are defined:
- level-1: Tasks with low uncertainty and easy to reduce uncertainty, which include questions that the model can answer through its knowledge or with a simple one-time web search.
- level-2: Such as multi-hop QA, where initial uncertainty is relatively high, but there is a clear solution path. Although multi-step reasoning is needed, the related entities have clear logical connections, and the uncertainty can be systematically reduced through structured reasoning steps.
- level-3: The focus of this study, problems with high uncertainty that are difficult to eliminate. In such tasks, the relationships between entities are complex, making it hard to predetermine reasoning paths. Solving these problems requires creative exploration and novel reasoning methods.
To make the model learn complex reasoning patterns that surpass humans, it first needs to be exposed to enough highly uncertain problems. So, the WebSailor team constructed a large-scale synthetic dataset, SailorFog-QA. This dataset is generated through Subgraph Sampling and Obfuscation, specifically designed to train the model’s search and reasoning capabilities in environments with extreme uncertainty.
1. Random Walk Graph Construction to Create Complex Knowledge Structures
Researchers simulate the random walk method to crawl information from real web pages and build knowledge graphs. They first select fuzzy entities from the Wikidata Query Service (SPARQL) as starting points to ensure the questions have hidden complexity. Then, the knowledge graph is expanded randomly across the entire internet, adding more related entities and relationships to form a highly non-linear, densely connected knowledge network. Unlike traditional multi-hop reasoning’s linear, chained structure, this random graph has no predefined solution path. It creates a foggy information space for the model, forcing it to develop creative exploration strategies.
2. Subgraph Sampling to Generate Questions, Adding Uncertainty Through Obfuscation of Details
After obtaining a complex knowledge graph, randomly sample subgraphs from it to generate question-answer pairs accordingly. These questions involve multiple entities and relationships within the subgraph, and the questions and answers often span multiple domains, forming a rich and diverse collection of information.
To further increase difficulty, WebSailor deliberately applies ambiguity to the question content. For example, replacing precise dates with vague periods
in the early 2010s, hiding part of a person’s name an institution founded by someone with the initial 'F', or replacing precise numerical values with descriptive phrases a market share of less than 1%.This obfuscation increases the initial uncertainty, preventing the model from finding the answer through simple matching or lookups. It must understand, reason, and sift through large amounts of information to extract the clues.
The generated SailorFog-QA data has the following advantages:
- Close to real internet distribution: All questions come from real internet content, ensuring that the challenges the model faces during training are close to real-world web environments. The model needs to find answers in massive and messy information, rather than dealing with idealized synthetic data. It also avoids having all information come from a single source like Wikipedia.
- Diverse complex reasoning patterns: Different subgraph topologies produce various question types, covering multi-step deduction, compositional reasoning, comparative analysis, and other complex reasoning needs. This forces the model to practice a broad range of reasoning skills rather than being limited to a single approach.
- Scalable expansion: Because multiple sub-questions can be sampled from a single graph, and the graph itself can be randomly expanded, SailorFog-QA can be generated with high scalability. It can non-linearly create a massive number of challenging questions, which provide enough data for large-scale training.
Through the above strategies, this study generated a massive number of training questions with very high difficulty. During testing, some questions even required up to 40 tool calls by OpenAI’s o3 to find the answer, which demonstrates the high initial uncertainty of SailorFog-QA tasks and the long reasoning chains needed to answer them.
2. Generate Reasoning Path Data and Cold-Start Training
To make the model learn how to solve these complex problems, it is also necessary to train it using solution trajectories. This study leveraged open-source LRMs (such as QWQ and DeepSeek R1) to attempt to answer SailorFog-QA questions and collected their interaction trajectories with the environment. Although open-source models have low accuracy on some complex problems, by using rejection sampling, sufficient cold-start data can still be obtained. However, the raw reasoning outputs from these models are still not suitable for direct fine-tuning:
- It exhibits highly fixed and verbose thinking processes. If the models to be trained imitate these patterns, it may limit the model’s ability to develop flexible, exploratory strategies.
- Long-period web tasks requiring dozens of tool calls, their verbose reasoning chains can easily exceed the context window, reducing both performance and readability.
To address these issues, this study proposed a new approach:
Reconstructing Reasoning from Expert LRM Trajectories
The original path generated by LRMs can be represented by , where:
- is the reasoning process, or thinking process
- is the action, or the model’s output
- is the observation, or the output from the tool
Because the reasoning process is very long, this study removes the reasoning process and gets only the successful action-observation sequence . Given the action and observation, this study reconstructs the reasoning path. Suppose the history process step is:
The model is prompted to generate a new thought , which serves as a concise and reasonable reasoning process for action :
By iteratively applying this for each step, a complete and concise reasoning trajectory is generated as training data.
With high-difficulty question-answer pairs and their corresponding solution trajectories prepared, this study performs RFT cold-start training. RFT (Rejection Sampling Fine-Tuning) selects high-quality solution trajectories to perform initial alignment of the model. In recent years, some researchers have suggested skipping SFT before RL training, but in such complex web searching environments, the team found that appropriate RFT cold-start is crucial. This is because the rewards in these tasks are extremely sparse. If the model is not first taught the basics of tool usage and thought structuring, it is nearly unable to solve any problem during direct RL. WebSailor used only a few thousand high-quality trajectories for cold-start RFT, which effectively allowed the model to learn the basic ReAct-style reasoning and tool-use ability. This equips the model necessary foundation for the subsequent RL stage.
3. DUPO: An Efficient RL Training Framework
After the cold-start RFT, WebSailor entered the RL phase to further enhance the model’s decision-making strategies. In complex web searching tasks, Agent RL is extremely challenging, where each rollout involves multi-turn interactions with the browser environment (searching, reading web pages, etc.), and may contain dozens of steps. Such a multi-turn tool call leads to long training times per iteration and low sample efficiency. Traditional RL methods are very slow without improvements
To address this, this study proposed a new efficient RL algorithm, Duplicating Sampling Policy Optimization (DUPO), based on DAPO (See Section 3.4.5 DAPO). DUPO introduces two dynamic sampling strategies during training, greatly improving training efficiency. It ensures that even in tool-intensive environments, the training can iterate quickly.
DUPO’s core ideas:
- Filtering out simple samples before training: Before starting formal RL training, over simple QAs (cases that the model can already solve easily) are removed. These cases have little contribution to strategy improvement and instead waste training resources. By pre-filtering, training can focus mainly on challenging Q&As.
- Duplicate non-zero standard deviation trajectories during training: For each training batch, DUPO monitors the performance differences of each trajectory. If the answers to certain problems in the same batch show uncertainty (large variation in output), it indicates these problems have learning value. Instead of filling the batch with unrelated samples, duplicate these samples that are not learning well multiple times and place them back into the current batch for additional training. This allows the model to practice these tough cases multiple times in a single iteration, increasing training intensity for difficult samples. For samples that the model has already learned or completely has no clue, no extra effort is wasted. This dynamic duplication strategy ensures every batch is fully utilized.
Compared with DAPO’s (See Section 3.4.5 DAPO) dynamic sampling that simply fills batches, DUPO increased the RL training speed of complex agents by about 2-3 times. During training, WebSailor also employed a strict reward design: trajectories are evaluated from two aspects, format correctness and answer accuracy.
The model can only receive high rewards if it both follows the predefined format (such as correctly using
<think> and <tool_call> tags) and ultimately finds the correct answer. This design avoids reward hacking and encourages the model to improve the efficacy of its reasoning chains.Experimental Results
WebSailor surpasses a series of open-source and closed-source models such as DeepSeek R1, GPT-4.1, and Grok-3. After the training steps above, WebSailor performed well on multiple benchmarks, becoming the best open-source web agent. The study evaluated WebSailor on challenging benchmarks such as BrowseComp-en, BrowseComp-zh, XBench-DeepSearch, and GAIA.
WebSailor’s capabilities validated the authors’ hypothesis: To make the model learn general-purpose reasoning and planning strategies, the model should experience highly uncertain problems during training.
Compatibility with Simple Tasks
WebSailor was trained only on high-difficulty data, while according to the definitions, BrowseComp-en/zh, GAIA, and Xbench all belong to level-1 or level-2 tasks. To verify whether WebSailor still performs well in simpler tasks, this study evaluated it on a subset of SimpleQA. The complete SimpleQA dataset contains 4,326 QA pairs. Since testing the entire dataset is time-consuming, it randomly sampled 200 pairs for evaluation. This benchmark is characterized by high accuracy and fact-based, simple questions, which are still challenging for advanced LLMs.
As shown in the results below, almost all Agent-based methods outperform direct answering. Among them, WebSailor’s performance surpassed all other methods, demonstrating strong compatibility and efficacy even on simple tasks.
Industry Significance
- WebSailor has greatly reduced the performance gap between open-source and closed-source information-seeking models. WebSailor has already open-sourced part of the SailorFog-QA dataset and will soon open-source the model checkpoints, which can benefit the resource-limited teams and researchers.
- Second, WebSailor provides a general workflow that can be adapted to problems in other domains. The strategy of “high-difficulty task synthesis + small-scale cold start + efficient RL optimization” has strong generalizability. In the future, the open-source community can refer to WebSailor’s approach to tackle more super difficult tasks across various domains, such as complex reasoning Q&A in open domains, academic knowledge discovery, and even cross-modal data integration.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...