AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
9.5 LLM-as-Judge
Problem Background and Progress
In the past year, large language models (LLMs) have made rapid progress across various natural language processing (NLP) tasks, approaching human-level performance. These tasks include question answering, text summarization, dialogue generation, and code generation. Despite these advances, LLMs still frequently produce hallucinations, unreliable, and inconsistent outputs. Therefore, how to evaluate the content generated by LLMs has become a highly valuable and important area of discussion.
In the current field of natural language processing (NLP), assessing the quality of text generation is a crucial challenge. With the emergence of LLMs such as GPT-4 and other advanced models, the capabilities of automatic text generation have made remarkable progress. These models can generate coherent and contextually relevant text and exhibit unprecedented abilities in multi-step reasoning, information provision, and producing accurate and information-rich responses. However, as these models are widely applied, evaluating the accuracy and fairness of their generated text has become increasingly important.
In traditional work, the quality evaluation of text generation mainly focuses on two aspects:
- Whether the generated text reaches a correct conclusion.
- The informativeness and usefulness of the text content.
Below are some commonly used metrics for evaluating text quality:
- BLEU: Used to evaluate the quality of machine translation, but can also be applied to any text generation task. It works by comparing the overlap between machine-generated text and one or more reference texts.
- ROUGE: Mainly used for automatic summarization. It evaluates the content coverage by comparing the overlap between generated summaries and reference summaries.
- METEOR: Similar to BLEU but takes into account synonym matching and the influence of word order on translation quality.
- BERTScore: Uses a pre-trained BERT model to evaluate the semantic similarity between generated and reference texts.
- Perplexity: Evaluates the fluency of text generated by language models. Lower perplexity usually indicates higher text fluency.
The above metrics mainly focus on evaluating the surface form of text, such as word choice and grammatical structure, but provide limited assessment of logical consistency and semantic coherence in language. This is because most of these metrics are based on statistical methods and word-level matching, which make it difficult to fully capture the deep semantic information within the text. For evaluating logical coherence in language, higher-level comprehension and reasoning abilities are required. Logicality often involves understanding argumentation, causality, and logical relationships, which is a relatively complex task. Traditional automatic evaluation metrics struggle to accurately capture these aspects.
Evaluation Frameworks
Critic Models
(Coming Soon) Shepherd: A Critic for Language Model Generation
(Coming Soon) TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks
(Coming Soon) Digital Socrates: Evaluating LLMs through Explanation Critiques
(Coming Soon) CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation
Strong Base-Models
(Coming Soon) LM vs LM: Detecting Factual Errors via Cross Examination
(Coming Soon) A Closer Look into Automatic Evaluation Using Large Language Models
(Coming Soon) Large Language Models are not Fair Evaluators
Progress in LLM-as-Judge Research
.png?table=block&id=2b426e5a-7de0-80d6-918e-df50367eb40d&t=2b426e5a-7de0-80d6-918e-df50367eb40d)
Prev
9.4 Prompting
Next
10.1 Why GRPO training loss starts at 0
Loading...