5.3 Retrieval-Augmented Generation (RAG)

5.3 Retrieval-Augmented Generation (RAG)

5.3.1 Introduction

• Inherent Limitations of LLMs:
• Knowledge Limitations: LLMs derive their knowledge entirely from their training data, which usually consists of publicly available information. Consequently, they may lack real-time or non-public data, limiting their effectiveness in certain contexts.
• Hallucination Problem: Since AI model outputs are based on statistical probabilities, LLMs can sometimes produce “confident” but incorrect responses, often referred to as hallucinations. This problem often happens when the model lacks enough context or domain-specific knowledge.
• Data Security: In enterprise applications, uploading sensitive data to third-party LLM services raises security concerns. To solve this, organizations often consider deploying LLMs in secure environments—though this can involve trade-offs of performance, scalability, and cost.

• Key Features of RAG:
• Integration with LLMs for Enhanced Output: RAG leverages the generative ability of LLMs by combining it with retrieval mechanisms. Without the support of an LLM, the retrieval data alone has limited value in generating coherent responses.
• Effective Use of External Data: By connecting to external databases, RAG can access up-to-date and specialized knowledge. This integration improves the general-purpose LLMs.
• Data Privacy and Security Protection: The external databases used by RAG do not participate in the training of the LLM. This separation helps ensure that sensitive data remains secure.
• Variable Performance: The performance of RAG systems depends on multiple factors, including the underlying LLM's capabilities, the quality of the retrieved data, the effectiveness of the retrieval algorithms, and the overall system design.

5.3.2 RAG Workflow and Types

Overall RAG Workflow

1. Knowledge Document Preparation

2. Embedding Model

3. Vector Database

4. Query Retrieval

5. Answer Generation

Note that, in the above 7. Knowledge Return, the returned could be either knowledge vector representation (then it’s concatenated after input embeddings) or corresponding knowledge text form (then it’s tokenized and combined with the original input tokens)

• Knowledge Document Preparation (step 1 in the figure): The first step in creating an efficient RAG system is to prepare the knowledge documents. In real-world scenarios, knowledge may originate from various formats, such as:
• Word documents, TXT files, CSV tables, and Excel sheets
• PDF files, images, and videos
To convert these diverse sources into structured textual data that large language models (LLMs) can understand:
• Document Parsers: Specialized tools like PDF extractors are used to extract text from files.
• Multimodal Models: OCR (Optical Character Recognition) and related techniques are applied to images and videos.
Additionally, since many documents can be lengthy, it is advisable to split them into smaller chunks. This segmentation not only reduces the processing burden on the LLM but also enhances the accuracy of subsequent information retrieval. As mentioned in [2.5.3 Information Retrieval], finding a needle in a haystack won’t be easy.

• Embedding Model (step 2): This component’s core task is to convert text into dense vector representations that capture the key context and semantics. These vectors help the system recognize semantically similar content despite linguistic differences (e.g., synonyms or function words). Common embedding models include:
• Word2Vec
• BERT
• The GPT series
• The BGE series

• Vector Database (step 3): Once the text is transformed into vectors, these representations are stored in a vector database. This type of database is specially designed to handle and index large-scale vector data.

• Query Retrieval: When a user submits a query (step 4), the system follows these steps:
• The query is embedded into a vector using an embedding model (step 5).
• The vector database then searched for semantically similar documents or dialogue records (step 6).
• The most relevant pieces of information are returned for further processing (step 7).

• Answer Generation: Finally, the retrieved information is combined to form an enhanced, context-rich prompt (step 8). This prompt is fed into the LLM for final answer generation (step 9). This phase could also use the context provided by the previous steps (memory storage) to produce context-aware and coherent responses.

Types of RAG

• Naive RAG

• Advanced RAG

• Modular RAG

1. Naive RAG: This classic approach implements a straightforward "retrieve-read" process consisting of three main steps:
1. Indexing: Documents are split into shorter chunks, and an encoder builds a vector index.
2. Retrieval: Relevant document fragments are identified by comparing the similarity between the query and these chunks.
3. Generation: The language model uses the retrieved fragments as context to generate an answer.
The Naive RAG has problems like “Lost in the Middle,” where important context in longer documents might be overlooked.

2. Advanced RAG: Based on Native RAG, advanced RAG incorporates various optimizations that improve both retrieval and generation quality:
• Pre-Retrieval Enhancements: Improved data cleaning, document structuring, and metadata integration enhance consistency and accuracy.
• Query Refinement: Techniques like question rewriting, routing, and expansion help align the query with relevant document semantics.
• Post-Retrieval Enhancements: Methods such as re-ranking retrieved documents and adjusting context window sizes by context filtering or compression to mitigate the “Lost in the Middle” problem.
While these improvements boost performance, they also introduce additional system complexity.

3. Modular RAG: This variant provides the greatest flexibility by integrating extra functional components that allow for customized system designs:
• Additional Modules: Components like Search Module, Memory Module, Extra Generation Module, Task Adaptable Module, Alignment Module, and Validation Module enable dynamic handling of diverse queries.
• Integration with Other Techniques: Methods such as reinforcement learning are employed.
• Flexibility: Modular RAG supports the replacement or recombination of modules based on the specific context of a query.

Summary of RAG Frameworks

5.3.3 RAG Evaluation

Common Challenges in RAG

Evaluating the Retrieval Stage

• MRR (Mean Reciprocal Rank). MRR measures how early the first relevant result appears in a ranked list of retrieved items.
where is the position of the first relevant result for the -th query.
• Interpretation:
• A higher MRR means relevant results appear earlier in the ranking, which is good.
• If no relevant result exists for a query, the reciprocal rank is 0.
• Example: Suppose we have 3 queries, with the first relevant document appearing at ranks 2, 5, and 1, respectively. Their reciprocal ranks are , , and . The MRR is:

Note: ranked_lists is a collection of lists, each list indicating whether each item at a given rank is first relevant (1) or not (0). This measure could be applied to search engines and recommendation systems.

• Hit Rate. Also often referred to as precision in machine learning. It measures how many retrieved documents (in the top results) are actually relevant.

• Recall. Measures how comprehensive the retrieval is.

• Normalized Discounted Cumulative Gain (NDCG): Measures the ranking quality by assigning higher importance to relevant documents at higher ranks.
• DCG: The Discounted Cumulative Gain for the top results is
where is the relevance score for the item at rank .
• IDCG: The ideal DCG is calculated with DCG after ranking documents with the , i.e., the ideal ranking.
• NDCG

Evaluating the Generation Stage

• Qualitative:
• Completeness: Does the answer cover all necessary points?
• Accuracy: Are the facts correct?
• Contextual Relevance: Is the answer aligned with the retrieved context?
• Faithfulness: Does the model “stick” to the retrieved documents rather than hallucinate?

• Quantitative: ROUGE, BLEU, and Other Text Similarity Scores, comparing the generated text to a reference (ground truth).

RAG Evaluation Frameworks

1. Ragas (paper, open-source): focused on evaluating faithfulness, response relevance, and context relevance in Retrieval-Augmented Generation (RAG) systems. Below is a summary of key metrics used in the framework:
• Faithfulness: Captures how factually consistent the response is with the retrieved context. A response is considered faithful if all its claims are directly supported by the retrieved information.
• Response Relevancy: Measures how relevant the response is to the original user input. Higher scores reflect better alignment.
• It involves generating synthetic questions based on the response and comparing them with the user input via embedding similarity or comparing the response with a ground-truth answer.
• Context Relevancy: Checking whether the retrieved context is helpful or not for generating the answer.

All above could involve another LLM for evaluation.

2. LangSmith (closed-source, but includes free plan): a framework for testing, experimenting, and monitoring LLM-powered chains and agents, with native support for LangChain. In RAG evaluation, LangSmith emphasizes flexible, LLM-based evaluators across four key dimensions: Correctness, Relevance, Groundedness, and Retrieval Relevance.

5.3.4 RAG Optimization

Document/Knowledge Preparation Stage

• Data Cleaning: RAG relies on accurate and clean knowledge data. On one hand, to ensure data accuracy, it's necessary to optimize document readers and multi-modal models. This is especially important when processing CSV and other tabular data, as pure text conversion may lose the original structure. Therefore, it’s recommended to preserve structural elements by using separators to separate numbers. On the other hand, knowledge documents need to go through basic data cleaning, which includes:
• Basic Text Cleaning: Normalize text formats, remove special characters and irrelevant information. Eliminate redundant content in the documents.
• Entity Parsing: Eliminate inconsistencies in entities and terms for unified citation. For example, standardize “LLM”, “large language model”, and “language model” into a common term.
• Document Planning: Divide documents by topic. If documents on different topics are either too concentrated or scattered, it becomes difficult for both humans and systems to identify valuable information.
• Data Augmentation: Use synonyms, notations, or even translations in other languages, etc., to enrich the document knowledge base.
• User Feedback Loop: Continuously update data based on user feedback to maintain their authenticity.
• Time-Sensitive Updates: For documents updated over time, establish a mechanism to ensure they are updated efficiently.

PDF Parsing

• Rule-Based Parsing Algorithm

• Analysis Technology Based on Multimodal Large Models

• How to Handle Multi-level Headings

• Other useful PDF File Parsing Tools

Chunking

• Semantic continuity: If chunks are too small, they lose coherence.

• Reducing noise: If chunks are too large, they might contain unrelated content, reducing retrieval precision.

Embedding Model Stage

Vector Database Stage

• Metadata: Many vector databases allow you to store additional metadata alongside your vectors for improving retrieval efficiency. Common metadata fields (e.g., date, author, section title) help refine or filter search results. For instance:
• Dates: Useful for sorting or filtering (e.g., when searching email history, prioritize recent emails). Maybe you should prioritize less recent emails because users choose AI searching just because they forgot when this email was received (maybe they cannot find it, maybe just too long ago). Anyway, dates are important.
• Section titles or keywords: Provide extra context to the retrieval system, improving relevance.

Query Indexing Stage (Retrieval and Ranking)

1. Multi-Index: If a single index doesn’t capture the range of contexts in your dataset, you might build multi-index:
• One index might handle summarization queries, another for direct Q&A, and another for time-sensitive queries.
• A multi-path routing mechanism directs queries to the appropriate index (or indexes), improving efficiency and accuracy.
For example, a query asking “recommend the latest sci-fi movies” might first be routed to a go-viral index, then to an entertainment/movies index.

2. Retrieval/Querying Algorithms

Approximate search is the key: Given massive vector datasets, exact nearest-neighbor searches can be computationally expensive. Approximate Nearest Neighbor Search (ANNS) is often preferred because it’s faster and offers “close enough” similarity results, which is typically sufficient for user-facing systems.

• Filtering: Narrow the dataset based on metadata or other criteria (e.g., date ranges, categories) before performing vector search. This ensures fewer irrelevant vectors are scored.

• Approximate Nearest Neighbor (ANN): Instead of scanning every vector for an exact match, ANN algorithms quickly find “close enough” neighbors. While slightly less precise, this approach is typically fast and accurate enough for most user-facing applications. Below are some common algorithms.

• Clustering (e.g., k-means)
Cluster similar vectors into groups. When a query comes in, the system first identifies the cluster(s) most relevant to the query embedding and then finds the best one within that cluster.
• Locality-Sensitive Hashing (LSH): In traditional hashing algorithms, we usually hope that each input corresponds to a unique output value and strive to reduce duplication of output values. However, in LSH, our goal is the opposite: we want to increase the probability of output collisions. These collisions are the key to grouping/bucketing. LSH narrows the search space by bucketing similar vectors together. Vectors close in space produce similar hash values, so you only search within that bucket.

• Quantization, Hierarchical Navigable Small Worlds (HNSW), etc. More details can be seen in [5.2.2 Mechanism of LLM-Based Agents, Memory], where fast memory access shares a similar spirit.

3. Query Rewriting: In RAG, the user’s query is embedded and matched against the database. Sometimes the query itself may need improvement for better retrieval:
• Rephrasing based on dialogue history: Two similar questions may produce very different embeddings. We can rephrase the original question using an LLM. Additionally, during multi-turn conversations, part of the user’s query may refer to earlier context, so we can merge prior dialogue and the current query, and let the LLM rephrase it accordingly.
• Hypothetical Document Embedding (HyDE): The key idea is to use an LLM to generate a hypothetical answer without external knowledge (and optionally combine it with the original query) for embedding. This can help locate similar documents better.

• Step Back Prompting: If the query is too complex (“Which school did the XXX attend between 2005 and 2010?”), we can generate a step-back version (“educational background of XXX”) and combine it with the original one for retrieval to improve recall.
• Multi-query search / multi-path retrieval: Generate multiple sub-queries for complex questions, each targeting different aspects or angles, then merge the results.

4. Optimize Retrieval Parameters in Vector Database: Vector databases usually let you fine-tune retrieval behavior with parameters such as:
• Sparse and Dense Search Weighting: Sparse search retrieves results based on matching character strings (keywords). In some scenarios where vector search has limitations, keyword search can be used as a complementary approach. A common keyword search algorithm is BM25, which calculates scores based on term frequency. Rare terms are often treated as important keywords and receive higher weights. So, for example, you might weight keyword results at 0.4, and vector results at 0.6, which means, on average, 40% of results are coming from sparse search and 60% from dense search.
• Result quantity (top-K): Determining how many results to retrieve is crucial. More results provide comprehensive coverage (helpful for complex queries), but can introduce noise and increase compute cost.
• Similarity measurement: Many RAG systems use cosine similarity because it measures directional similarity and is invariant to vector length. Check which methods (Euclidean distance, Jaccard distance, etc.) your embedding model supports.

5. Advanced Retrieval Strategies: Sometimes standard retrieval isn’t enough, especially if your documents are large or your queries are complex. In such cases you may consider the following strategies:
• Contextual compression: If an entire document or chunk is large, use an LLM to compress or summarize it to only the relevant parts before passing it on.
• Context Window Expansion: If chunks are small, you risk losing context. When a chunk is matched, also retrieve neighboring chunks for additional context.
• Parent Document Retrieval: Store large “parent” chunks and subdivide them into smaller “child” chunks. Match the query to “child” chunks first, then retrieve the corresponding “parent” chunk for broader context.
• Auto-Merging Retrieval: Similar to parent-child, but more hierarchical. If many leaf chunks are relevant, the system automatically merges or retrieves their parent level.
• Multi-vector retrieval: Represent a document with multiple vectors (e.g., one for summary, others for different sections). This covers various aspects of the content for multifaceted queries.
• Multi-agent retrieval: Distribute sub-queries to different “agents,” each specialized in certain indexes or aspects. Finally, rank or merge the results. Useful for deeply interrelated documents.

Note that there are different versions of this method on the internet. The specific optimization steps to be taken have not yet been a consensus. You may have to explore this based on specific usage scenarios.

• Self-RAG: The LLM decides whether it needs retrieval, conducts searches, and generates answers for each return. Then, scores will be generated via reflection for these answers to evaluate each retrieved document. It then picks the highest-scoring document as the final reference.

6. Reranking models: After initial retrieval, a secondary model (e.g., a reranking model like Cohere) orders or filters results to ensure the final output aligns closely with user intent. This helps push the most relevant documents to the top.

For example, when a user searches for something like “latest sci-fi movie,” the system first retrieves a broad range of results based on keywords—this could include historical articles, sci-fi novel summaries, and news about new movies. Then, in the reranking stage, the model analyzes these results more deeply and pushes the most relevant ones (e.g., current sci-fi movie lists, reviews, or suggestions) to the top. Less relevant or outdated content is ranked lower.

Answer Generation Stage

• Prompting: The way you format your prompt significantly affects the output. Because the decoder predicts the next token based on the input, carefully crafted instructions can reduce hallucinations and ensure the model uses only retrieved data. Example prompt: You are an intelligent assistant. Your goal is to provide accurate information and assist users. Answer only from the provided context and do not make up anything not in the context.
Of course, you can let the model output some subjective answers depending on your needs. You can also employ few-shot examples to help the LLM produce more relevant answers.

• Choosing an LLM: Depending on cost, reasoning ability, context window, or domain, you might opt for:
• Open-source models for customization.
• Commercial models for robust performance.
• Specialized or domain-specific models if you need advanced reasoning in a particular area (e.g. medical, legal, etc.).
Tools like LlamaIndex or LangChain can help build these RAG workflows, providing debugging tools for us to see which documents were retrieved, what context was included, etc.

5.3.5 RAG Potential Directions

Main Industry Directions

• The industry's focus is mainly on content supply. Methods include query rewriting, query routing, chunking, reranking, and interactions involving large models and search engines.

• From the perspective of content summarization, academic papers focus more on methods for eliminating hallucinations. The industry emphasizes more on aspects such as the comprehensiveness and robustness of generated responses, but they usually do not open-source relevant reports or datasets.

Potential Directions

• Retrieval
• Query Expansion
• Objective: Rewrite or refine the original user query.
• Benefit: Increases the diversity of retrieved results, which can improve the comprehensiveness of the final output.
• Chunking + Vector Retrieval
• Objective: Break down documents into smaller chunks (e.g., paragraphs or sections) and utilize vector-based retrieval to locate relevant information at a finer granularity.
• Major retrievals typically rely on Top-N retrieval using both keyword-based and vector-based similarity. Building chunk-level indexing allows the system to extract information more accurately from large documents, handling the Lost in the Middle issue.

• Generation
• Passage Order Optimization in Prompts
• Observation: Large language models pay more attention to the tail portions of the prompt (i.e., they attend to more recent information).
• Tips: Place higher-quality documents toward the end of the prompt.
• Response Data Refinement
• Refine model outputs and human annotations with models like GPT-4, then use the refined results as high-quality SFT data for further training.
• Benefit: Improves factual accuracy, style consistency, and relevance.
• Chain-of-Verification (CoVe)
• Let the model generate an initial “baseline response.”
• Prompt the model to create verification questions related to the facts in the base response.
• Have the model answer these questions independently from the initial context to avoid bias.
• Combine the independently verified facts into a final response to reduce hallucinations.

• Search-Augmented Factuality Evaluator (SAFE)
• As an extension of CoVe, it breaks down the base response into multiple factual statements, strips out irrelevant parts, and uses retrieved results to verify each fact individually. It enables fine-grained judgments on whether each piece of information is supported by retrieved facts, further reducing hallucinations.

• Decoding by Contrasting Layers (DoLa)
• Concept: Focus simultaneously on representations at both upper and lower Transformer layers during the generation process. It strengthens knowledge captured in deep layers while suppressing superficial “noise” from earlier layers, thereby reducing hallucinations without needing fine-tuning or external search.

RAG Products Overview

5.3.6 RAG Hands-On Practices Using LangChain

• OpenAI API (LLM Power): Calls the language model (e.g., GPT-4) to generate responses, providing the core computational intelligence.

• LangChain (Orchestration Engine): Builds, composes, and runs RAG pipelines by managing prompt templates, chains, retrievers, and other workflow elements. This layer simplifies the construction of sophisticated LLM-based applications.

• LangSmith API (Observability): Tracks and logs every aspect of the LangChain operations—from prompts and outputs to retriever steps and errors. This observability is essential for debugging, monitoring, and performance evaluation.

• LangSmith Dashboard (User Interface): Provides a GUI that allows you to inspect the complete execution trace of your RAG pipeline, facilitating easy visualization of each process step. We will use this one later.

Preparation

1. Register and Create a Project: Visit the LangSmith website and sign up. Once registered, create a new project. You will receive an API key that serves as your project’s identifier.

2. Install Necessary Packages: Run the following commands to install the required Python packages:

3. Configure the Environment: Set up the necessary environment variables as shown below:

4. Prepare knowledge, vector database, and question:

Above settings can be reused for all following algorithms, so you just need to run them once.

Example: Multi-Query Retrieval Workflow

1. Query Rewriting: A GPT-4-based agent is used to rewrite the initial user query into multiple variants (e.g., five different queries). This step increases the likelihood of retrieving a diverse and comprehensive set of relevant documents. For each rewritten query, the system retrieves relevant documents accordingly.

2. Retrieval Chain: The results from each query are then combined (unioned) to form a consolidated set of knowledge documents.
In my case, this process resulted in the retrieval of seven unique documents. You may get different results because of the randomness of each run. For more details on why randomness, you can refer to [1.6.2 Decoding/Sampling Policy].

3. Final Answer Generation
The aggregated documents serve as the contextual knowledge for the LLM, which then generates the final response to the user's query.
Outputs:

4. Monitoring with LangSmith: Once the pipeline is done, you can see the running record with indicators like latency, used tokens, cost, etc.



• Dashboard Overview: The LangSmith dashboard offers visual insights into how the initial query was transformed and processed. It allows you to review individual steps such as:
• The rewriting of the initial user query by GPT.
• The subsequent document retrieval steps: 5 retrievers.
• The final answer generation process: get_unique_union

RAG Fusion Using Reciprocal Rank Fusion (RRF)

Note that before we mentioned the Mean Reciprocal Rank (MRR), which requires groudtruth for ranking evaluation. RRF, on the other hand, is used for merging multiple ranked outputs. In this method, we assume the documents returned in each retrieved list are already ordered according to their priority by LLMs.

Decomposition

1. Creating a Decomposition Prompt Template: Begin by defining a prompt template specifically designed for decomposing complex questions.

2. Decomposing the Question: Use the decomposition prompt to break the main question into multiple sub-questions. For each sub-question, execute a retrieval process from the RAG database and then ask the LLM to generate an answer.
Outputs:

3. Creating a Step-by-Step Prompt Template: Once you have the sub-questions and their corresponding retrieved information, build a step-by-step prompt template. This template is used by the LLM to process and synthesize the information from the sub-questions into a coherent final answer.

4. Generating the Final Answer: Finally, prompt the LLM with the step-by-step prompt along with the collected sub-question answers. The LLM processes the information in a structured manner and produces the final, comprehensive response.
Final answers:
Obviously, the final answer kinda distracts from what we ask. This is common when it comes to a too-simple question like what we asked: What is task decomposition for LLM agents? It was decomposed into seven questions, as shown below, extending its original intent.

5. Tracing the runs on LangSmith
You will notice that this method can consume a lot of tokens, even for such a simple question. That’s why I always stick with gpt-4o-mini — it’s cheap. Sorry about that!

Step Back

1. Preparing Few-Shot Examples: To guide the LLM to step back, we first create few-shot prompts using example pairs.

2. Creating the Step Back Prompt: Next, we build a prompt that instructs the LLM to generalize the question.
Outputs:

3. Integrating the Step Back Process into the Response Pipeline: The next step is to incorporate the step back query into the retrieval-augmented generation chain. This involves retrieving context for both the original and the step-back versions of the question and then using these contexts to generate a final answer.
Outputs:
LangSmith Trace

Hypothetical Document Embedding (HyDE)

1. Generating Pseudo-Documents: In this step, we create a prompt that asks the LLM to generate an answer to a given question.
Outputs:

2. Retrieval and Final Response Generation: We first retrieve relevant context using the generated document, and then generate a final answer.
Outputs:
LangSmith Trace: the second row is for retrieval using the generated pseudo-document; the first row is for answering the original question based on retrieved context.