5.4 Enterprise-level RAG Practice (I)

How can we search the wanted key information from 10,000+ pages of PDFs within 2.5 hours? For fact check, how do we implement it so that answers are backed by page-level references, minimizing hallucinations?

5.4.1 Key Components

1. PDF Parsing to Plain Text (in JSON format): Customized Docling, an IBM-developed PDF parser.
1. Used GPU acceleration via Runpod to parse 15,000 pages in under 40 minutes.

2. Text Cleaning:
1. Used OCR for non-regularly organized text and documents with Caesar cipher
🔍
The Caesar Cipher is one of the simplest and oldest encryption techniques, and it is a type of substitution cipher. It is named after Julius Caesar, who used this method of encryption in military communications.
The Caesar Cipher encrypts a message by shifting each letter in the alphabet by a fixed number of positions. For example: If the shift amount is set to 3
• Plaintext: A B C D E
• Ciphertext: D E F G H
It has Encryption formula
Where:
• C: Ciphertext character
• P: Numeric value of the plaintext letter
• k: Key (shift amount)
And Decryption formula:

3. Format Conversion:
1. The cleaned JSON format is then converted to Markdown and HTML formats

4. Ingestion:
1. Converted Markdown text into 300-token chunks with 50-token overlap using RecursiveCharacterTextSplitter from LangChain.

To facilitate page reference, each chunk stores its ID and the parent page number is in its metadata.

2. Vecterization using text-embedding-3-large
3. Created separate FAISS vector databases for each company to ensure precise data isolation. 100 companies = 100 vector databases.

5. Retrieval:
1. Retrived top-30 chunk search.
2. Parent Page Retrieval: the full pages corresponding to the most relevant chunks are used as context
3. Reranking pages using LLM scoring (GPT-4o-mini) with structured prompts and weighted scores (70% LLM, 30% embedding relevance).

LLM scoring (in increments of 0.1) 0 = Completely Irrelevant: The block has no connection or relation to the query.

0.1 = Virtually Irrelevant: Only a very slight or vague connection to the query.

0.2 = Very Slightly Relevant: Contains an extremely minimal or tangential connection. …

4. Select top-10 pages

6. Augmentation:
1. Used modular prompts stored in Python scripts.
• System prompts as general instructions
• Pydantic schema as expected response format
• QA examples as few-shot prompting
• Template for provided context and user question

7. Generation:
1. Applied Structured Output (SO) with Chain of Thought (CoT) prompting.
2. Crafted type-specific prompts (e.g., numeric, boolean, names) based on the expected answer format.
3. For complex comparisons (e.g., between companies), used query decomposition.

5.4.2 Hands-on Practice

1. Pull the repo and install necessary packages (Colab recommended)
After which, you may have to restart the session in order to use newly installed versions.

2. Move to the project path and set your OpenAI API
Just replacing the YOUR_OPENAI_API_KEY below with your API

3. Download the documents that you want to ask about
Here we use two papers (ResNet and Transformer) as demonstration

4. Prepare a simple meta info file and create the pipeline
Note that the sha1 could be any string that’s matched to your file name. Previously we save the papers into ResNet.pdf and Transformer.pdf, so we set it like this. If you want to calculate sha1 for it, feel free to go and rename the file names to their corresponding sha1.

5. Parse the paper PDFs!
Then you can find the parsed papers in the JSON format as shown below.

6. Merge the parsed texts
Because the parsed PDFs are in pieces, we need to merge them into pages for later chunking.
You can observe the merged content is ordered by page number.

7. As we said before, then we convert them to the Markdown format (Just for visualization).
The complete markdowns can be found here for the ResNet and the Transformer. You can observe it’s actually reasonably good!

8. Let’s chunk the documents now!
The chunking is done on previous merged JSON format using RecursiveCharacterTextSplitter. You can see that each page is chunked into multiple chunks, each of which does not exceed 300 tokens.

9. The last step before asking is to vectorize them for storage.
After all of the above steps, we will end up with the files shown below. All we need are the .faiss files for retrieval.



<ins/>

10. Prepare your questions and start the RAG!
Here we ask 3 questions, each with a different required output format: a number, a name, and a boolean. Let’s see how this RAG system would perform on them by running the code below.
You may observe outputs like
Open the answer JSON file, you will obtain:
• The answer for the first question: What's the best BLEU score of the Transformer on the English-to-French newstest2014 test?, is given as 41.8, with reference to the 8th page (index 7, starting from 0). Let’s check the paper on page 8: it’s correct!

• The answer to the second question: Who is the first author of the ResNet paper?, is given as Kaiming He, which is referenced on the first page (index 0) and is obviously correct.
• As for the third question: Is the ResNet paper published before the Transformer?, the answer is given as Yes, and we can see the referenced pages include both papers and the corresponding first page.

A complete Colab code of all above sections can be found here

📌

Warning

In this tutorial, we simply replaced the PDFs with ours and didn’t modify the code, and it works fine. But if you plan to use it in real-world scenarios, you will absolutely want to modify the hard-coded prompts, which are specifically customized for the challenge.

For example, here is a prompt for the final answer. You may want to change the company or product to fit your needs. The prompts can be found in this file.

References Enterprise RAG Challenge RAG Challenge Winner Solution Ilya Rice: How I Won the Enterprise RAG Challenge