Building a RAG (Retrieval-Augmented Generation) demo takes an afternoon. Building a RAG system that doesn't hallucinate or miss obvious answers takes months of tuning. Building a RAG (Retrieval-Augmented Generation) demo takes an afternoon. Building a RAG system that doesn't hallucinate or miss obvious answers takes months of tuning. We have all been there: You spin up a vector database, dump in your documentation, and hook it up to an LLM. It works great for "Hello World" questions. But when a user asks something specific, the system retrieves the wrong chunk, and the LLM confidently answers with nonsense. The problem isn't usually the LLM (Generation); it's the Retrieval. Retrieval In this engineering guide, based on real-world production data from a massive Help Desk deployment, we are going to dissect the three variables that actually move the needle on RAG accuracy: Data Cleansing, Chunking Strategy, and Embedding Model Selection. Data Cleansing Chunking Strategy Embedding Model Selection We will look at why "Semantic Chunking" might actually hurt your performance, and why "Hierarchical Chunking" is the secret weapon for complex documentation. The Architecture: The High-Accuracy Pipeline Before we tune the knobs, let’s look at the stack. We are building a serverless RAG pipeline using AWS Bedrock Knowledge Bases. The goal is to ingest diverse data (Q&A logs, PDF manuals, JSON exports) and make them searchable. AWS Bedrock Knowledge Bases Optimization 1: Data Cleansing (The Hidden Hero) Most developers skip this. They dump raw HTML or messy CSV exports directly into the vector store. This is a fatal error. This is a fatal error. Embedding models are sensitive to noise. If your text contains <div> tags, random hyphens -------, or system-generated headers, the resulting vector will be "pulled" away from its true semantic meaning. The Experiment We tested raw data vs. cleansed data. Raw Data: Direct export from CRM/Salesforce.
Cleansed Data: Removed HTML tags, standardized terminology (e.g., "FAQ" vs "F.A.Q."), and stripped headers/footers. Raw Data: Direct export from CRM/Salesforce. Raw Data: Cleansed Data: Removed HTML tags, standardized terminology (e.g., "FAQ" vs "F.A.Q."), and stripped headers/footers. Cleansed Data: The Result: The Result: Search Accuracy improved by ~30%.
In specific technical domains, accuracy jumped from 59% to 77%. Search Accuracy improved by ~30%. ~30% In specific technical domains, accuracy jumped from 59% to 77%. 59% to 77% The Code: A Simple Cleaning Pipeline Don't overcomplicate it. A simple Python pre-processor is often enough. import re
from bs4 import BeautifulSoup

def clean_text_for_rag(text):
    # 1. Remove HTML tags
    text = BeautifulSoup(text, "html.parser").get_text()
    
    # 2. Remove noisy separators (e.g., "-------")
    text = re.sub(r'-{3,}', ' ', text)
    
    # 3. Standardize terminology (Domain Specific)
    text = text.replace("Help Desk", "Helpdesk")
    text = text.replace("F.A.Q.", "FAQ")
    
    # 4. Remove extra whitespace
    text = re.sub(r'\s+', ' ', text).strip()
    
    return text

raw_data = "<div><h1>System Error</h1><br>-------<br>Please contact the Help Desk.</div>"
print(clean_text_for_rag(raw_data))
# Output: "System Error Please contact the Helpdesk." import re
from bs4 import BeautifulSoup

def clean_text_for_rag(text):
    # 1. Remove HTML tags
    text = BeautifulSoup(text, "html.parser").get_text()
    
    # 2. Remove noisy separators (e.g., "-------")
    text = re.sub(r'-{3,}', ' ', text)
    
    # 3. Standardize terminology (Domain Specific)
    text = text.replace("Help Desk", "Helpdesk")
    text = text.replace("F.A.Q.", "FAQ")
    
    # 4. Remove extra whitespace
    text = re.sub(r'\s+', ' ', text).strip()
    
    return text

raw_data = "<div><h1>System Error</h1><br>-------<br>Please contact the Help Desk.</div>"
print(clean_text_for_rag(raw_data))
# Output: "System Error Please contact the Helpdesk." Optimization 2: The Chunking Battle How you cut your text determines what the LLM sees. We compared three strategies: Fixed-Size Chunking: Split text every 500 tokens. (The baseline).
Semantic Chunking: Split text based on meaning shifts (using embedding similarity).
Hierarchical Chunking: Retrieve small chunks for search, but feed the "Parent" chunk to the LLM for context. Fixed-Size Chunking: Split text every 500 tokens. (The baseline). Fixed-Size Chunking: Semantic Chunking: Split text based on meaning shifts (using embedding similarity). Semantic Chunking: Hierarchical Chunking: Retrieve small chunks for search, but feed the "Parent" chunk to the LLM for context. Hierarchical Chunking: The Surprise Failure: Semantic Chunking We expected Semantic Chunking to win. **It lost.
\ In a Q&A dataset, the "Question" and the "Answer" often have different semantic meanings. Semantic chunking would sometimes split the Question into Chunk A and the Answer into Chunk B. Result: The system found the Question but lost the Answer. Accuracy dropped by 10-18% compared to Fixed Chunking. Result: The system found the Question but lost the Answer. Accuracy dropped by 10-18% compared to Fixed Chunking. Result: 10-18% The Winner: Hierarchical Chunking Hierarchical chunking solved the context problem. By indexing smaller child chunks (for precise search) but retrieving the larger parent chunk (for context), we achieved the highest accuracy, particularly for long technical documents. Business Domain Accuracy: 94.4% (vs 88.9% for Fixed). Business Domain Accuracy: 94.4% (vs 88.9% for Fixed). Business Domain Accuracy: 94.4% Optimization 3: Embedding Model Selection Not all vectors are created equal. We compared Amazon Titan Text v2 against Cohere Embed (Multilingual). Amazon Titan Text v2 Cohere Embed (Multilingual) The Findings Short Q&A (Science/Technical):

Cohere Embed outperformed Titan. It is highly optimized for short, semantic matching and multilingual nuances.
Accuracy: 77.3% (Cohere) vs 54.5% (Titan).


Long Documents (Business/Manuals):

Titan Text v2 won. It supports a larger token window (up to 8k), allowing it to capture the full context of long policies or manuals.
Accuracy: 94.4% (Titan) vs 88% (Cohere). Short Q&A (Science/Technical):

Cohere Embed outperformed Titan. It is highly optimized for short, semantic matching and multilingual nuances.
Accuracy: 77.3% (Cohere) vs 54.5% (Titan). Short Q&A (Science/Technical): Cohere Embed outperformed Titan. It is highly optimized for short, semantic matching and multilingual nuances.
Accuracy: 77.3% (Cohere) vs 54.5% (Titan). Cohere Embed outperformed Titan. It is highly optimized for short, semantic matching and multilingual nuances. Cohere Embed Accuracy: 77.3% (Cohere) vs 54.5% (Titan). Accuracy: Long Documents (Business/Manuals):

Titan Text v2 won. It supports a larger token window (up to 8k), allowing it to capture the full context of long policies or manuals.
Accuracy: 94.4% (Titan) vs 88% (Cohere). Long Documents (Business/Manuals): Titan Text v2 won. It supports a larger token window (up to 8k), allowing it to capture the full context of long policies or manuals.
Accuracy: 94.4% (Titan) vs 88% (Cohere). Titan Text v2 won. It supports a larger token window (up to 8k), allowing it to capture the full context of long policies or manuals. Titan Text v2 Accuracy: 94.4% (Titan) vs 88% (Cohere). Accuracy: Developer Takeaway: Do not default to OpenAI text-embedding-3. If your data is short/FAQ-style, look for models optimized for dense retrieval (like Cohere). If your data is long-form documentation, look for models with large context windows (like Titan). Developer Takeaway: The Final Verdict: How to Build It Based on our production deployment which reduced support ticket escalation by 75%, here is the blueprint for a high-accuracy RAG system: 75% 1. Know Your Data Type Is it Q&A / Support Logs?

Use Fixed-Size Chunking. (Don't let Semantic chunking split your Q from your A).
Use an embedding model optimized for short text (e.g., Cohere).


Is it Manuals / Long Docs?

Use Hierarchical Chunking.
Use an embedding model with a large context window (e.g., Titan v2). Is it Q&A / Support Logs?

Use Fixed-Size Chunking. (Don't let Semantic chunking split your Q from your A).
Use an embedding model optimized for short text (e.g., Cohere). Is it Q&A / Support Logs? Use Fixed-Size Chunking. (Don't let Semantic chunking split your Q from your A).
Use an embedding model optimized for short text (e.g., Cohere). Use Fixed-Size Chunking. (Don't let Semantic chunking split your Q from your A). Fixed-Size Chunking Use an embedding model optimized for short text (e.g., Cohere). Is it Manuals / Long Docs?

Use Hierarchical Chunking.
Use an embedding model with a large context window (e.g., Titan v2). Is it Manuals / Long Docs? Use Hierarchical Chunking.
Use an embedding model with a large context window (e.g., Titan v2). Use Hierarchical Chunking. Hierarchical Chunking Use an embedding model with a large context window (e.g., Titan v2). 2. Clean Aggressively Garbage in, Garbage out. A simple RegEx script to strip HTML and standardize terms is the highest ROI activity you can do. 3. Don't Trust Smart Defaults Semantic Chunking sounds advanced, but for structured data like FAQs, it can actively harm performance. Test your chunking strategy against a ground-truth dataset before deploying. RAG is not magic. It is an engineering problem. Treat your text like data, optimize your retrieval path, and the "Magic" will follow.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

This story will praise and/or roast a product, company, service, game, or anything else people like to review on the Internet.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

3 Proven Strategies to Boost RAG Accuracy Beyond the Baseline

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Developer’s Guide to Fixing Computer Vision Annotations

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

A Developer’s Guide to Fixing Computer Vision Annotations

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps