Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%

How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10

The RAG Reality Check

The standard approach to implementing Retrieval Augmented Generation (RAG) often starts with a simple formula: chunk documents into fixed-size blocks (typically 512 tokens), convert these chunks into embeddings using a model like OpenAI's text-embedding-3-small, retrieve the top k most similar chunks (often k=5), and then feed these into the Large Language Model (LLM) context window. This method works adequately for basic demonstrations and simple use cases. However, it quickly reveals its limitations when deployed in production environments dealing with diverse and complex data.

Real-world data presents significant challenges:

• Legal Contracts: Fixed-size chunking can split critical clauses mid-sentence, destroying contextual integrity and rendering the retrieved information useless or misleading.
• API Documentation: Large, verbose documents with dense information can lead to 1000-token chunks that dilute the relevant signal with excessive noise, making it hard for the LLM to pinpoint the exact answer.
• Customer Support Tickets: Conversational data requires overlapping chunks to maintain conversational flow and context. Fixed windows often break this continuity, leading to fragmented understanding.
• Latency: The cumulative time for embedding queries, performing vector searches, and processing the LLM response can easily exceed one second (e.g., 500ms embedding + 200ms vector search + 300ms LLM), which is unacceptable for interactive applications.

Recognizing these shortcomings, we embarked on a project to rebuild our retrieval layer from first principles. The goal was to move beyond a heuristic-based approach to one that is measured, tunable, and demonstrably improves key metrics like recall and latency.

Chunking: One Size Fits None

The fundamental flaw in the conventional RAG setup is the assumption that a uniform chunk size is optimal. In reality, the ideal chunk size and strategy depend heavily on the nature of the document and the type of query being made. We discovered that a one-size-fits-all approach to chunking is a major bottleneck.

Contextual Chunking Strategies

Instead of fixed-size chunks, we adopted a more nuanced strategy:

• Sentence Window Retrieval: For conversational data or documents where context is sequential, we retrieve a larger chunk (e.g., 1000 tokens) but then use a smaller window (e.g., 128 tokens) around the most relevant sentences for the LLM. This preserves surrounding context without overwhelming the LLM.
• Recursive Chunking: For structured documents like legal contracts or reports, we implemented recursive chunking. This process starts with large chunks and recursively breaks them down based on semantic boundaries (e.g., paragraphs, sections, specific clause markers) until a target size or semantic coherence is achieved. This ensures that related information stays together.
• Metadata-Aware Chunking: We leverage document metadata (e.g., chapter titles, section headings, author information) to create semantically meaningful chunks. For instance, chunks related to a specific chapter on "Security Protocols" are grouped and can be retrieved together.

This flexible chunking not only improved the relevance of retrieved information but also helped in managing the token count passed to the LLM, indirectly impacting latency.

Retrieval: Beyond Top-K

The standard top-k retrieval method, while simple, often fails to capture the full spectrum of relevant information. Retrieving only the k most similar chunks can miss valuable context that might be slightly less similar but still crucial for a comprehensive answer. Furthermore, simple cosine similarity on embeddings can be a blunt instrument for complex information needs.

Introducing Bayesian Optimization for Search

To address the limitations of basic top-k retrieval, we implemented a more sophisticated search strategy leveraging Bayesian optimization. This technique allows us to intelligently explore the search space to find the optimal retrieval parameters dynamically.

Here’s how it works:

1. Defining the Search Space: We identified key parameters that influence retrieval quality and latency, including: chunk size, embedding model choice, retrieval strategy (e.g., hybrid search, MMR), and the number of documents to retrieve before re-ranking.
2. Objective Function: We defined an objective function that balances retrieval recall (e.g., recall@10, meaning finding the correct answer within the top 10 retrieved documents) with query latency.
3. Bayesian Optimization Algorithm: Using libraries like Scikit-Optimize, we employed a Gaussian Process-based Bayesian optimizer. This algorithm intelligently selects the next set of parameters to evaluate based on previous results, aiming to find the optimal combination efficiently. It models the objective function and uses acquisition functions (like Expected Improvement) to decide where to sample next, balancing exploration of unknown regions with exploitation of promising areas.
4. Dynamic Parameter Tuning: Instead of fixed parameters, our RAG system now dynamically tunes these retrieval parameters based on the query characteristics and the objective function. For example, a complex query might warrant a broader search with more documents considered for re-ranking, while a simpler query could use a tighter, faster search.

This approach is akin to a skilled librarian who doesn't just grab the first five books on a shelf but intelligently considers multiple sources, cross-references information, and adapts their search strategy based on the subtlety of your request. The Bayesian optimizer acts as this intelligent agent, learning from each query to refine the search process.

Measuring Success: Recall and Latency

The impact of these changes was significant. By optimizing chunking strategies and implementing Bayesian search for retrieval, we achieved a remarkable 40% reduction in average query latency. More importantly, our retrieval pipeline now boasts a 95% recall@10 metric.

This means that for 95% of queries, the correct or most relevant information is found within the top 10 retrieved documents. This level of performance is crucial for building reliable and effective RAG applications that can handle complex, production-level workloads. The transition from a simple, heuristic-based RAG setup to a data-driven, tunable system is what truly unlocks its potential.

What’s Next?

While we've made substantial progress, the journey to perfect RAG is ongoing. Future work includes exploring more advanced embedding techniques, integrating knowledge graph information for richer context, and further optimizing the interplay between retrieval and generation steps to minimize hallucination and improve factual accuracy.