The Problem: Slow RAG Ingestion
Ingesting documents for Retrieval-Augmented Generation (RAG) pipelines can be a significant bottleneck. One developer found that processing a single document took nearly 50 seconds. This wasn't due to a lack of processing power; CPU metrics showed minimal utilization. The application was simply waiting, sequentially, for each HTTP network request to the AWS Bedrock embedding API to complete.
This sequential, blocking approach creates a performance cliff. Each document chunk requires an embedding call. If these calls are made one after another, the total time is the sum of each individual call's latency, plus any overhead. For a 33-chunk document, this added up to a painful 49.61 seconds.
The Solution: Asynchronous HTTP Requests
The key to unlocking significant performance gains lay not in optimizing the embedding model itself or changing the underlying infrastructure, but in how the requests were managed. By refactoring the blocking embedding module into an asynchronous workflow, the application could initiate multiple HTTP requests concurrently. Instead of waiting for one request to finish before starting the next, the system could fire off all requests and then wait for them all to complete, or process them as they arrive.
This is analogous to ordering food at a busy restaurant. If you order one dish at a time, and the kitchen only makes one dish before taking your next order, the wait will be long. If you can place your entire order at once, and the kitchen works on all dishes concurrently, the overall service time is dramatically reduced.
The specific implementation involved converting the synchronous HTTP calls to AWS Bedrock's embedding endpoint into non-blocking, asynchronous operations. This typically involves using libraries or language features that support asynchronous I/O, such as Python's asyncio and aiohttp. The goal is to keep the CPU busy by allowing network operations to happen in the background while the main thread is free to manage other tasks or initiate more requests.
Benchmark Setup and Results
The benchmark was designed to isolate the impact of this change. The setup was straightforward:
- Model: Amazon Titan Text Embeddings V2 on AWS Bedrock.
- Dataset: 33 text chunks derived from a single document. This provides a consistent workload.
- Region:
us-east-1. This ensures consistent network latency for testing. - Test Scenarios: Two distinct approaches were tested:
- Sequential Blocking Requests: The original method, where each embedding request was made and waited for before the next.
- Concurrent Asynchronous Requests: The refactored method, where multiple embedding requests were initiated simultaneously.
The results were stark. The original approach, using sequential blocking requests, took 49.61 seconds to process all 33 chunks. After converting to asynchronous calls, the processing time plummeted to just 1.56 seconds. This represents a speedup of approximately 31.8 times (49.61 / 1.56 ≈ 31.8).
Crucially, this dramatic improvement was achieved with zero changes to the underlying infrastructure. No new servers were provisioned, no more powerful instances were used, and no changes were made to the AWS Bedrock service configuration. The only modification was within the application code itself, specifically how it interacted with the Bedrock API.
Implications for RAG Pipelines and LLM Integrations
This finding has significant implications for anyone building or operating RAG systems, or indeed any application that makes numerous independent API calls to external services. The latency introduced by sequential network I/O can be a major performance killer, especially as the number of required calls increases.
For developers working with AWS Bedrock or similar managed AI services, adopting asynchronous patterns is not just an optimization; it's often a necessity for achieving practical performance levels. This is particularly true for embedding generation, which is a common preprocessing step in RAG, and for generating text with large language models, which often involves numerous token-by-token calls or batched requests.
The surprising detail here is not the magnitude of the speedup—though 31.8x is impressive—but that such a significant improvement came from a single file change, without any infrastructure cost. It highlights a common pitfall in application architecture: focusing on hardware or model tuning before addressing fundamental I/O inefficiencies.
What nobody has addressed yet is how widespread this issue might be across other managed AI services and what standardized best practices or libraries could be developed to automatically detect and mitigate such sequential I/O bottlenecks in common LLM/RAG frameworks.
If you're building a RAG pipeline that feels sluggish, examine your embedding and LLM API call patterns. Converting blocking calls to asynchronous ones is likely your quickest path to a massive performance boost.