The Challenge of Reproducing Retrieval Baselines
Reproducing state-of-the-art retrieval models, particularly those used in Retrieval Augmented Generation (RAG) systems, often conjures images of massive GPU clusters and terabytes of data. However, the reality for many developers and researchers is far more constrained: a standard laptop, perhaps with 16GB of RAM, and a desire to experiment without incurring significant cloud costs. This article details a practical approach to reproducing three fundamental retrieval baselines—BM25, Dense Retrieval (using Sentence-BERT), and SPLADE—on such limited hardware. The goal is not just to get them running, but to understand the practicalities, the inevitable crashes, the necessary fixes, and how to verify their performance.
Many RAG implementations rely on robust retrieval mechanisms to fetch relevant context before a large language model generates a response. BM25, a classic term-based scoring algorithm, remains a strong baseline. Dense retrieval methods, leveraging neural networks like Sentence-BERT, capture semantic similarity. SPLADE, a more recent sparse representation technique, aims to combine the interpretability of sparse methods with the semantic power of dense ones. Successfully running these on a modest machine democratizes experimentation and provides a crucial sanity check for more complex RAG pipelines.
Reproducing BM25 on Limited Hardware
BM25, a probabilistic model, is computationally less demanding than its neural counterparts. The primary challenge on a 16GB MacBook is not CPU or GPU power, but memory management when indexing large datasets. Libraries like rank_bm25 in Python provide a straightforward implementation. The process involves tokenizing a corpus of documents and then building an index. For a corpus that might exceed available RAM if loaded all at once, a common strategy is to process documents in batches. This means reading a chunk of documents, tokenizing them, and then updating the BM25 index incrementally.
A key hurdle encountered is the memory footprint of the inverted index. If the corpus is very large, the in-memory representation of the index can quickly overwhelm 16GB of RAM. Solutions include using more memory-efficient tokenization (e.g., avoiding unnecessary intermediate data structures), or, for truly massive datasets, exploring disk-based indexing strategies, though this significantly impacts retrieval speed. For this reproduction, the focus was on a moderately sized dataset (e.g., a few thousand documents) where careful batching and Python's garbage collection could manage memory. Score checking involves comparing retrieval results against a small, manually curated ground truth set to ensure the indexing and scoring logic is sound.
Implementing Dense Retrieval with Sentence-BERT
Dense retrieval, typically using models like Sentence-BERT, involves encoding documents and queries into dense vector embeddings. The retrieval process then becomes a nearest neighbor search in this high-dimensional vector space. While the inference for a single document or query is fast, encoding an entire corpus can be memory-intensive and time-consuming, especially if the model itself is large.
The standard approach involves loading a pre-trained Sentence-BERT model (e.g., from the sentence-transformers library). The corpus is then iterated through, encoding each document into a vector. Storing these vectors requires significant memory. A dataset of 100,000 documents, each encoded into a 768-dimensional vector, would require approximately 100,000 * 768 * 4 bytes (for float32) = 307MB per dimension, totaling around 230GB if all vectors were held in memory simultaneously. This is clearly unfeasible on a 16GB MacBook. The solution lies in batching and efficient storage. Documents are processed in batches that fit within available RAM. The resulting embeddings are then saved to disk, often in a format optimized for fast I/O, such as NumPy arrays or specialized vector databases (though for this reproduction, simple NumPy arrays were sufficient). Libraries like FAISS or Annoy can be used for efficient similarity search, but even simple cosine similarity calculations on smaller batches can be performed directly.
The main challenge here is not just memory but also the GPU requirement for faster encoding. Running Sentence-BERT on a CPU is significantly slower. However, for a moderately sized corpus, batch encoding on the CPU is still feasible within a reasonable timeframe. The fix often involves carefully tuning the batch size: large enough to leverage some level of parallelization but small enough to avoid out-of-memory errors. Verifying performance involves encoding a small set of known queries and documents and checking if the similarity scores align with expectations, ensuring the model is behaving as intended.
Reproducing SPLADE
SPLADE (Sparse Lexical and Expansion Model) is an interesting hybrid. It uses a BERT-based model to produce sparse, high-dimensional vectors where each dimension corresponds to a token in the vocabulary, and the value indicates the token's importance for the document. This offers some interpretability while capturing semantic meaning. Reproducing SPLADE involves fine-tuning a BERT model for this specific task.
The primary resource constraint for SPLADE is the memory required to load and fine-tune a BERT-sized model. Even smaller BERT variants can consume several gigabytes of RAM and VRAM. On a 16GB MacBook, this means using smaller BERT models (e.g., DistilBERT or TinyBERT variants) or employing techniques like gradient accumulation and mixed-precision training if a GPU is available (though often not on standard MacBooks). The process involves a training dataset (often derived from existing retrieval benchmarks), a BERT model, and a custom loss function designed to encourage sparsity and relevance.
The