The Limits of Current Retrieval Benchmarks
Information retrieval (IR) benchmarks have long been a cornerstone for evaluating search and retrieval systems. However, as models become increasingly sophisticated, many of these benchmarks face a challenge: they are becoming "benchmaxxed." This means models can achieve near-perfect scores, obscuring real-world performance differences. This situation is particularly problematic for applications where the objective is not just to find relevant documents, but to pinpoint the exact document that answers a specific product-related question. The QontoFAQ benchmark emerges from this need, aiming to bridge the gap between academic evaluation and practical utility.
The core issue with many existing benchmarks is their reliance on broad relevance metrics. While a document might be broadly relevant to a query, it may not contain the precise answer a user is seeking. For instance, a search for "how to reset my Qonto password" might return Qonto's general support page, which is relevant but doesn't directly provide the steps. A more effective system would surface the specific FAQ entry detailing the password reset procedure. QontoFAQ seeks to evaluate systems on this granular level of direct answerability.
This new benchmark introduces a novel metric designed to be more proportional to document relevance in the context of question answering. Instead of a binary relevant/not relevant classification, or a coarse-grained score, the QontoFAQ metric aims to capture how well a retrieved document directly addresses the user's query. This is crucial for product-oriented scenarios where users are often seeking specific instructions, troubleshooting steps, or factual information rather than a general overview of a topic.
Introducing QontoFAQ: Dataset and Metric
The QontoFAQ project comprises two key components: a new benchmarking dataset and an associated evaluation metric. The dataset is built around real-world product questions, mirroring the types of queries users might pose when interacting with a service like Qonto. This grounding in practical use cases is what differentiates QontoFAQ from more abstract or generalized IR datasets.
The development team focused on creating a dataset where each question is intended to be answered by a specific document or a very small, focused passage within a document. This forces retrieval systems to be precise. The process involved curating questions and identifying the definitive answers within Qonto's extensive documentation and knowledge base. The goal was to create a scenario where a model that can accurately identify the single best source for an answer would perform significantly better than one that simply returns a list of generally related articles.
Complementing the dataset is a new metric. While specific details of the metric's mathematical formulation are available in the accompanying article, the conceptual aim is to provide a score that more accurately reflects the user's success in finding the answer. This metric is designed to penalize systems that return documents containing tangential information, even if those documents are broadly relevant. Conversely, it rewards systems that surface the precise source of the answer, much like a well-organized internal knowledge base or a highly effective search engine.
Why This Matters for Embedding Models
The QontoFAQ benchmark is particularly relevant for evaluating embedding models. These models are the backbone of many modern retrieval systems, converting text into numerical vectors that capture semantic meaning. The quality of these embeddings directly impacts the retrieval performance. By using QontoFAQ, developers can gain a clearer understanding of how well different embedding models perform on tasks that require precise, question-specific retrieval, rather than just general semantic similarity.
Traditional IR benchmarks often evaluate embedding models based on their ability to retrieve documents that are semantically similar to a query. However, semantic similarity does not always equate to answerability. Two pieces of text can be semantically similar because they discuss the same broad topic, but one might contain the specific answer while the other discusses it peripherally. QontoFAQ aims to highlight this distinction. A model that excels on QontoFAQ likely possesses a deeper understanding of contextual nuances and the ability to discern specific factual content within documents.
The implications for fine-tuning and selecting embedding models are significant. Teams can use QontoFAQ to benchmark different models and fine-tuning strategies, ensuring they select or develop models that are optimized for their specific use case: answering user questions directly and efficiently. This moves beyond simply optimizing for recall or precision in a broad sense, towards optimizing for user success in finding information.
Broader Implications for Information Retrieval
The QontoFAQ project signals a broader trend in information retrieval research and development: a move towards more pragmatic and user-centric evaluation. As AI models become more capable, the evaluation metrics need to evolve to keep pace. Benchmarks that accurately reflect real-world user goals, such as finding a specific answer to a product question, are becoming increasingly valuable.
This approach could influence how future IR benchmarks are designed. Instead of relying on static datasets and generic relevance judgments, future benchmarks might incorporate more dynamic elements, user simulation, or task-oriented evaluations. The emphasis will likely shift from measuring how well a system can find *any* relevant document to measuring how well it can help a user achieve a specific information goal. This is especially critical in enterprise search, customer support, and internal knowledge management systems, where precise answers are paramount.
The availability of the QontoFAQ dataset and its associated code on GitHub provides a valuable resource for the community. Researchers and practitioners can now use this benchmark to test and compare their retrieval systems, pushing the boundaries of what is possible in accurate, question-focused information retrieval.