The Problem with LLM Memory Benchmarks
For a long time, SovanTica held off on releasing benchmark results for LLM memory systems. The reason is simple: most benchmarks offer a single number without any transparency or reproducibility. A blog post might claim "X% recall," but users have no way to verify the setup. They don't know which reader model was used, how the judge scored the results, the allowed context for the retriever, or if an LLM was secretly performing the heavy lifting within the "memory" layer itself. This lack of detail renders the reported number largely meaningless for understanding the actual memory system's performance.
This opacity makes it impossible for developers and researchers to compare systems meaningfully or to trust the reported metrics. Without knowing the exact parameters of the evaluation—the specific reader and judge models, their configurations, the retrieval strategy, and the scoring methodology—any reported performance figure is essentially an anecdote, not a reliable data point.
Introducing LongMemEval-S: A Reproducible Standard
To address this critical gap, SovanTica has developed and is now using LongMemEval-S, a comprehensive benchmark designed for reproducibility. This benchmark provides a complete, auditable framework for evaluating LLM memory systems. The goal is to offer a metric that users can not only trust but also run themselves to validate performance claims.
The LongMemEval-S benchmark consists of a full set of 500 questions. This extensive question set is designed to probe various aspects of memory recall and context utilization, pushing the limits of LLM memory capabilities. By standardizing the question set, the scoring mechanism, and the evaluation environment, LongMemEval-S aims to become a de facto standard for LLM memory performance evaluation.
Engrava 0.6.0 Performance on LongMemEval-S
In August 2026, SovanTica ran Engrava version 0.6.0 through the LongMemEval-S benchmark. The results demonstrate a strong performance for the system. Engrava 0.6.0 achieved an 81.6% micro-average score across the entire 500-question set. When averaged across the six distinct question categories within the benchmark, the score was 81.76%.
This performance is particularly significant because the benchmark run adhered strictly to reproducible standards. The evaluation utilized the canonical LongMemEval scorer, which was pinned to a specific, known upstream commit. This ensures that the scoring logic itself is static and verifiable. Furthermore, the run employed the standard gpt-4o-2024-08-06 model for both the reader and the judge, accessed via the OpenAI API. This specific model version and API access method are critical for reproducibility, as LLM behavior can subtly change even with minor updates.
The retrieval strategy used in this benchmark run was a top_k of 20 retrieved turns. This parameter controls how many of the most relevant retrieved documents or turns are considered by the reader and judge. By specifying this value, the benchmark clearly defines the scope of information available to the system for answering questions, preventing ambiguity about the amount of context provided.
The Significance of Reproducibility
The commitment to reproducibility in LongMemEval-S is a deliberate departure from the norm in LLM benchmarking. Traditional benchmarks often suffer from a "black box" problem, where the methodology is opaque, making it difficult or impossible for external parties to replicate the results. This can lead to inflated performance claims and a lack of trust within the community.
With LongMemEval-S, SovanTica aims to provide a benchmark that is as transparent as possible. By specifying the exact scorer version, the reader/judge model and its version, the retrieval parameters, and the question set, any developer or researcher can theoretically set up the same evaluation environment and arrive at the same or a very similar result. This open approach fosters healthy competition and allows for genuine progress in the field of LLM memory systems.
The implications of such reproducible benchmarks are far-reaching. For developers building applications on top of LLMs, it means they can make more informed decisions about which memory systems to integrate. They can move beyond marketing claims and rely on verifiable performance data. For researchers, it provides a solid foundation for iterating on new techniques and for understanding the true impact of their innovations. It shifts the focus from "what number did they get?" to "how did they get that number, and can I verify it?"
What's Next for LongMemEval-S
SovanTica's release of Engrava's performance on LongMemEval-S is just the beginning. The company plans to continue using this benchmark for its own development and encourages other teams to adopt it. The LongMemEval-S framework, including the scorer and question set, is intended to be an open standard, allowing the community to contribute and improve it over time.
The availability of a reproducible benchmark is crucial for the maturity of LLM memory systems. As these systems become more sophisticated and integrated into critical applications, the need for reliable, verifiable performance metrics will only grow. LongMemEval-S represents a significant step toward achieving that goal, moving the field away from opaque claims and towards transparent, auditable performance evaluation.