The Cost of Encrypted Substring Search in MongoDB 8.2
MongoDB 8.2 introduced public preview support for advanced querying on encrypted fields, specifically enabling prefix, suffix, and substring searches. Until this release, MongoDB's Queryable Encryption was limited to equality and range lookups on encrypted data. This new functionality, exposed via aggregation operators like $encStrStartsWith, $encStrEndsWith, and $encStrContains, aims to bridge the gap for use cases that previously required decrypted data, such as searching for patterns within an email address or a product description. However, early testing reveals a significant performance penalty for these advanced encrypted search capabilities.
A direct comparison of insert operations highlights the substantial overhead introduced by enabling substring search on an encrypted field. In a test scenario, inserting 200 documents into a MongoDB collection where a specific field was encrypted and indexed for substring search took 4.6 seconds. For the same 200 documents in a collection with the same field encrypted but only indexed for equality searches, the operation completed in 232 milliseconds. For context, inserting these documents into a collection with no encryption whatsoever took a mere 7 milliseconds. This indicates that enabling encrypted substring search incurs a cost approximately 20 times higher per insert operation compared to encrypted equality search, and over 650 times higher than unencrypted inserts.
Technical Setup and Methodology
The performance measurements were conducted using MongoDB version 8.2.12 in its Community Edition, deployed via Docker (docker run mongo:8.2). The test involved creating a collection and populating it with documents, each containing an 'email' field. This field was then configured and indexed differently across three test scenarios:
- Unencrypted: The 'email' field was stored as plaintext.
- Encrypted Equality: The 'email' field was encrypted using MongoDB's Queryable Encryption, and indexed for equality lookups.
- Encrypted Substring: The 'email' field was encrypted using Queryable Encryption, and specifically indexed to support prefix, suffix, and substring searches using the new aggregation operators.
The primary metric for comparison was the time taken to insert a batch of 200 documents. This specific number was chosen to provide a representative sample size for measuring the impact of the indexing and encryption strategies on write performance. The tests were repeated to ensure consistency, though specific details on the number of repetitions or averaging methods were not provided in the source material beyond the single reported duration for each scenario.
Understanding the Performance Gap
The dramatic difference in insert times stems from the inherent complexity of performing substring searches on encrypted data. When a field is encrypted for equality or range lookups, the database can often leverage specialized index structures that maintain a direct mapping between the encrypted ciphertext and the original value's order or identity. This allows for efficient lookups by comparing encrypted values directly.
However, substring searches, especially those that involve patterns anywhere within the string (like LIKE '%foo%'), are fundamentally different. To support such queries on encrypted data without decrypting the entire field, MongoDB's Queryable Encryption must employ more sophisticated cryptographic techniques. This typically involves using deterministic encryption schemes where possible, but also potentially more complex transformations or specialized index structures that can handle partial matches. The process of encrypting and indexing data in a way that supports these granular searches requires significantly more computational resources and time during the write operation. Each insert involves not just the encryption of the plaintext but also the computation and storage of index entries that can be queried for partial matches, leading to the observed performance degradation.
The new aggregation operators $encStrStartsWith, $encStrEndsWith, and $encStrContains are built upon this underlying mechanism. While they unlock powerful new querying capabilities for sensitive data, the trade-off is evident in the write path. The 4.6-second duration for 200 documents (which translates to 23 milliseconds per document) for encrypted substring search is a substantial increase from the 232 milliseconds for 200 documents (1.16 milliseconds per document) with encrypted equality search. This suggests that the indexing and cryptographic overhead for substring matching is considerable, impacting how quickly data can be added to the database when this feature is enabled.
Implications for Developers and Architects
The findings have significant implications for developers and database architects considering MongoDB's Queryable Encryption, particularly for applications that require both strong data privacy and the ability to perform flexible text searches on encrypted fields. The choice between different levels of encryption and search capabilities now involves a stark performance trade-off.
For applications where data must remain encrypted at rest and in transit, but substring searches on sensitive fields are a core requirement (e.g., searching for specific terms within encrypted customer support tickets, or partial matches in encrypted personal identification details), the new features in MongoDB 8.2 offer a path forward. However, the substantial increase in insert latency must be factored into system design. Workloads that are write-heavy and rely on frequent insertions of data requiring encrypted substring search may face significant performance bottlenecks. Strategies such as batching writes, optimizing document structure, or potentially reconsidering the need for substring search on encrypted fields versus alternative approaches (like indexing a searchable token alongside the encrypted data, if feasible and secure) might be necessary.
Conversely, if the primary need is to protect sensitive data from unauthorized access while still allowing for fast lookups of exact matches or ranges, then the existing Queryable Encryption capabilities for equality and range searches remain highly performant. The introduction of substring search functionality is a powerful addition, but it is crucial for users to benchmark their specific use cases and understand the operational costs before deploying it in production environments. The gap between plaintext performance, encrypted equality, and encrypted substring search is a clear indicator that this advanced feature comes with a premium, not just in terms of potential licensing but also in raw system throughput.
The current state, where encrypted substring search is in public preview, suggests that MongoDB is likely working to optimize this functionality further. However, for now, developers must make informed decisions based on the observed performance characteristics. If write performance is paramount, and substring search is essential, alternative architectural patterns or a careful re-evaluation of data fields eligible for this specific type of encryption might be warranted. The decision point is clear: unlock advanced search on encrypted data, or prioritize write speed.