Manticore Search Embraces UUIDs for Document IDs

Manticore Search, a high-performance full-text search engine, has introduced native support for Universally Unique Identifiers (UUIDs) as document IDs, starting with version 28.5.0. This move aims to streamline data management by allowing developers to use the same UUID across their primary databases and search indexes, eliminating the need for separate ID mapping or synchronization layers. This practical guide explores how to implement and leverage UUIDs within Manticore.

The primary benefit of using UUIDs as document IDs is the simplification of data synchronization. When a UUID already exists in the primary database, it can be directly used as the identifier in Manticore. This avoids the complexity of managing two distinct ID systems, reducing potential errors and development overhead. Manticore's implementation allows for seamless integration, treating UUIDs as first-class citizens for document identification.

Manticore Search table schema definition for UUID document IDs

Defining the UUID ID Field

To configure a table in Manticore Search to use UUIDs as its primary document ID, the field must be declared with the type uuid and designated as the id field. The schema definition for a Real-Time (RT) table is straightforward:

CREATE RT MANTICORE_TABLE_NAME (
    id uuid,
    field1 text,
    field2 bigint,
    INDEX field1(field1)
) PRIMARY KEY (id)

In this example, id uuid specifies that the document ID will be a UUID. The PRIMARY KEY (id) clause explicitly designates this field as the primary key for the table. Other fields, such as field1 (text) and field2 (bigint), can be defined as usual, along with their respective indexes. The core structure of an RT table remains unchanged, with the primary modification being the type of the ID field.

Core Operations with UUID IDs

Manticore Search supports standard SQL operations and a JSON API for interacting with tables, including those using UUID IDs. When inserting or updating documents, the UUID must be provided. If the UUID is not provided during an insert operation and the ID field is configured to auto-generate, Manticore will create a new UUID for the document. However, for seamless synchronization with external databases, it is recommended to supply the UUID explicitly.

SQL API Operations

Performing operations via the SQL API is consistent with standard Manticore usage. For instance, inserting a document with a predefined UUID looks like this:

INSERT INTO MANTICORE_TABLE_NAME (id, field1, field2) VALUES ('123e4567-e89b-12d3-a456-426614174000', 'example text', 12345)

Retrieving a document by its UUID is equally direct:

SELECT field1, field2 FROM MANTICORE_TABLE_NAME WHERE id = '123e4567-e89b-12d3-a456-426614174000'

The <generated UUID> placeholder in Manticore's responses indicates a UUID generated by Manticore itself. Users should not copy this placeholder string; instead, they must use the actual id value returned from their own query results when performing subsequent operations.

JSON API Operations

The JSON API provides an alternative interface for interacting with Manticore. Inserting a document via JSON would involve a request similar to this:

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "field1": "example text",
  "field2": 12345
}

The structure for querying documents via the JSON API also accepts the UUID as a parameter:

{
  "query": {
    "id": "123e4567-e89b-12d3-a456-426614174000"
  }
}

Bulk Loading with UUIDs

For ingesting large volumes of data, Manticore's /bulk API endpoint is essential. This endpoint supports processing multiple documents in a single request, significantly improving ingestion performance. When using UUIDs as document IDs, each document within the bulk request must include its UUID.

A sample /bulk request payload would look like this:

POST /bulk
Content-Type: application/json

[
  {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "field1": "document one",
    "field2": 100
  },
  {
    "id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
    "field1": "document two",
    "field2": 200
  }
]

Manticore processes each JSON object in the array as a separate document. The id field within each object is crucial for correctly indexing the document. If an id is missing or malformed for a specific document in the bulk request, Manticore may generate a new ID or flag the operation as an error, depending on the configuration and the nature of the missing data.

Implications for Data Management

The introduction of native UUID support for document IDs in Manticore Search offers a robust solution for applications that already utilize UUIDs for their primary data storage. This eliminates the need for complex ETL processes or custom ID mapping services, reducing system complexity and potential points of failure. Developers can now synchronize data between their primary SQL or NoSQL databases and Manticore Search with greater ease and confidence.

This feature is particularly beneficial for distributed systems or microservices architectures where maintaining consistent identifiers across various services and data stores is critical. By using UUIDs, Manticore aligns itself with modern data management practices, making it a more attractive option for scalable and interconnected applications. The ability to directly map database UUIDs to search index IDs simplifies querying, debugging, and data reconciliation. This is not just an incremental feature; it represents a fundamental improvement in how Manticore handles document identity, making it a more integrated and less intrusive component in a developer's stack.