Trino, Spark, and DuckDB Query Apache Iceberg Tables Differently

The Promise of Unified Data Access

Apache Iceberg has emerged as a critical table format for modern data lakes, primarily because it decouples table structure and data from the underlying storage system. This separation allows multiple compute engines—like Trino, Spark, and DuckDB—to query the same table without requiring data duplication or complex ETL processes to move data between engine-specific formats. The core value proposition of Iceberg lies in its ability to provide ACID transactions, schema evolution, and time travel for data stored in object storage like S3 or ADLS. However, the practical implementation of querying a single Iceberg table across these diverse engines often reveals subtle yet significant differences in how each engine interacts with Iceberg's metadata and cataloging mechanisms.

The challenge for data engineers and analysts working with a lakehouse architecture is not just about *if* an engine can read an Iceberg table, but *how* it does so, and what implications this has for performance, consistency, and operational complexity. While many resources detail how to set up Trino, Spark, or DuckDB independently, understanding the nuances of their Iceberg integration is crucial for building robust, multi-engine data platforms. This article breaks down the distinct approaches each of these popular engines takes to query a shared Iceberg table, highlighting their respective cataloging methods and metadata access patterns.

Shared Iceberg Model: Metadata is Key

At its heart, Apache Iceberg abstracts away the complexities of managing large datasets in object storage. Unlike traditional Hive-style tables that rely on directory structures and manifest files within the file system, Iceberg employs a layered metadata approach. A snapshot of the table's state is represented by a metadata file (typically JSON), which points to manifest lists. These manifest lists, in turn, reference manifest files that contain individual data file paths, statistics, and partition information. This hierarchical structure allows for efficient metadata operations, such as finding data files for a query, and enables ACID guarantees through atomic commits of new metadata files.

The physical location of these metadata files is managed by a catalog. The catalog's role is to provide a discoverable endpoint for table metadata. Different engines leverage catalogs in distinct ways. Trino, for instance, typically relies on a catalog service (like AWS Glue Data Catalog, Hive Metastore, or a custom REST catalog) to locate the Iceberg metadata file for a given table. Spark integrates Iceberg through its own catalog mechanisms and the Spark Iceberg extensions, which provide optimized readers and writers. DuckDB offers a more direct approach, capable of reading a single Iceberg table directly from its metadata files on object storage, or by attaching a catalog for more advanced features.

Trino: Catalog-Driven Metadata Discovery

Trino, a distributed SQL query engine designed for interactive analytics, accesses Iceberg tables through a configured catalog. When you define an Iceberg catalog in Trino (e.g., using the `iceberg-catalog` connector), you specify how Trino should find the table's metadata. The most common methods involve pointing Trino to a metastore service like AWS Glue or the Hive Metastore, which then holds the location of the Iceberg table's current metadata file. Alternatively, Trino can be configured to use an Iceberg REST catalog, which directly exposes table metadata over HTTP.

When a query is issued against an Iceberg table in Trino, the engine first consults its catalog configuration to find the table's metadata location. It then reads the Iceberg metadata file, followed by the manifest lists and manifest files, to identify all the data files that constitute the table at that snapshot. Trino's query planner uses the statistics within these files to optimize query execution. This catalog-centric approach ensures that Trino always reads the most up-to-date table state as registered in the catalog service.