File Compression: The Unseen Driver of Storage, Latency, and Compute Costs

The Invisible Engine of Data Infrastructure

Somewhere in your data platform right now, a single configuration property is quietly deciding a meaningful percentage of your storage bill, your query latency, and your compute spend. It is probably set to whatever the defaults were in 2019, nobody has looked at it since, and it is the compression codec.

Compression is the most consequential invisible decision in data infrastructure. Every Parquet file in your lakehouse, every message crossing your network, every backup in your archive passed through a compressor, and the choice of which one, at which setting, ripples through everything downstream: bytes stored, bytes transferred, requests billed, CPU burned on every read for the life of the data. Yet most engineers' working knowledge of the topic amounts to a vague ranking: gzip is old, Snappy is fast, Zstandard is good, without the mechanics that would let them reason about a new situation.

This article fixes that. We will build the theory from the ground up, explaining how data gets smaller, why codecs differ, and what to actually use in the lakehouse.

How Data Gets Smaller: The Core Principles

At its heart, compression works by identifying and eliminating redundancy in data. Think of it like writing a shorthand for frequently repeated phrases. Instead of writing "The quick brown fox jumps over the lazy dog" every time, you might assign it a symbol like "#1". If that phrase appears 100 times, you've saved a lot of ink.

Data compression algorithms exploit various forms of redundancy:

• Repetitive Sequences: Finding identical sequences of bytes and replacing them with a shorter reference. Run-Length Encoding (RLE) is a simple example, where "AAAAA" becomes "5A".
• Statistical Redundancy: Identifying characters or patterns that occur more frequently than others and assigning them shorter codes. Huffman coding and Arithmetic coding are classic examples. If 'e' appears 100 times and 'z' appears once, 'e' gets a shorter bit code.
• Structural Redundancy: Exploiting predictable patterns or structures within the data. For structured data like CSV or Parquet, this might involve recognizing that a column contains many of the same values, or that values follow a pattern (e.g., dates, sequential IDs).

Lossless compression, which is crucial for data storage and transfer, guarantees that the original data can be perfectly reconstructed from the compressed version. This is achieved by carefully encoding the redundancy information and then decoding it upon retrieval. Lossy compression, common in media like images and audio, discards some information deemed less perceptible to humans to achieve higher compression ratios, but it's generally unsuitable for raw data storage.

Why Codecs Differ: A Spectrum of Trade-offs

The specific algorithms and techniques used to identify and encode redundancy define a compression codec. Different codecs make different trade-offs, primarily between compression ratio (how small the data gets) and speed (how fast it can compress and decompress).

Common Codecs and Their Characteristics

Understanding these trade-offs is key to choosing the right tool for the job:

• Gzip (DEFLATE): One of the oldest and most widely supported. It offers good compression ratios but is relatively slow, especially for decompression. It's a good general-purpose choice when speed isn't paramount.
• Snappy: Developed by Google, Snappy prioritizes speed over compression ratio. It decompresses very quickly, making it ideal for scenarios where read performance is critical and storage cost is less of a concern. Its compression ratio is significantly lower than Gzip or Zstandard.
• LZO: Similar to Snappy in its focus on speed. It provides decent compression and very fast decompression, often used in big data systems for real-time processing.
• Brotli: Developed by Google, Brotli often achieves better compression ratios than Gzip, especially for text-based data, and offers competitive decompression speeds. It's more common in web content delivery but can be applied to data files.
• Zstandard (Zstd): Developed by Facebook, Zstandard is a modern codec that strikes an excellent balance between compression ratio and speed. It offers compression ratios comparable to Gzip but with significantly faster compression and decompression speeds. It also has a wide range of compression levels, allowing fine-tuning.
• LZ4: Another extremely fast codec, often even faster than Snappy, at the cost of a lower compression ratio. It's best suited for applications where latency is measured in microseconds and data volume is massive.

The Lakehouse Context: Choosing What to Use

In a data lakehouse, data is typically stored in columnar formats like Parquet or ORC. These formats are designed with compression in mind, often compressing data column by column, which is highly effective because data within a single column tends to be similar (e.g., all timestamps, all user IDs, all product names). This makes the choice of codec particularly impactful.

Factors to Consider for Lakehouse Data

When deciding which codec to use for your lakehouse data, consider these factors:

• Read vs. Write Patterns: Are you writing data once and reading it many times? If so, a codec with a better compression ratio (like Zstandard or Gzip) will save storage and transfer costs over the long term, even if writing is slightly slower. If you have frequent, interactive reads where latency is paramount, a faster codec like Snappy or LZ4 might be preferred.
• Data Characteristics: Highly repetitive data (e.g., categorical data with few unique values) compresses very well with most codecs. Textual or semi-structured data might benefit more from codecs with advanced dictionary coding. Data that is already somewhat random or encrypted will compress poorly regardless of the codec.
• Compute Budget: Compression and decompression both consume CPU. If your compute resources are heavily constrained, you might opt for a faster decompression codec to reduce the CPU load on query engines, even if it means slightly larger files. Conversely, if storage is the primary cost driver and compute is plentiful, prioritize compression ratio.
• Ecosystem Support: Ensure the codec is well-supported by your data processing frameworks (e.g., Spark, Trino, Dremio). Zstandard, Snappy, and Gzip are generally well-supported.

Recommendations for Lakehouse Use Cases

For most general-purpose lakehouse workloads, Zstandard (Zstd) is the current champion. It offers compression ratios often close to Gzip but with significantly better performance on both compression and decompression. Its configurable levels allow you to tune the balance. For instance, Zstd level 3 is a common sweet spot, providing good compression without excessive CPU overhead.

If extreme read latency is the absolute top priority and storage/compute costs are secondary, Snappy or LZ4 remain viable options, especially for frequently accessed, hot data. However, the cost savings from Zstandard's better compression often outweigh the marginal speed differences for typical analytical workloads.

Gzip is still a safe fallback if compatibility with older systems or a very broad range of tools is essential, but its performance limitations make it less ideal for new deployments where Zstandard is supported.

The Unanswered Question: Dynamic Compression in Streaming

While static choices work for batch data, what about streaming data? As data flows in, its characteristics can change. What if a new type of data enters the stream that compresses exceptionally well or poorly with the currently configured codec? The current approach often involves setting a codec and level and forgetting it. But what if we could dynamically adjust compression based on the real-time characteristics of the incoming data to maintain optimal storage, latency, and compute costs? This would require sophisticated monitoring and adaptive algorithms, a frontier that remains largely unexplored in practical lakehouse implementations.