You Might Be Using The Wrong Compression Algorithm
If you work in data engineering, you’ve probably used gzip, Snappy, LZ4, or Zstandard (zstd). More likely — you inherited them. Either the person who set these defaults is long gone, there’s never enough time to revisit the choice, or things work well enough and you’d rather not go digging.
Most engineers stick with the defaults. Changing them feels risky. And let’s be honest — many don’t really know what these algorithms do or why one was chosen in the first place.
I’ve been that person myself: “Oh, we’re using Snappy? OK.” Never thinking to ask why, or what else we could use.
This post explains the most common compression algorithms, what makes them different, and when you should actually use each.
Reproducible Benchmark
The recommendations below are now backed by an open, reproducible benchmark. View the GitHub repository or open the live July 2026 report. The benchmark downloads public datasets on demand and publishes exact settings, raw samples, summary CSVs, hardware details, and matrix charts.
Why Compression Choices Matter
Compression decisions aren’t just about saving space. They directly impact:
- Storage costs
- CPU utilization
- Throughput
- Latency
In modern pipelines — Kafka, Parquet, column stores, data lakes — the wrong compression algorithm can degrade all of these.
Two metrics matter most:
- Compression ratio: How much smaller the data gets.
- Throughput: How quickly data can be compressed and decompressed.
Your workload — and whether you prioritize CPU, latency, or bandwidth — determines which trade-offs are acceptable.
Main Culprits
gzip
- What it is: Uses the DEFLATE algorithm (LZ77 + Huffman coding).
- Goal: Good compression ratio and broad compatibility.
- Speed: Slow to compress; moderate decompression speed.
- Strength: Ubiquitous. Supported everywhere.
- Weakness: Outclassed in both speed and compression ratio by newer algorithms.
- When to use: Broad compatibility, established web tooling, or legacy systems where changing formats has limited value.
Snappy
- What it is: Developed by Google. Based on LZ77 without entropy coding.
- Goal: Maximize speed, not compression ratio.
- Speed: Very fast compression and decompression.
- Strength: Low CPU overhead. Stable. Production-proven at Google scale.
- Weakness: Larger compressed size than other options.
- When to use: Real-time, low-CPU systems where latency matters more than storage — or if you’re stuck with it.
LZ4
- What it is: LZ77-based. Prioritizes speed.
- Goal: Fast compression and decompression with a moderate compression ratio.
- Speed: Workload-dependent. In this run, LZ4 level 0 compressed enwik9 at 344 MB/s and decompressed it at 634 MB/s.
- Strength: Extremely fast. Low CPU usage.
- Weakness: Compression ratio lower than gzip or zstd.
- When to use: High-throughput, low-latency systems. Datacenter transfers. OLAP engines (DuckDB, Cassandra).
zstd (Zstandard)
- What it is: Developed by Facebook. Combines LZ77, Huffman coding, and FSE.
- Goal: High compression ratio with fast speed.
- Speed: Workload-dependent. In this run, zstd level 3 compressed enwik9 at 114 MB/s and GitHub Archive JSON at 389 MB/s.
- Strength: Tunable. Balances speed and compression. Strong performance across data types.
- Weakness: Slightly more CPU than LZ4/Snappy at default settings.
- When to use: General-purpose. Parquet files. Kafka. Data transfers. Usually the best all-around choice.
Brotli
What it is: A modern codec developed by Google and widely used for HTTP content. Measured result: 3.60× ratio, 34 MB/s compression, and 219 MB/s decompression on enwik9 at level 5. When to use: Web assets and static content when better density than gzip is worth additional compression CPU.XZ/LZMA
What it is: A high-density codec commonly used for archives and software distribution. Measured result: The best enwik9 ratio at 4.34×, but only 1.3 MB/s compression and 61 MB/s decompression at level 6. When to use: One-time archival or static distribution, not latency-sensitive pipelines.Strengths and Weaknesses (At a Glance)
| Algorithm | Compression Ratio | Compression Speed | Decompression Speed | Best For |
|---|---|---|---|---|
| gzip | Moderate | Slow | Moderate | Archival, web content |
| Snappy | Low | Very Fast | Very Fast | Real-time, low-CPU systems |
| LZ4 | Moderate | Extremely Fast | Extremely Fast | High-throughput, low-latency systems |
| zstd | High | Fast | Fast | General-purpose, Parquet, Kafka, data transfers |
| Brotli | High | Slow | Fast | HTTP and static web content |
| XZ/LZMA | Very High | Extremely Slow | Slow | One-time archival and distribution |
Live Benchmark Results
The July 2026 standard run processed 2.43 GiB of public data with six codecs, five measured repetitions per case, and zero failures. These enwik9 results show the trade-off on a 1 GB Wikipedia text input:
| Codec | Ratio | Compress MB/s | Decompress MB/s | RSS MiB |
|---|---|---|---|---|
| gzip 6 | 3.09× | 18 | 178 | 30.5 |
| Snappy 1 | 1.97× | 313 | 358 | 35.4 |
| LZ4 0 | 1.96× | 344 | 634 | 33.0 |
| zstd 3 | 3.19× | 114 | 142 | 41.9 |
| Brotli 5 | 3.60× | 34 | 219 | 60.8 |
| XZ 6 | 4.34× | 1.3 | 61 | 110.9 |
On 977.8 MiB of GitHub Archive JSON, zstd reached an 11.3× ratio at 389 MB/s, while LZ4 decompressed at 617 MB/s and Snappy compressed at 570 MB/s. The data type changes the result, so use the matrices rather than treating one codec as universally best. Explore the live result matrices.
Real-World Scenarios: When to Use What
High-throughput streaming (Kafka)
- Use: zstd or LZ4
- Why: zstd gives better compression with good speed. LZ4 if latency is critical and CPU is limited. Snappy is acceptable if inherited, but usually not optimal anymore.
Long-term storage (Parquet, S3)
- Use: zstd
- Why: zstd delivers a strong compression ratio with practical throughput. XZ can compress smaller, but its much slower encoding is usually unsuitable for recurring data-pipeline workloads.
Low-latency querying (DuckDB, Cassandra)
- Use: LZ4
- Why: Prioritize decompression speed for fast queries. LZ4 is the common choice in OLAP engines.
CPU/memory constrained environments
- Use: Snappy or LZ4
- Why: Low CPU overhead matters more than compression ratio. zstd can still be used at low compression levels if needed.
Fast network, low compression benefit (datacenter file transfer)
- Use: LZ4
- Why: Minimal compression overhead. On fast networks, speed beats smaller file sizes.
Slow network or internet transfers
- Use: zstd
- Why: Better compression reduces transfer time despite slightly higher CPU cost.
What to Remember
- No algorithm is best for every workload.
- zstd has become the Swiss Army knife of compression. Unless you have a good reason not to, it’s a smart pick.
- LZ4 is often the leading choice when speed matters more than compression density.
- Snappy is still acceptable in latency-sensitive, CPU-constrained setups but is generally being replaced.
- gzip remains for legacy systems or when maximum compatibility is required.
What’s Underneath The Hood
- LZ77 — Replaces repeated sequences of data with references to earlier copies in the stream (a sliding window).
- Huffman Coding — Assigns shorter codes to more frequent data patterns to save space.
- FSE (Finite State Entropy) — An advanced entropy coding method that efficiently compresses sequences by balancing speed and compression ratio.
Why it matters: Most compression algorithms combine finding patterns (LZ77) with efficient encoding (Huffman, FSE) to shrink data without losing information.
Closing Thoughts
Compression choices tend to stick around. There’s rarely time to revisit legacy pipelines, and if something works, it’s easy to assume it’s good enough. But if you can make the time, you’re now better equipped to review your defaults — and see if a different choice might better fit your needs.