Introducing OpenZL: An Open Source Format-Aware Compression Framework
Meta's OpenZL achieves format-specific compression ratios and speeds without the maintenance cost of bespoke compressors, by making data structure an explicit input and decoding everything with one universal binary.
Every general-purpose compressor is secretly making a bet that it can figure out the structure of your data at runtime. OpenZL starts from the opposite assumption: tell it what your data looks like, and let the compression plan write itself.
Source
Introducing OpenZL: An Open Source Format-Aware Compression Framework
By: Meta Published: Oct 06 2025 Type: Engineering article
Saved because it names the actual failure mode of general-purpose compressors — they treat structured data as byte soup — and then shows a coherent architecture for fixing it without fragmenting the decompression ecosystem.
Format-aware compression and the universal decoder
The case for Zstandard, when it launched, was simple: compression ratios on par with older tools but at datacenter-viable speeds, achieved by pairing strong entropy coding with a design that made full use of modern CPUs. That was a genuine leap, and the pitch held up. But it assumed the compressor has to discover structure on its own, which burns cycles and leaves ratio gains unrealised. For a while that tradeoff was fine. Then the datasets Meta was compressing started making the limitation obvious.
The pattern that kept appearing was this: data isn’t just bytes in sequence. Columnar tables have fields with narrow numeric ranges. Star catalogues have sorted positional indexes. Log records repeat field names thousands of times per file. A bespoke compressor that leans into those shapes can beat Zstandard on both ratio and throughput — sometimes by a factor of two. The catch is that every bespoke scheme adds a new compressor and decompressor to build, ship, audit, patch, and trust. At Meta’s scale that compounds into a fragmentation problem.
OpenZL’s answer is to split what compression actually does into two separable concerns. The first is understanding structure, which is the hard, format-specific part. The second is compressing the resulting streams, which is general. OpenZL handles the first concern at an offline training stage, not at runtime, and then serialises the resulting compression plan into each frame so the same universal decoder can execute it without any out-of-band coordination.
In concrete terms: you describe your data to OpenZL using its Simple Data Description Language — a schema-like notation that maps bytes to fields, rows, columns, and enums. The offline trainer runs a budgeted search over transform choices and parameters, finds a Pareto-optimal plan, and records it. When encoding, that plan resolves into a concrete recipe embedded in the frame. The universal decoder reads the recipe and runs the steps in order. If the plan is later improved and re-trained, old frames still decode with the same binary; new frames benefit from the better plan. No new decompressor ships.
The gains are not marginal. On the SAO star-catalogue dataset from the Silesia Compression Corpus, OpenZL reaches a 2.06× compression ratio at 340 MB/s compression and 1200 MB/s decompression on an M1 CPU. Zstd at level 3 reaches 1.31× at 220 MB/s compression and 850 MB/s decompression. That improvement comes entirely from structure awareness: OpenZL destructures the array of star records into a structure of arrays, then applies different transforms per field. The sorted X-axis position gets delta coding, which collapses its numeric range. The Y-axis position gets a transpose that exploits its value bounds. The low-cardinality enum fields get tokenised, turning repetition into a dictionary plus a compact index list. Each resulting stream goes to a dedicated sub-graph tuned to its actual content.
The runtime side adds one more lever: control points. A plan can include lightweight statistics evaluated at compression time — string repetition, histogram skew, run-length — that choose between branches within the plan. This handles data that drifts over time without requiring full re-training. The chosen branch is recorded into the frame so the decoder just executes a recorded path. Exploration is bounded to keep speed predictable.
The limits are worth naming plainly. OpenZL requires structure to exploit. Pure text — enwik, Dickens — has no columns, no sorted fields, no bounded ranges, and OpenZL falls back to Zstandard, gaining nothing. CSV is a harder case: OpenZL can parse it and learns a good plan, but the cost of CSV parsing caps compression throughput at around 64 MB/s, far below Zstandard’s 1 GB/s. Formats like Parquet and binary columnar data are where the framework shines. The mental model is: if you can describe the schema, OpenZL can probably beat a generic compressor; if the data is genuinely unstructured text, it cannot.
A common misread of this kind of system is to think of it as a compression level dial with extra steps. It’s closer to a compiled query plan: the output of training is a serialised graph of operations that the decoder executes directly, and the space of possible plans is vastly larger than the small integer range that compression levels give you. That flexibility is what lets OpenZL trade ratio for speed along a Pareto frontier rather than a single line.
My take
The most transferable idea here is not the compression technique itself but the architecture of separating structure-discovery (offline, per-format, expensive) from execution (online, universal, audited once). That separation is what makes format-specific performance viable at a large organisation’s operational scale. The same pattern applies anywhere the cost of specialised logic is offset by the operational simplicity of a single execution path.
Where this is useful
- Compressing structured datasets at scale: columnar tables, ML tensors, time-series, Parquet files
- Pipelines where schema is known and stable enough to train against
- Any system managing many distinct data formats that can’t afford a separate decompressor per format
- Not useful for unstructured text corpora or any format where no schema description is available
Apply this: When evaluating compression for a structured dataset, measure against a format-aware baseline, not just generic Zstandard levels — the ratio delta is often large enough to justify the training investment.
What I want to remember
Remember: Making structure an explicit input to the compressor — not something to be discovered at runtime — is the difference between marginal tuning and a step-change in ratio, and the universal decoder is what makes that tradeoff operationally viable.