Composable Data Management at Meta
How Meta dismantled a decade of siloed data systems by extracting Velox as a shared execution engine, aligning with Apache Arrow, and opening the path to hardware-accelerated analytics.
Specialization is supposed to be the mark of a mature engineering organization. Meta spent ten years specializing every data system it had — and ended up with engineers duplicating the same query optimizer logic in six different codebases.
Source
Composable data management at Meta
By: Meta
Published: May 22 2024
Type: Engineering article
Worth saving for the organizational argument it makes as much as the technical one. The composable systems framing is a useful lens for any team that has built two or more data pipeline components independently and now wonders why they feel so expensive to maintain.
When Each System Reinvents the Wheel
Meta’s data infrastructure at the start of this journey was a collection of well-engineered silos. Each system — offline ETL, interactive dashboarding, ad hoc exploration, stream processing, ML feature preprocessing — had been built by a specialist team for a specific workload, and each performed its core task well. The compound cost only appeared over time. If you found a more efficient join algorithm, you wrote it in each system separately. If you wanted a new string function, every system required its own implementation. If you hired a database execution expert, their improvements landed in one codebase rather than all of them. Scale doesn’t just amplify demand; it amplifies the cost of duplication.
The composable systems insight starts from an observation that feels obvious in retrospect: all data systems are built from the same five logical layers. A language frontend parses user input — SQL, a dataframe API, or something else — into an internal form. An intermediate representation captures that computation as a logical or physical query plan. A query optimizer transforms the plan into a more efficient one. An execution engine locally evaluates plan fragments. An execution runtime provides the distributed environment in which those fragments run. The differences between an OLAP warehouse, a stream processor, and an ML preprocessing pipeline show up at the edges — dialect semantics, streaming-specific operators, tensor manipulation. At the core, the algorithms and data structures are largely the same across all of them. This reframing carries a direct engineering prescription: build to what the systems share, because the intersection is larger than the union of what makes each one unique.
Velox is Meta’s bet that the execution layer is the right abstraction boundary for shared investment. Created in late 2020 and open-sourced in 2022, Velox is designed as an engine-agnostic and dialect-agnostic execution library that any data system can embed. By the time this article was published, it was active in more than ten Meta systems. The most public result is Prestissimo — Velox as Presto’s execution backend — which delivered 3–10x efficiency improvements in production workloads. Apache Gluten, which positions Velox as the execution engine inside Spark, hit a similar 3x gain on benchmarks. Newer internal systems — a time-series database, a low-latency interactive engine — were built in record time by teams that could borrow a vetted execution core rather than write one from scratch. This is the organizational payoff: a focused team of execution specialists produces a single implementation that all other teams benefit from.
Velox’s performance characteristics rest on four techniques working together. Columnar and vectorized execution organizes data into tight loops with predictable memory access patterns that CPUs can prefetch efficiently and SIMD-vectorize. Compressed execution treats dictionary encoding as both a compression mechanism and a processing shortcut — a filter over a dictionary-encoded column can operate on the compact dictionary rather than materializing decoded values for every row. Lazy materialization delays decoding to the last possible moment, sometimes avoiding it entirely when an operation can be expressed as a transformation on the encoding itself. Adaptivity watches execution across successive batches: it reorders filter conjuncts based on observed hit rates, restructures join probe logic based on join-key cardinality, and adjusts column prefetch behavior based on access history. These techniques compound — the columnar layout enables SIMD; lazy evaluation reduces the volume of bytes the SIMD loops touch; adaptivity ensures the engine keeps improving as it learns the actual data distribution of a running workload.
The Arrow alignment work resolved the interoperability boundary that Velox’s intentional format divergence had created. Velox had introduced its own columnar layout (Velox Vectors) to unlock processing optimizations that Apache Arrow’s original format couldn’t express — gaining performance at the cost of a format boundary that every cross-component integration had to bridge. After a year of collaboration with Voltron Data and the Arrow community, three Arrow extensions landed in new releases: StringView for variable-length string processing, ListView for list column access, and Run-End-Encoding for cardinality-reducing operations like filters and joins. The result is zero-copy data transfer between Velox and Arrow-native systems, which matters anywhere an API boundary currently forces a serialization round-trip — and in a composable stack, those boundaries appear at every component seam.
One clarification worth making explicit: composable doesn’t mean generic. Velox exposes extensibility APIs because domain-specific behavior is real and shouldn’t be flattened away. A stream processor needs streaming-specific operators; an ML system needs tensor manipulation. The composable model says: share the execution loop, the encoding handling, the optimizer scaffolding — but expose clean extension points for what each domain does differently. That distinction separates a viable composable system from a bloated general-purpose framework that satisfies nobody. The extensibility surface is where specialization lives; the common core is where investment concentrates.
Nimble (formerly Alpha, now open-sourced) applies the same logic to the file format layer. Parquet and ORC were designed for moderately-wide analytical tables. AI training tables can span thousands of feature streams, benefit from recursive encoding schemes that traditional formats can’t represent, and require parallel decoding throughput fast enough to saturate GPU trainers ingesting data faster than a single decoder can produce it. Nimble is being positioned as a portable library that other systems can adopt — the same reusable-component argument applied one layer lower in the stack.
The paper includes an observation that the article doesn’t linger on but that deserves its own sentence: reusability challenges are “not only technical but commonly also cultural and even economic.” Most engineering organizations measure team success by product output, not infrastructure investment. Building shared components requires staffing a horizontal team whose job is to make other teams faster — and keeping that team intact across planning cycles is harder than the technical decomposition. The Velox effort only happened because the accumulated cost of duplication became impossible to ignore and because leadership was willing to resource a team focused on the intersection rather than any individual system’s roadmap.
My take
The most underappreciated line in this article is the organizational one. Any team can look at a composable architecture and agree it’s elegant. The hard part is funding a team that owns shared infrastructure without a direct product attribution chain. Meta could only do this after the duplication cost became visible at budget scale. Smaller organizations can shortcut that wait by treating the execution library question as a build-vs-extend decision at the start of a new system, not a refactor decision after five years of divergence.
Where this is useful
- Deciding where to draw library boundaries when building a new data processing stack
- Evaluating whether to fork an open-source query engine or contribute domain-specific behavior upstream instead
- Understanding why DuckDB, Polars, and Velox are all converging on the same columnar execution techniques despite serving different use cases
Apply this: Before writing a new data pipeline component, check whether an existing execution-layer library can be extended with domain-specific logic — the difference between embedding Velox and writing an execution engine is measured in years of engineering time.
What I want to remember
Remember: The intersection of what all data systems share — query planning, vectorized execution, columnar encoding — is larger than the union of what makes each one unique. Build to the intersection first, and put specialization in the extension layer.