Meta's AI Storage Blueprint at Scale
How Meta rebuilt its exabyte-scale BLOB storage layer from first principles — scrapping a multi-layer metadata chain and adding a tiered caching architecture — to eliminate GPU stalls and compress multi-hour ingestion cycles.
Everyone optimizes the GPU. The storage layer is the overlooked variable — until you realize that accelerators costing tens of thousands of dollars are sitting idle waiting on metadata lookups that used to take hundreds of milliseconds and nobody noticed.
Source
Meta’s AI Storage Blueprint at Scale
By: Meta
Published: Jul 01 2026
Type: Engineering article
A clean example of redesigning a system by auditing its founding assumptions rather than patching its symptoms. The architecture that results is interesting — but the methodology of naming the broken assumptions explicitly before writing a line of new code is the more transferable lesson.
When a Hundred-Millisecond Lookup Stalls a Hundred Thousand GPUs
Meta operates hundreds of exabyte-scale storage clusters serving Facebook, Instagram, Reality Labs, internal databases, and AI training infrastructure. The foundational layer is Tectonic — a regional, multi-tenant block storage fabric that handles erasure coding, tiering across HDD and flash, and placement of hot, cold, and warm data across tenants. On top of Tectonic sits the BLOB-storage layer, which exposes object storage, file system, and block-device APIs to every application that touches data at Meta. This layered architecture served Meta well across a decade of web and social workloads. The problem is that it was designed around assumptions that AI training workloads invalidate at every layer simultaneously.
AI training access patterns differ from conventional storage access in three ways that compound. Throughput is not merely high but sustained and bursty — hundreds of thousands of GPUs iterating over the same datasets multiple times across epochs, with concurrent access patterns during checkpoint events that can spike sharply and then vanish. Tail latency (p99, pMax) matters more than median latency, because training clusters synchronize across GPUs periodically. If one GPU is slow at loading its batch, every other GPU in the cluster waits at the synchronization barrier — a stall that’s invisible in a web-serving report is a full training halt in a synchronized AI cluster. And the “hot objects” problem is structural, not incidental: model weights and frequently reused dataset shards are accessed by hundreds of GPUs concurrently during restarts, which creates recurrent traffic spikes over the same small set of objects.
The legacy BLOB-storage architecture accumulated metadata layers organically over time: a namelayer, a volumeslayer, a containerlayer, each stateful, each maintaining its own metadata stores. A single getObject("/bucket/path") call triggered sequential lookups across all of these layers, some of which crossed region boundaries before resolving the path to a set of (blockId, offset, size) tuples. It wasn’t unusual for total metadata latency to reach hundreds of milliseconds per call. For HDD-backed workloads where read latency was already in the tens of milliseconds, adding metadata overhead of that magnitude was an acceptable tradeoff that rarely showed up as a bottleneck. For flash-backed AI workloads aiming at single-digit millisecond data access, it was a structural blocker — and every stall in that chain propagated directly to GPU idle time.
Rebuilding the foundation started by explicitly naming the four assumptions that no longer held. Performance expectations had changed: modest latency was previously acceptable; AI workloads require bounded and predictable latency all the way to pMax, not just at the median. Reliability expectations had changed: the legacy system replicated data globally by default, but AI training data doesn’t need the same global durability guarantees as user-generated content — regional availability is sufficient and far cheaper. Cost structure had changed: flash is required for the IOPS profile of AI workloads, and the per-byte cost premium over HDDs is negligible compared to the cost of idle GPUs. Power constraints had changed: datacenters running AI clusters are power-constrained, not space-constrained, which means every kilowatt spent on a storage proxy is a kilowatt not available to run GPU compute.
The new architecture makes three targeted changes. The metadata subsystem was rewritten around a unified flat schema backed by ZippyDB, collapsing the multi-layer lookup chain into a single O(1) lookup per chunk that maps a path directly to (blockId, offset, size) without any cross-region hops. The dataplane proxy was eliminated: instead of an API server fetching data from Tectonic and forwarding it to the client — consuming a full network hop and a power-consuming server tier — a fat client SDK with an embedded Tectonic block client streams bytes directly from storage nodes to the caller. The power saving here isn’t incidental; in a power-constrained datacenter, removing the proxy tier is as much a capacity decision as a latency decision. The BLOB-storage stack is now lean enough to deploy regionally, colocated with GPU clusters, rather than running as a centralized global service.
With the foundation fixed, the next problem was the concurrency spikes. AI checkpointing events drive sharp bursts where hundreds of GPUs simultaneously read or write the same hot objects. The solution adapted an existing Meta subsystem — Owl — to a new role. Spare memory on GPU hosts is pressed into service as a distributed data cache for frequently accessed objects, and the cache achieves an 80% hit rate in practice, absorbing most checkpoint spikes before they reach the storage layer. A separate readplan cache stores the path-to-storage-address mappings in a distributed memory store, providing 1–2ms metadata access in place of what was previously a chain of RPC calls. Together, these two mechanisms do three things: they absorb traffic spikes, they eliminate hot metadata shard pressure, and they move p50 and p99 latencies from storage node response time into memory access time.
Protocol-level work closed the remaining gap. Hedged reads address the laggard problem: when one storage node is slow, the client issues a duplicate request to a replica and takes whichever responds first, preventing a single slow node from inflating tail latencies for an entire training step. Dynamic concurrency control addresses checkpoint egress spikes: the client SDK monitors application-level congestion signals and automatically backs off parallelism when it detects congestion building, preventing the spike-to-congestion-to-timeout-to-retry cascade that would otherwise stall the training job at exactly the moment the system is under most load.
The research velocity problem required a different kind of architecture entirely. Researchers were spending hours — sometimes the majority of their iteration cycle — pre-copying datasets from global BLOB storage into the regional deployment colocated with their GPU jobs before they could even submit a training run. The core insight that broke the problem open was an analogy to operating systems: training datasets are write-once, read-many, exactly like files on a disk. An OS doesn’t require an application to pre-copy files from disk into the CPU L2 cache before running; it transparently hydrates data on demand through a tiered cache hierarchy. The BLOB-storage team applied the same model. GPU host memory becomes the L1 cache, GPU host flash becomes L2, and a regional disaggregated flash tier becomes L3. A standard prefetch covers the next immediate batch. A deep prefetch API, invoked in the background by the dataloader, warms the L3 cache for the next few minutes of training. TTL and LRU eviction policies manage data lifecycle automatically. Researchers submit training jobs without pre-ingestion and begin iterating in minutes rather than hours. Both approaches — on-demand hydration for the majority of smaller research jobs, and pre-ingestion for large multi-week training runs that need deterministic maximum performance — coexist in production because the tradeoff is explicit rather than hidden.
My take
The redesign is interesting less for any individual technique than for the sequence: audit the broken assumptions first, name them explicitly, then build a new foundation against the corrected assumptions rather than patching the old one. The team could have added caching layers on top of the legacy architecture and recovered some of the latency headroom. Instead they identified that the proxy, the multi-layer metadata chain, and the global-by-default replication policy were each individually wrong for the new workload, and rebuilt from those four violations outward. That approach only works if you’re willing to call the original system correct for its original context and wrong for the present one — which is harder than it sounds when the original system is a decade old and battle-tested.
Where this is useful
- Designing ML training data pipelines where GPU utilization is the primary cost driver
- Understanding when distributed in-memory caching is more effective than regional data replication for hot objects
- Evaluating whether a storage proxy tier is justified for a given workload’s power and latency budget
Apply this: Before adding caching layers to a storage system that wasn’t built for its current workload, audit the four founding assumptions — performance expectations, reliability requirements, cost structure, and power constraints — because optimizing on top of the wrong assumptions yields diminishing returns.
What I want to remember
Remember: Eliminating the dataplane proxy wasn’t just a latency optimization — it was a power optimization. In a GPU-dense datacenter, every kilowatt spent on storage infrastructure is a kilowatt not running compute.