Meta's Infrastructure Evolution and the Advent of AI
A 21-year arc from LAMP-stack Facebook to gigawatt AI clusters, tracing the specific moments when AI workloads broke every assumption that had scaled web infrastructure for two decades.
The standard account of AI infrastructure is “we needed more compute, so we added more GPUs.” Meta’s account of the last four years is a story of discontinuities — points where the existing model didn’t just need to be larger, it needed to be structurally replaced.
Source
Meta’s Infrastructure Evolution and the Advent of AI
By: Meta
Published: Sep 29 2025
Type: Engineering article
Meta’s most complete public account of what it actually took to scale from 4k-GPU clusters to gigawatt-scale training infrastructure. The historical framing is what makes it valuable — you understand why each architectural decision was made rather than just what was chosen.
From Retries to Checkpoints: When One GPU Stalls Everything
Twenty-one years of scaling at Meta is, at its core, a story about fault models. The web infrastructure era was built on a single organizing assumption: any individual request could fail, be retried on a different machine, and complete without the user noticing. That assumption justified buying cheap, unreliable commodity hardware by the thousands and then building software to mask the failures. Memcache absorbed database load. TAO managed the social graph. Twine scaled cluster management to millions of machines. Maelstrom handled datacenter-scale disasters while minimizing user impact. The consistency and reliability problems that emerged as Facebook grew globally — cache invalidation lag, out-of-order message delivery across regions — were solved by more software layered on top of the same retry-friendly foundation.
The fault model breaks the moment you need a hundred thousand GPUs to execute one job synchronously. In web serving, a stalled machine sheds some requests and the others retry elsewhere. In LLM pretraining, every GPU in the cluster is executing the same step of the same model update at the same moment. If any single GPU stalls — from a memory error, network jitter, or a slow storage read — every other GPU in the cluster waits. A failure that’s invisible in a web-serving tail-latency report is a complete training halt when it appears in a synchronized cluster. Jobs that run for weeks accumulate enough hardware-induced interruptions that, without active mitigation, training progress approaches zero. Meta drove the interruption rate down by roughly 50x through work spanning hardware, firmware, software, and collaboration with NVIDIA and AMD — and that number, not raw FLOP capacity, is what determined whether large training runs produced usable models.
GPUs entered Meta’s infrastructure not because of LLMs but because of short-form video recommendations. The earlier Facebook ranking problem was relatively bounded: surface content that a person’s few hundred friends had interacted with. Short-form video changed the ranking domain entirely. To recommend from a corpus of millions of videos to every individual user, the system needs to embed the full content library and then score it against per-person representations — that’s orders of magnitude more computation than social-graph-based ranking, and it’s exactly the matrix multiplication that GPUs are built for. This origin matters because it means Meta’s GPU infrastructure was designed and validated on recommendation workloads first, not on LLMs, and recommendation inference remains the primary revenue workload today. LLMs arrived as an accelerant applied to an infrastructure that already existed.
When LLM training started in 2022, the transition in job size happened in weeks. Jobs went from 128 GPUs to 2k to 4k almost immediately, because the empirical relationship between compute investment and model quality was direct: more pretraining FLOPs produced better models, with no obvious saturation in sight. At 4k GPUs, the jobs mostly worked with tuning. At 24k GPUs, Meta ran two parallel clusters simultaneously — one on InfiniBand, one on RoCE — partly to evaluate network technology choices under production load, and partly because both were needed to train Llama 3. Scaling to 129k H100s required emptying five production data center buildings and repurposing them for a single training cluster in a matter of months. This is not a capacity expansion in the conventional sense — it’s a reconfiguration of physical infrastructure faster than most organizations plan a single major project.
The hardware heterogeneity that followed is one of the less discussed operational complications. Meta is now deploying five or six different GPU and accelerator SKUs per year: NVIDIA H100s, GB200s, AMD MI300s, and its own MTIA chip for ranking and recommendation inference. Each new hardware generation potentially requires rewriting kernels, libraries, and scheduling logic. A GB200 rack consumes around 140kW of power — so much that Meta had to deploy four air-assisted liquid cooling units per rack to prevent heat from damaging the hardware. Managing a fleet with this level of diversity makes it hard to place workloads efficiently: heterogeneous hardware leads to underutilized capacity because workloads can’t move freely between SKUs. The software abstraction layer — PyTorch, Triton, frameworks that hide hardware differences from ML engineers — is the only thing preventing a full rewrite of the ML stack with every new chip generation.
The open standards argument flows directly from this operational pain. If hardware vendors and customers share a common workload definition — open weight models give hardware engineers a fixed benchmark to optimize for — then hardware evolution can decouple from software evolution. Meta has contributed roughly 25% of all technical content to the Open Compute Project since its founding, and at the 2024 OCP summit it unveiled Catalina, an open-design high-powered rack for AI workloads. The standardization work extends from physical rack specs through network protocol (pushing for open scale-up and scale-out standards so different GPU types can be mixed in a single cluster) to software (PyTorch and Triton as consistent programming interfaces across hardware). At gigawatt scale, standardization isn’t idealism — it’s the only viable operational model when five hardware vendors are shipping incompatible systems every twelve months.
Prometheus — Meta’s 1-gigawatt cluster — is built across multiple data center buildings, weatherproof tents, and adjacent colocation facilities in a single region. The software challenge is that Twine and MAST, Meta’s cluster management and job scheduling systems, now need to support long-distance training jobs coordinated across a geographically distributed set of buildings. The networking challenge is that gradient synchronization for a synchronous training job can’t tolerate the latency variance of paths that cross the public internet. Hyperion, expected to scale to 5 gigawatts beginning in 2028, compounds these challenges and adds silicon photonics as a required ingredient — optical interconnects are the only viable path to pushing shoreline bandwidth beyond 3.2T while keeping power budgets under control.
One distinction the article doesn’t draw explicitly but that changes how you read the scale numbers: pretraining and inference have different bottlenecks. The “more compute improves model quality” relationship applies cleanly to pretraining, where additional FLOPs during training produce measurably better models. Inference, RLHF, test-time reasoning, and supervised fine-tuning are increasingly constrained by memory bandwidth and latency, not raw compute throughput. MTIA is efficient for recommendation inference precisely because that workload is memory-bound and throughput-bound in ways that general GPU hardware doesn’t optimize for. The next phase of infrastructure evolution will likely be as much about routing heterogeneous workloads to the hardware that matches their actual bottleneck as about adding more of any one chip type.
My take
What’s most striking in this account is the pace at which normal engineering timelines were compressed. Going from 128-GPU training jobs to 129k-GPU clusters happened in roughly eighteen months. That speed required relaxing some constraints that normally govern data center deployments — hence the tents and co-lo facilities — while tightening others, most visibly the 50x reduction in training interruptions. Knowing which constraints to relax and which to reinforce under pressure is the actual infrastructure judgment this article documents, and it’s not something that comes from a framework.
Where this is useful
- Planning datacenter topology decisions for synchronous GPU workloads where fault tolerance requires a different strategy than web serving
- Evaluating the operational cost of hardware heterogeneity when running multiple accelerator SKUs
- Understanding why open standards investment (OCP, PyTorch, open weights) is an operational decision, not just a community gesture
Apply this: When designing an AI training cluster, model the fault domain before the capacity plan — a synchronous training job requires a different reliability strategy than independently retryable requests, and the difference shows up in how you budget for hardware failure rates.
What I want to remember
Remember: In web infrastructure, a machine failure is an individual event you route around. In AI training, it’s a cluster-level stall — the fault model isn’t just larger in degree, it’s structurally different in kind.