How Meta Trains Large Language Models at Scale
A plain-English tour of Meta's LLM training infrastructure — failure modes, checkpointing, and twin 24k-GPU clusters on both RoCE and InfiniBand.
The hard part of training a frontier model isn’t the GPUs — it’s that at 24,000 of them, hardware failure stops being an accident and becomes a schedule you plan around. Meta couldn’t even settle on a “right” network fabric, so it built the two leading options and ran them side by side.
Source
How Meta trains large language models at scale
By: Adi Gangidi, KR Kishore, Jenya Lee (Meta)
Published: June 12, 2024
Type: Engineering article
This one is worth saving because most infrastructure write-ups arrive after the bet has paid off and quietly drop the doubts. This article admits the fabric question was genuinely open, then explains the four problems that actually decide whether a giant training run finishes: reliability, fast recovery, preserved training state, and GPU-to-GPU connectivity.
What a 24,000-GPU training run actually needs
Meta’s training fleet used to look like the opposite of an LLM run. It trained a massive number of recommendation models — the ranking systems behind the feed — and each job needed a comparatively small number of GPUs. Generative AI flipped that shape: fewer jobs, each enormous. Once a single job spans tens of thousands of GPUs, two new facts dominate. Some piece of hardware will fail before the run finishes. And every GPU has to keep exchanging data in sync, because a slow subgroup compounds and drags the whole job down to its pace.
The mental model to hold: a large training run is one logical computer stretched across a data hall, and it only makes progress while all its parts are healthy and talking. Meta reduces the engineering to four factors — reliable hardware, fast recovery when it isn’t, efficiently preserved training state, and optimal connectivity. The rest of the piece is a stack tour where every layer serves those four.
The software layer comes first. Meta leans on PyTorch and other open source developments to move research into production quickly, alongside new algorithms and tools for large-scale training. Scheduling sits above it, allocating resources by job needs and adapting dynamically as workloads shift. Neither gets much ink — at this scale, the interesting failures live lower in the stack.
Hardware is where the constraints bite. Meta modified its Grand Teton platform — the server design built around NVIDIA H100 GPUs — raising each GPU’s TDP (the power a chip is rated to dissipate) to 700 watts and moving to HBM3, the generation of high-bandwidth memory stacked directly on the GPU. There was no time to change the cooling infrastructure, so the fleet had to stay air-cooled. That single constraint forced mechanical and thermal redesign plus a fresh validation cycle before large-scale deployment. The authors’ summary of the whole exercise: a very small degree of freedom, a very tight schedule.
The same frugality shapes the data hall itself. Power and cooling can’t be changed quickly, so Meta relocated supporting services such as readers out of the hall and packed in as many GPU racks as it could — maximizing compute density and the largest network cluster the layout allowed.
Then comes the section the whole piece leans on: reliability. Failures scale with cluster size, and a job spanning the cluster needs spare capacity standing by so it can restart fast. The failure modes Meta lists are disarmingly ordinary:
- GPUs falling off the bus — the host stops detecting them over PCIe, the link GPUs use to talk to the server. Mostly an early-life problem that settles as machines age.
- Uncorrectable memory errors in DRAM and SRAM — tracked against vendor thresholds, with repeat offenders sent back through RMA.
- Network cable failures — servers becoming unreachable, again concentrated in a machine’s first weeks.
That reads like the classic bathtub curve: high infant mortality, then stability. Buying thousands of new servers means buying their awkward first weeks too. The patching side — keeping this fleet updated while jobs run — gets its own companion piece on maintaining AI capacity.
The network section carries the article’s most interesting decision. Both leading options move data directly between machine memories at high speed. InfiniBand is a purpose-built fabric; RoCE (RDMA over Converged Ethernet) pushes the same direct-memory trick over ordinary Ethernet. Meta had four years of RoCE operations, but its largest such cluster topped out at 4K GPUs. Its InfiniBand clusters had reached 16K, but they were research systems — never tightly integrated into production, never built for the newest GPUs and networking. Neither option clearly won, so Meta built both: two 24k-GPU clusters, one per fabric, tuned for different goals — RoCE for quick build time, InfiniBand for full bisection bandwidth, meaning the capacity available across the middle of the network when you split it in half.
Both clusters trained Llama 3 — the largest model on the RoCE one — and both were tuned to equivalent performance on these workloads. Equivalence wasn’t inherited; it took three deliberate optimizations. Meta mapped the communication patterns of model, data, and pipeline parallelism onto different layers of the network topology. It rewrote collectives — the synchronized exchanges where every GPU shares its results — to be topology-aware, replacing the usual ring pattern with recursive doubling or halving so they tolerate latency better. And it load-balanced the fat flows these jobs produce, so a few huge transfers don’t saturate one path while others sit idle.
Storage gets a brief section of its own: vast capacity, high speed, and new solutions shaped per workload. The close looks ahead to hundreds of thousands of GPUs, longer distances, higher latencies, and newer GPU architectures.
My example, since the article names checkpointing but doesn’t walk through it: imagine a run that needs three weeks of GPU time, and on day nineteen a GPU falls off the bus. With no checkpoints, the run restarts at zero. With checkpoints taken every hour, you lose roughly an hour plus the time to reschedule onto spare hosts. That’s why the four factors are coupled: recovery speed only helps if the checkpoint cadence kept up.
The misunderstanding worth dropping: that adding GPUs makes training proportionally faster. Past a point, adding GPUs mostly adds interruption surface, and the real gains come from recovery, checkpoints, and topology-aware communication. It’s also why the fabric with the smaller track record could train the largest Llama 3 model.
My take
The quiet reveal is that at this scale, infrastructure decisions become experiments you run in production. Building twin 24k clusters turned a fabric argument into operational data. Most organizations settle that argument in a design review with no evidence either way; Meta paid for both and learned from both.
I’d also flag how much weight the short reliability section carries. The failure-mode list — GPUs vanishing from the bus, flaky memory, bad cables — is why checkpoints, spare capacity, and fast recovery exist. Naming your mundane failure modes candidly does more architectural work than any architecture diagram.
The caveat: “equivalent performance” is asserted, not shown. There are no utilization numbers, no failure rates, no cost comparison, so read the claim as “both fabrics were made to work,” not “the choice stopped mattering.” And the piece is Meta-shaped in a specific way — when your schedule forces you to reuse air-cooled halls, your constraints become your architecture. A company with different timelines could reasonably land elsewhere.
Where this is useful
- Designing any long-running distributed job — the four factors apply at eight GPUs, not just 24,000, and checkpoint cadence is the cheapest one to fix.
- Weighing RoCE against InfiniBand for a new cluster; this is a rare documented case of both fabrics pushed to the same scale on the same workloads.
- Hardware planning where power (TDP), cooling, and on-package memory (HBM) bound what you can actually deploy — not just GPU counts.
- Less applicable if you’ll never run synchronized multi-host jobs; this entire architecture exists for them.
Apply this: For the longest-running job you own, write down its checkpoint cadence and what a hardware failure at 90 percent completion would cost. If the honest answer is “restart from zero,” that’s the work that matters — before buying more hardware.
What I want to remember
Remember: A training cluster is a reliability system that happens to compute. Checkpoints, spare capacity, and recovery speed deserve the same design effort as the model itself — and a contested architectural choice at scale can be settled by running the contenders side by side instead of arguing.