RoCE Networks for Distributed AI Training at Meta Scale
Meta's SIGCOMM 2024 paper shows how RoCEv2 over Ethernet became the backbone for training Llama-scale models across tens of thousands of GPUs — and why routing, not hardware, was the hard problem.
Everyone assumes that making AI training faster is about getting faster GPUs. The actual bottleneck — for training runs spanning tens of thousands of GPUs over weeks — is what happens between them. Meta’s 2024 SIGCOMM paper shows that the network is not just infrastructure for training; at this scale, it is the training constraint.
Source
RoCE networks for distributed AI training at scale
By: Meta
Published: Aug 05 2024
Type: Engineering article
Saved because it documents every iteration of a production decision — from star topology to AI Zones to cross-building ATSW layers — with the performance data that killed each prior approach. The routing section alone is worth the read: Meta tried ECMP, path pinning, and two variants of QP scaling before landing on a solution, and the paper shows the cost of each failure.
Why RDMA over Ethernet became the bet
The paper opens with a problem that doesn’t read like a network problem: how do you train a model that needs hundreds of billions of parameters? At that scale — Meta’s Llama 3.1 405B was a concrete example — a training job lasts weeks and requires tens of thousands of GPUs to execute a tightly coordinated dance of gradient exchanges. The network connecting those GPUs is not a commodity concern. It is the schedule of every compute step.
Meta’s answer was RDMA Over Converged Ethernet version 2, or RoCEv2 — a protocol that encapsulates the RDMA service inside UDP packets, giving it the throughput and latency characteristics of InfiniBand while running on standard Ethernet switches. The choice was deliberate and came with trade-offs. InfiniBand offers better out-of-box performance and tighter hardware integration, but it binds you to a single vendor ecosystem. Ethernet gives you vendor choice, operational familiarity, and the ability to share physical infrastructure across workloads — so Meta took the harder path of making Ethernet behave like a purpose-built fabric.
The physical separation is the first design decision worth holding onto. Every training cluster runs two completely independent networks: a frontend (FE) network that handles data ingestion, checkpointing, and logging, and a backend (BE) network reserved entirely for the RDMA traffic that constitutes training itself. Separating these lets the BE evolve independently, and it lets the team tune it for one traffic class: enormous, bursty, elephant flows from GPU collectives.
The BE topology started as a simple star — a few racks connected to one switch — and moved through several iterations as GPU counts grew. The production form is a two-stage Clos arrangement called an AI Zone. At the leaf sits the Rack Training Switch (RTSW), which connects GPU servers inside a rack over copper DAC cables. At the spine sit modular Cluster Training Switches (CTSWs), wiring together all RTSWs across the cluster with deep buffers to absorb the bursts that AI collectives generate. When training demands grew beyond a single AI Zone — once LLMs needed more GPUs than one zone could hold — Meta added an Aggregator Training Switch (ATSW) layer connecting CTSWs across buildings, expanding the RoCE domain at the cost of intentional oversubscription on the cross-zone links.
Routing is where the paper earns its place on the shelf. The traffic characteristics of AI training are nothing like web traffic: flows are few and predictable (low entropy), they run at near line rate in milliseconds-long bursts, and each burst is a single enormous flow (elephant flows). Standard Equal-Cost Multipath (ECMP) routing, which distributes flows by hashing on a five-tuple, breaks badly here because hashing a small number of flows onto a large number of paths produces collisions that look like congestion. The first fix — path pinning, routing packets to a specific path based on destination slice — worked when racks were fully allocated to one job, but failed on partial allocations and collapsed under any link failure. Under fragmented job placement, path pinning caused 30% training performance degradation because uplink capacity became uneven and collision-prone. Meta’s short-term fix was to 2× the RTSW uplink bandwidth, which solved the symptom by buying back headroom but proved too expensive to sustain.
The real fix was Queue Pair (QP) scaling combined with Enhanced ECMP (E-ECMP). QP scaling, implemented in the collective communication library, spreads a single logical communication relationship across many queue pairs — multiplying the number of distinct five-tuples in flight and restoring the entropy that ECMP needs to work well. E-ECMP extends the hash to include the destination QP field from inside the RoCE header, using the switch ASIC’s User-Defined Field (UDF) capability. Together, QP scaling and E-ECMP produced a 40% improvement in AllReduce collective completion time versus baseline ECMP. The key insight: you don’t fix low-entropy traffic by changing the routing algorithm alone — you fix it by redesigning the entropy source at the application layer and then making the switch aware of it.
Congestion control runs on DCQCN (Data Center Quantized Congestion Notification), Meta’s chosen variant for lossless RDMA over Ethernet. DCQCN uses ECN marks from the network to signal congestion to senders, with PFC as a backstop that pauses frames before packets drop. The paper documents careful parameter tuning — marking thresholds, timers, the alpha factor — across the evolving topology. The goal was to maintain lossless operation without triggering PFC storms, where pause frames propagate backward across the fabric and stall traffic far from the original congestion point.
Two edge cases matter. Cross-zone traffic is deliberately oversubscribed; Meta’s response was a topology-aware training scheduler that solves a “minimum cut” problem to assign GPU ranks in a way that minimizes cross-zone communication. The same scheduler learns the logical position of servers in the fabric and recommends rank assignments before the job starts — a feedback loop between the network’s physical structure and the compute job’s communication pattern that most discussions of distributed training ignore entirely. And the paper is explicit that performance results depend on the specific combination of QP scaling, E-ECMP, and DCQCN being tuned together; none of the three works in isolation.
The misunderstanding to avoid is treating RoCEv2 as a drop-in alternative to InfiniBand. It isn’t; it requires explicit choices about congestion control, routing entropy, and scheduler integration that InfiniBand handles differently or through vendor tooling. The performance you get depends almost entirely on how well those three pieces are tuned together — which is also why the paper is useful: it shows the exact sequence of failures that led to each tuning decision.
My take
The paper is a rare document because it publishes the failures in order. Most infrastructure writing describes the final system as if it arrived fully formed. Here you get path pinning failing at 30%, the 2× bandwidth band-aid, the ECMP collision problem, and the routing entropy fix in sequence — which makes it possible to reason about when each approach breaks rather than just knowing which one Meta ended up with. The part that stays with me is the scheduler integration. The network and the training job have to know about each other: the scheduler learns where GPUs sit in the topology, and that physical knowledge determines which rank assignment minimizes cross-AI-Zone traffic. That coupling is invisible in diagrams that show network and compute as separate layers, but it is where a significant fraction of the performance actually comes from.
Where this is useful
- Designing the network for a new GPU cluster where you can choose the transport protocol and topology from scratch.
- Diagnosing training performance degradation that seems correlated with job placement or rack allocation patterns.
- Evaluating whether to implement QP scaling in a collective communication library before investing in hardware changes.
- Running workloads with high flow entropy (e.g., web serving, recommendation inference with many small flows) — the routing problems here are specific to low-entropy AI collective traffic and do not transfer directly.
Apply this: Before attributing training throughput problems to GPU hardware or the collective library, measure the per-path traffic distribution at the RTSW uplinks. Uneven uplink utilization under a single job is the fingerprint of the entropy problem, and it tells you whether routing or hardware is the actual bottleneck.
What I want to remember
Remember: Low-entropy AI traffic breaks ECMP not because ECMP is wrong but because the entropy it needs doesn’t exist in the flow headers by default — fix the entropy source at the application layer (QP scaling), make the switch hash on the new entropy (E-ECMP), and then tune congestion control to keep the fabric lossless. Those three changes are not separable.