Disaggregated Scheduled Fabric: Scaling Meta's AI Journey
Meta's Disaggregated Scheduled Fabric (DSF) splits the traditional switch chassis into two hardware tiers coordinated in software, eliminating elephant flow and entropy problems by scheduling traffic as cells rather than routing it as flows.
The chassis switch is the unexamined assumption at the center of most data center networking. You scale up by buying a bigger chassis, and you scale out by adding more of them. Meta’s Disaggregated Scheduled Fabric paper starts from the premise that the chassis itself is the bottleneck — and that the answer is to break the switch open.
Source
Disaggregated Scheduled Fabric: Scaling Meta’s AI Journey
By: Meta (Ron He and Ankur Singh)
Published: Oct 20 2025
Type: Engineering article
Saved because it is the clearest published account of what happens when the routing-based approach to AI network scaling is genuinely exhausted — when ECMP tuning, path pinning, and traffic engineering have all been tried and each has failed in a different direction. DSF is not a routing improvement; it is a structural replacement.
How DSF breaks the chassis model to build a bigger switch
The paper is about what happens when the most demanding compute workload in the world — AI training — finally outgrows the physical limits of the most capable single switch chassis available. Meta’s answer was Disaggregated Scheduled Fabric, or DSF, a next-generation network architecture that splits the logical chassis into two separate hardware tiers and coordinates them in software. The result is a virtual switch with no theoretical hardware ceiling, built from cheaper, focused components instead of ever-larger monolithic boxes.
The background section documents why the conventional fixes failed before DSF, and it is worth reading carefully because each failure explains a design decision. Running traditional IP fabric with ECMP for AI training ran into three specific failure modes: elephant flows from GPU collectives that hash onto a single fabric link and cause head-of-line blocking; low entropy from the small number of large flows that ECMP distributes unevenly despite available capacity; and suboptimal fabric utilization as a combined effect, forcing significant overprovisioning just to maintain steady performance under skewed load. The proposed fixes each addressed one problem at the cost of another. BGP path pinning reduced the entropy problem in steady state but reverted to ECMP under failure, reintroducing collisions at precisely the worst moment. Load-aware ECMP handled fat flows but created out-of-order packets, which RDMA communication cannot tolerate. Pre-computed traffic engineering tailored to specific model communication patterns worked until network scale made it computationally intractable and too slow to respond to failures. No single routing-layer fix closed all three problems simultaneously.
DSF addresses all three failure modes together with a single architectural change: separate the fabric from the servers with an explicit domain boundary, and schedule traffic inside the fabric domain instead of routing it. The system has two hardware components. Interface Nodes (INs), also called Rack Disaggregated Switches (RDSWs), connect to the servers and the external data center network; they handle Layer 3 routing, external connectivity, and the protocol boundary between Ethernet and the fabric domain. Fabric Nodes (FNs), called Fabric Disaggregated Switches (FDSWs), sit purely inside the fabric domain; they are high-speed switching elements with no Layer 3 routing responsibilities whatsoever. To the external network, the entire collection of INs and FNs presents as a single logical switch, with an aggregate external port count equal to all the external ports across all INs — a virtual chassis switch that scales far beyond what any physical chassis could offer.
What makes the fabric domain different from conventional Ethernet is how traffic moves through it. Instead of sending whole packets across the fabric as discrete units, DSF breaks packets into fixed-size cells at the ingress IN and sprays those cells across all available fabric paths simultaneously. At the egress IN, cells are reassembled into complete packets before delivery to the destination server, which receives in-order Ethernet frames as if nothing unusual happened inside the fabric. This packet-spraying approach bypasses the elephant flow and low entropy problems entirely: the unit of distribution is a cell, not a flow, so every path in the fabric carries exactly equal load regardless of the size or frequency of the flows that produced those cells. A 100 GB allreduce operation spreads evenly across every FDSW path rather than hashing onto one or two links.
The scheduling mechanism that coordinates cell spraying is credit-based. Ingress INs do not spray freely; they request credit tokens from the appropriate egress INs, which grants transmission permission based on current path availability and congestion state. This real-time credit exchange ensures that spraying decisions are made with current information rather than stale estimates, and that ingress traffic never exceeds what egress can absorb. Virtual Output Queuing (VOQ) backs this up by giving each packet a virtual queue targeted at its specific destination port and service class, with each VOQ scheduled independently. The combination — credit-based spraying plus per-VOQ independent scheduling — is what gives DSF its lossless delivery guarantee without PFC or pause frames at the server-facing ports. The system is VOQ-based end to end, running on the open OCP-SAI standard with FBOSS as the network operating system.
The control plane is FBOSS extended to manage multi-ASIC coordination across disaggregated nodes. FBOSS communicates state across the distributed IN/FN system through the FBOSS State Database (FSDB), providing real-time synchronization across the virtual chassis. The hardware choices reflect the role split: RDSWs use Jericho3-AI chips, chosen for their deep buffer characteristics that absorb the bursts AI collectives generate at the network edge. FDSWs use Ramon3 chips, which are pure fabric-domain switching elements optimized for high-speed cell distribution without the buffer requirements of an external-facing node. RDSW-to-FDSW connections use 2×400G FR4 optics.
The production topology builds an AI Zone from Scaling Units — groups of GPU racks connected to RDSWs — all wired through a common layer of FDSWs. GPU-to-RDSW connections are rail-optimized, meaning each GPU’s NIC connects to a specific RDSW in a way that keeps allreduce and allgather traffic within a single RDSW before it touches the fabric domain. Both allreduce and allgather are latency-sensitive operations in LLM training, so keeping the first hop local reduces round-trip time for the most frequent collective patterns. Two identical network planes are stacked to form a DSF L1 zone, which becomes the building block for the larger data-center-scale cluster.
The honest edge case is operational complexity. DSF replaces a familiar model — one switch, one management plane, one failure domain — with a distributed system that must coordinate state across dozens of RDSWs and FDSWs simultaneously. The credit-based protocol means a misbehaving or slow FDSW can starve credit for specific egress INs, and failures inside the fabric domain present as credit availability problems rather than as conventional link-down events. The paper describes the failure handling architecture at a high level, but the operational depth of running a live DSF cluster at multi-thousand-GPU scale under production training workloads is the area where the engineering detail would not fit in a conference paper. FBOSS open-source development is presumably where that operational knowledge accumulates.
The misunderstanding to avoid is reading DSF as a leaf-spine variant with fancier load balancing. The key difference is the domain boundary. Traditional leaf-spine treats the entire network as one Ethernet domain where packets are routed from source to destination. DSF has an explicit boundary where packets are decomposed into cells, scheduled inside a non-routable fabric domain that the servers never see directly, and reconstructed before being returned to Ethernet. That boundary is the origin of the performance properties — near-optimal fabric utilization, lossless delivery without PFC, and linear scalability — and it is also what makes DSF incompatible with retrofitting into existing switch-based designs. You cannot add cell-spraying to a conventional CTSW; you have to replace the architecture.
My take
The paper makes visible a decision that most infrastructure engineers never face: when do you stop tuning the existing system and replace the abstraction? Meta tried BGP pinning, load-aware ECMP, and pre-computed traffic engineering in sequence. Each was a sensible response to the failure mode of the previous approach, and each introduced a new failure mode of its own. DSF is the admission that routing-layer solutions have a ceiling when traffic entropy is structurally low and flows are structurally large — and that the ceiling is below the GPU scale Meta needs to hit. The disaggregation move is clever because it does not ask the rest of the data center to change: the IN/FN collection still looks like a single switch to the network outside it. The complexity is contained inside the fabric domain, which is also where it can be managed by the people who understand it.
Where this is useful
- Evaluating next-generation AI cluster network designs where existing Clos-based fabrics are hitting oversubscription or PFC headroom limits.
- Understanding why traffic engineering approaches that work at moderate GPU scale (hundreds to low thousands) stop working at high GPU scale (tens of thousands and above).
- Reasoning about the operational trade-off between a monolithic chassis with a single failure domain and a disaggregated fabric with distributed failure modes and credit-based scheduling.
- Smaller clusters where traffic entropy is higher and elephant flows are less dominant — the problems DSF solves are specific to AI collective patterns; general data center traffic does not justify the architectural complexity.
Apply this: Before adding more overprovisioning to a Clos fabric to handle AI training load, measure per-link utilization during a training job to see whether the issue is total capacity or skewed distribution. Skewed distribution is the signature of the entropy and elephant flow problems DSF was built to eliminate, and overprovisioning does not fix distribution — it just makes the skew less visible.
What I want to remember
Remember: Routing-based approaches to AI network scaling hit a structural ceiling because routing operates on flows, but AI collective traffic has too few flows and too much per-flow volume for any hash-based approach to distribute evenly — the fix is to operate on cells below the flow level, which requires a domain boundary where packets are disassembled and reassembled.