Networking at the Heart of AI Scale: @Scale Networking 2025 Recap
A recap of @Scale: Networking 2025, where engineers from Meta, Google, Microsoft, and others reported how AI workloads have fractured into a portfolio of network requirements that no single fabric design can satisfy.
Engineers have treated networking as a solved layer for decades — configure the fabric, hand the bandwidth to the application, done. AI at the scale of 129,000-GPU clusters has broken that assumption. The network is now the research frontier, the abstraction target, and the operational bottleneck simultaneously.
Source
Networking at the Heart of AI — @Scale: Networking 2025 Recap
By: Meta
Published: Sep 26 2025
Type: Engineering article
Saved because this is a rare honest account of what actually breaks at GPU cluster scale — not a vendor pitch or a research preview, but practitioners from a dozen companies reporting what they are running into in production.
What AI Did to Networking
The @Scale: Networking 2025 conference was the largest in the event’s history. Engineers from Meta, ByteDance, Google, Microsoft, Oracle, AMD, Broadcom, Cisco, and NVIDIA gathered to report what they have learned building and operating the network infrastructure behind large-scale AI. The throughline was consistent across companies: networking has moved from a background concern to a foreground constraint, and the problems are not going to be solved by faster hardware alone.
Two structural shifts framed the day. The first is the scale of physical infrastructure commitment. Across the industry, hundreds of billions of dollars of infrastructure investment are being planned over the next several years. At Meta specifically, this means gigawatt-scale GPU clusters named Prometheus and Hyperion, the largest transoceanic fiber cable systems ever laid, and a construction portfolio that now includes sprung structures — large temporary buildings that can be erected faster than permanent data centers to bring compute capacity online without waiting for traditional construction timelines. The capital intensity is visible. The operational complexity that follows from connecting and maintaining infrastructure at this pace is less visible, and that is where the network story lives.
The second shift is workload fragmentation. For the first several years of large-scale GPU cluster operation, the dominant workload was foundational model training — a relatively homogeneous access pattern that allowed Meta to design its fabric around one well-understood use case. Meta went from 4,000-GPU clusters to 24,000-GPU to 129,000-GPU in under two years, using Ethernet and RoCE (RDMA over Converged Ethernet), solving new problems at each jump. But in the twelve months before the conference, the workload mix changed. Mixture-of-experts models, reasoning models, reinforcement learning from human feedback, post-training pipelines, synthetic data generation, and distributed inference are all now running in parallel. Each of these has different network requirements. Some are sensitive to tail latency. Some need high bisection bandwidth between all nodes. Some have irregular collective communication patterns that differ from the dense all-reduce workloads foundational training runs. The fabric that was tuned for one pattern is now being asked to serve a portfolio.
This is the practical tension the conference surfaced: the network was co-designed with the original training workload, and that workload has multiplied. There is no single fabric design that is obviously optimal for all of these patterns simultaneously, which is why the conversations at the conference were about the full stack — NIC behavior, congestion control algorithms, routing policy, and collective communication implementations — rather than any one layer.
The mental model Meta put forward for the network’s role in AI infrastructure is worth holding onto: the network’s job is to make the entire infrastructure look like one gigantic GPU to the model. The model should not need to know how many physical machines it spans, where they are located, or how bandwidth between them is provisioned. Everything below the collective communication API is the network’s problem to abstract. This is elegant as a principle and difficult as an engineering target. Achieving it requires handling varying distances and bandwidths in both the scale-up domain (interconnect within a node or rack) and the scale-out domain (interconnect between racks and pods), hardware heterogeneity across different accelerator families, NIC implementations, and fabric types, and tight co-design with the GPU-side software stack. A NIC that makes conservative retransmission decisions can look like a hung process to a training framework. A congestion control algorithm tuned for web traffic behaves badly inside an all-reduce collective at scale. The entire stack from the application-layer collective down to the transceiver has to be understood as a single system.
The reliability framing was equally grounding. At 129,000 GPUs, hardware failure is not an exception — it is a background constant. Mean time between any component failure in a cluster of that size is measured in hours. The operational question is not how to prevent failure; it is how to detect failures fast and route around them without stopping a training run that may have been running for weeks. The network operations posture has to be fault-tolerant by design. Every retry, every path-failover, every congestion event that drags one collective operation is a potential stall for the entire training step. The tail of the latency distribution matters more here than the mean.
The forward-looking emphasis from the keynotes was on optionality. The workload mix will keep changing. The hardware generation below the network will keep turning over on a sub-two-year cadence. The argument for open, programmable, and disaggregated network approaches — rather than vertically integrated vendor solutions — is not that they are better on any single axis today, but that they can absorb these changes without requiring a full redesign each time the workload shifts or a new GPU generation lands.
My take
The most honest thing about this recap is what it implies about specialization: you cannot hire network generalists and expect them to debug AI training performance. The problems that show up — collective communication stalls, congestion control interactions with NCCL, NIC firmware behavior under high-fan-out all-reduce — require understanding the entire stack simultaneously. That is a different job description than traditional network engineering, and most organizations are not staffed for it.
Where this is useful
- When planning large-scale distributed training infrastructure and evaluating fabric choices
- When debugging training throughput that does not match theoretical hardware specifications
- When building the team to operate AI infrastructure at scale
- When NOT applicable: small-scale fine-tuning or inference workloads where standard cloud networking capacity is sufficient and the overhead of custom fabric design is not justified
Apply this: Before sizing a GPU cluster, map the full workload portfolio — the network fabric needs to serve not just training but the inference, post-training, and synthetic data workloads that will run alongside it.
What I want to remember
Remember: At sufficient scale, the network is not the plumbing beneath AI workloads — it is the constraint that determines which workloads are possible and how efficiently they can run.