Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism
How Meta runs Llama 3 and Llama 4 in production by attacking three separate bottlenecks — model size, sequence length, and expert count — with three independent parallelism strategies, and why all three hit the same wall: inter-GPU communication.
A language model that fits on one GPU in training won’t fit on one GPU in production at real latency requirements. And when you spread it across many GPUs, you discover that the bottleneck isn’t how fast the chips compute — it’s how fast they talk to each other.
Source
By: Cen Zhao, Xiaodong Wang, Jianyu Huang
Published: Oct 17 2025
Type: Engineering article
Saved because it names a non-obvious unifying pattern across three different parallelism problems and gives concrete numbers (percentages of end-to-end latency, token counts, time measurements) instead of hand-waving about “communication overhead.”
The communication bottleneck across all three axes
Running a large language model for inference is not one problem — it’s two, with contradictory resource profiles. Prefill (processing the prompt) is compute-bound: the GPU multiplies matrices at full speed. Decoding (generating each output token one at a time) is memory-bandwidth-bound: the GPU loads weights from memory on every step but does relatively little compute per byte transferred. Most decisions that help prefill hurt decoding and vice versa, which is why Meta’s parallelism strategy has three independent axes rather than one unified approach.
Tensor parallelism (TP) handles the most fundamental scaling challenge: a model that won’t fit on one GPU. TP splits individual layers — attention blocks, MLP layers — across multiple GPUs so each device holds and processes a slice. The catch is that after each layer the devices need to synchronize partial results via allreduce, a collective communication operation. At Meta’s scale, this step consumes up to 30% of end-to-end latency. The team’s Direct Data Access (DDA) approach bypasses NCCL (NVIDIA Collective Communications Library) and RCCL (its AMD counterpart) by having each GPU read slices of each other’s results directly, avoiding the multi-hop gather in the standard library. On AMD MI300X hardware, DDA outperforms RCCL by 10–50% for decode (small messages) and 10–30% for prefill, cutting overall time-to-inference-token (TTIT) by roughly 10%.
Context parallelism (CP) addresses a different axis entirely: sequence length rather than model size. Llama 4 supports 1M and 10M token contexts, which can’t be computed in one pass on a single host — not because of weight size but because attention scales quadratically with sequence length. CP distributes the sequence across devices, with each device attending to its chunk and passing key-value data around a ring to accumulate the full attention result. Meta implemented two ring attention variants and achieved under one minute for 1M tokens on a single H100 host, and under one minute for 10M tokens across 32 H100 hosts. With Llama 3 405B, 128K-token prefill runs in 3.8 seconds across 16 nodes — near-linear scaling with host count, which is not something you get for free.
Expert parallelism (EP) applies specifically to Mixture-of-Experts (MoE) architectures, where the model routes each token to a subset of specialized sub-networks. A MoE model with hundreds or thousands of experts can’t fit on a single host by design, so EP shards the expert layers across devices. The routing step requires an all-to-all communication pass — each device needs to send tokens to whichever devices hold the matching experts, and receive back the results. This contributes 10–30% of end-to-end latency, particularly for decode messages in the 100KB–2MB range. Meta is exploring direct GPU-to-GPU communication and async overlap of compute with communication to reduce this.
The cross-cutting insight is that all three bottlenecks — TP allreduce, CP ring attention synchronization, EP all-to-all — are communication bottlenecks, not compute bottlenecks. Adding faster GPUs doesn’t fix them. The gains come from changing the communication pattern (DDA for TP) or overlapping communication with compute (async all-to-all for EP). This means hardware benchmarks that only measure FLOPS will miss where production LLM inference actually spends its time.
The logical extension is N-D parallelism: combining CP, TP, EP, and pipeline parallelism across a single job, so each GPU belongs to multiple parallelism groups simultaneously. A GPU might be in a TP group with three neighbors, a CP group with fifteen others, and a data-parallel replica group with another set. Managing overlapping groups without deadlock or stragglers requires precise scheduling, but it’s what makes a model like Llama 3 405B serveable in production rather than only on paper.
Prefill/decode disaggregation is the other direction this work points toward. Because the two phases have different resource profiles, running them on the same hardware means one phase always leaves capacity on the table. Disaggregated serving routes prompts to compute-heavy hardware for prefill and routes decoding to memory-bandwidth-heavy hardware. Parallelism choices for each phase can then be tuned independently — higher TP for prefill, higher EP for decoding — without the compromises a unified deployment forces. This is where the infrastructure is heading, and it’s worth understanding the three-axis parallelism framework before that transition arrives.
My take
The framing of “three separate bottlenecks, all communication” is the real payload here. Every axis of LLM parallelism hits a communication wall, not a compute wall. This changes what hardware procurement looks like — interconnect bandwidth and topology matter as much as GPU FLOPS for a production inference cluster. It also means that scaling laws from training don’t transfer cleanly to inference scaling decisions, because training can hide communication in pipeline bubbles in ways that autoregressive decoding cannot.
Where this is useful
- Choosing parallelism strategy when deploying a new model that doesn’t fit on a single GPU
- Hardware procurement for LLM inference clusters — where to spend on interconnect vs. compute
- Diagnosing slow inference that more GPUs haven’t fixed
- When NOT applicable: small models that fit on a single device without any parallelism
Apply this: Before adding more GPUs to a slow LLM inference deployment, profile end-to-end latency and check what fraction allreduce or all-to-all operations consume — if it’s above 20%, the bottleneck is communication, not compute.
What I want to remember
Remember: Prefill is compute-bound, decode is memory-bandwidth-bound — running both on the same hardware always leaves capacity on the floor, and all three parallelism axes eventually hit communication, not FLOPs.