Taming the tail utilization of ads inference at Meta scale
How Meta's ads inference team cut timeout error rates by two-thirds and squeezed 35% more throughput from the same hardware by fixing how load is measured, balanced, and predicted in a multi-tenant inference fleet.
Average utilization is a lie. When your fleet’s top 5% of servers are timing out, the other 95% running at 50% do nothing to help your SLA. The headroom you thought you had isn’t where you need it.
Source
Taming the tail utilization of ads inference at Meta scale
By: Rohith Menon, Bikash Sharma, Deepak Tiwari
Published: Jul 10 2024
Type: Engineering article
Saved because it makes the argument for fixing measurement before adding capacity — and the per-model load counter fix is one of those solutions that’s obvious in hindsight and invisible until you see it.
Why tail utilization breaks at inference scale
The surface metric for an inference cluster is usually average utilization. Average looks fine; the cluster hums along at 60% and there’s headroom everywhere. But ad delivery SLAs break not when averages rise but when outliers do. Meta defines tail utilization as the utilization level of the top 5% of servers when ranked by utilization. A fleet that looks 60% loaded in aggregate can have its top 5% servers at 90%, and those servers are where timeouts come from.
The problem compounds in two ways. First, the relationship between queries per second and CPU utilization is non-linear — when you push a server past a certain point, latency and failure rates climb steeply. Second, when you need to add headroom, you add capacity cluster-wide, even though only the tail servers actually need relief. The tail forces you to over-provision the entire fleet.
Meta’s ads inference runs on Shard Manager, where each model is a shard and multiple shards cohabit a host. The first thing the team addressed was that ServiceRouter — the load balancer — was making decisions with stale data. In the default “load-header” mode, the load signal piggybacked on responses, which meant the data could be seconds old. For inference requests taking tens of milliseconds apiece, stale load data steers requests straight into already-overloaded servers. Switching to polling — asking for fresh load before dispatching each request — added negligible overhead but dramatically improved routing accuracy. The power-of-two-choices algorithm, which samples two candidate servers and picks the less loaded, worked well precisely because the load data was now fresh.
The deeper fix was more counterintuitive. ServiceRouter was balancing load across hosts using a host-level counter that summed all outstanding requests on a host, across all models. This sounds reasonable until you notice that different models have wildly different resource profiles — a heavy embedding model and a lightweight ranker share a host but have nothing in common computationally. A host-level load counter makes hosts look equally loaded even when one model is crunching and another is idle. This misled ServiceRouter (routing decisions), Shard Manager (placement decisions), and ReplicaEstimator, which assumes all replicas of the same model carry similar load. The fix — a per-model load counter, where each model exposes its own resource consumption instead of lumping into the host total — aligned three systems that had been operating on incompatible assumptions.
There’s a lesson here about metric choice that transfers beyond ads infrastructure. CPU utilization as a load counter was tempting but wrong because it’s aggregated over time rather than instantaneous, and it ignores I/O wait. The team found that “outstanding examples CPU” — the estimated total CPU time for all active requests, normalized by core count — was the best proxy. It captures the cost of in-flight work rather than the current state of the machine.
The memory bandwidth problem was a surprise. When a new model replica landed on a host already running other models, CPU spikes appeared. Investigation with dynolog perf instrumentation showed that stall cycles were climbing and memory latency was increasing exponentially above 65–70% memory bandwidth utilization. The spike looked like a CPU problem but was actually a memory bus contention problem. The fix was to include memory bandwidth as a placement resource in Shard Manager, not just CPU cores. This kind of hidden resource contention — where the bottleneck is invisible at the metric level you’re watching — is worth keeping in mind whenever multi-tenant placement is involved.
Snapshot transitions are the final pressure point. Ads models retrain continuously, and when a large model transitions to a new snapshot, it briefly doubles the replica count on the cluster until the old version scales down. During peak traffic, this triggers Shard Manager rebalancing that violates CPU thresholds. The team’s answer was a snapshot transition budget: transitions only proceed when utilization is below a configured threshold. Fast scale-down of old snapshots kept staleness bounded while the budget prevented transitions from firing during peak load.
Predictive replica estimation closed the loop. Rather than reacting to load spikes by spinning up replicas — which takes time and typically over-provisions — the system now forecasts resource usage up to two hours ahead using recent patterns. This makes autoscaling proactive, which matters specifically because the non-linear QPS-to-utilization curve means spikes arrive faster than reactive systems can respond.
The combined result: 35% more throughput per unit of capacity, two-thirds fewer timeout errors, p99 latency halved.
My take
The per-model load counter fix reveals something broader: when multiple systems share an assumption about a metric, an error in that metric cascades silently through all of them. The interesting part isn’t the fix itself — it’s that ServiceRouter, Shard Manager, and ReplicaEstimator had been running on the same wrong shared premise since the service was built, and nobody had a unified view of all three assumptions at once. Distributed system bugs are often really distributed measurement bugs.
Where this is useful
- Multi-tenant inference serving where models share hosts
- Any service where reactive autoscaling visibly lags behind traffic spikes
- When planning capacity additions — measure utilization distribution first, not just mean
- When NOT applicable: single-model, dedicated-hardware deployments where multi-tenancy isn’t a factor
Apply this: Before adding capacity to a struggling service, plot the utilization distribution across your fleet and verify that your load counter actually measures the resource unit that’s competing — not an aggregated proxy.
What I want to remember
Remember: In multi-tenant systems, a host-level load metric is almost always wrong — derive load signals from the resource unit that actually competes for capacity, not the machine it runs on.