How Meta Keeps Its AI Hardware Reliable
When hardware failures caused over 66% of Llama 3 training interruptions at Meta, the response was a detection-and-mitigation stack spanning silicon design, cluster-level diagnostics, and changes to training algorithms.
The default assumption when a training run diverges is that something is wrong with the code. At Meta’s cluster scale, the right first suspicion is hardware — and the specific challenge isn’t that hardware fails loudly, but that it can corrupt a multi-week training run silently, without a single error log, until the model diverges and the job dies.
Source
How Meta keeps its AI hardware reliable
By: Meta
Published: Jul 22 2025
Type: Engineering article
Saved because this treats hardware reliability as a systems engineering problem across the full stack — from silicon telemetry through training algorithm design — rather than a hardware procurement or vendor QA problem. The mitigation taxonomy is practical and the PVF integration shows where the reliability and model teams meet.
The Three Fault Categories and Why Silent Ones Are the Hard Problem
Meta classifies hardware faults in AI clusters into three categories, and understanding why the third one is fundamentally different from the first two explains most of the engineering choices in this piece.
Static faults are the simplest: a device fails to power on or enumerate on the network. Health checks catch them before a training job starts. As clusters scale, static faults happen more often in absolute terms but remain tractable — a node that doesn’t come up gets flagged, pulled from the pool, and replaced. The diagnostic work is high-volume but well-understood.
Transient faults are temporary events: a cosmic-ray bit flip in an SRAM cell, a momentary voltage droop causing a logic error. Most transients are caught by error-correcting code (ECC) in HBM memory before they produce wrong values. The ones that slip through become the third category.
Silent data corruptions (SDCs) are hardware faults that produce wrong values without triggering any error signal. They pass ECC. They don’t crash anything. The corrupted value enters computation, propagates forward through the network, and eventually surfaces as a NaN, an exploding gradient, or a subtle, invisible degradation in model quality. Running the Llama 3 herd made this concrete: over 66% of training interruptions traced to hardware — SRAMs, HBMs, processing grids, and network switches all contributing. The majority-hardware failure rate seems counterintuitive until you consider that training runs on thousands of accelerators in a fully synchronous environment. A single bad node doesn’t degrade the job gradually; it stalls or corrupts the whole thing.
The detection challenge specific to SDCs is data-dependence. A corrupted memory cell in one GPU’s HBM only manifests when a specific address range is written under specific thermal and voltage conditions. A node that passes diagnostic health checks before training starts may fail two hours into a run when load and temperature cross a threshold. This means detection cannot happen only before training; it has to happen during training.
Meta’s infrastructure-level detection strategies converge on divide-and-triage. When NaN propagation appears in a training run, a binary search process splits the cluster — running mini-training iterations on progressively smaller subsets to isolate which node is producing the corruption. For data-dependent SDCs this requires reproducing the exact forward pass: same checkpoint, same batch, same data order. A complementary strategy runs a known-good reference model for a few iterations before any new job starts — not as a reliability guarantee, but as a fast filter for systematic configuration errors and non-data-dependent faults that would kill a run immediately anyway.
High-frequency checkpointing serves double duty. Normally, checkpointing frequency trades off against training throughput. During an active fault investigation, increasing checkpoint frequency narrows the window of potentially corrupted computation. Once a failure is suspected, the cluster rolls back to a recent clean state, the suspect node is quarantined, and investigation proceeds without blocking the full job.
At the training algorithm level, gradient clipping is standard practice for stability but also functions as an SDC canary. A sudden spike in gradient norms before NaN propagation often signals a parameter corruption in the backward pass rather than a legitimate training instability. Setting monitoring thresholds on gradient norm behavior catches many SDC events before they escalate to job termination. The most expensive and most reliable approach is shadow nodes: a shadow pool runs the same training steps as the live cluster from the same checkpoint, and outputs are compared at random intervals. If shadow and live diverge, only the nodes that produced differing results are investigated. The overhead is substantial in compute and infrastructure complexity, but the detection rate is high and the blast radius is contained.
The longer arc of this work converges on MTIA. Meta describes a factory-to-fleet reliability approach for its custom silicon program: SDC detection baked into the chip architecture from the design phase, manufacturing diagnostics feeding failure-mode signatures back to the design team, and fleet-level tooling — Hardware Sentinel for early wear-out detection, FleetScanner for identifying hardware outliers, Ripple for correlating fault patterns across the fleet. The ambition is to catch fault signatures before they affect production training runs rather than discovering them mid-job.
My take
The most revealing detail is that Meta’s SDC detection work started in 2016, matured through 2018–2022, and by 2023 they were co-authoring detection frameworks with Google, Microsoft, AMD, NVIDIA, and Intel under the Open Compute Project. Competitors sharing reliability research is a strong signal that no single vendor has solved this and that the problem scales with every generation of more complex silicon. Any team scaling AI training past a few hundred GPUs will eventually hit the same failure modes; the question is whether they’ve built the detection stack before the first long training run dies.
Where this is useful
- Diagnosing training instabilities in large clusters where NaN sources are ambiguous between code and hardware
- Designing fault detection into distributed training infrastructure before scaling — retrofitting shadow-node monitoring at 10,000 GPUs is far more expensive than building it at 100
- When NOT applicable: single-node or small multi-GPU training where SDC probability during a run is low enough that standard ECC coverage is sufficient without additional detection layers
Apply this: Add gradient norm monitoring with alerting thresholds to any training run longer than a few hours — a sudden norm spike before divergence is more often hardware than a hyperparameter issue.
What I want to remember
Remember: An SDC looks like a model divergence problem but is a hardware problem — the diagnostic is to halve the cluster, not adjust the learning rate.