How Meta Built Large-Scale Cryptographic Monitoring
Meta built a cryptographic monitoring system on top of FBCrypto that tracks every key and algorithm in use across the fleet without sampling, using buffered aggregation to make zero-loss logging viable at production scale.
You cannot migrate away from a weak cryptographic algorithm if you don’t know where it’s used — and at the scale of Meta’s infrastructure, “don’t know” is the default state unless you build something to change it.
Source
How Meta built large-scale cryptographic monitoring
By: Meta
Published: Nov 12 2024
Type: Engineering article
Saved because it solves a problem that sounds boring — logging — but is actually the prerequisite for every cryptographic migration, post-quantum readiness effort, and security posture decision at fleet scale.
The system design behind zero-loss crypto logging
FBCrypto is Meta’s managed cryptographic library. Most of Meta’s core infrastructure services use it, which means it sits on critical and high-volume code paths. Knowing how it is used — which keys, which algorithms, at what frequency, with what success rate — is both a reliability signal and a security requirement. The problem is that naive logging at the frequency of cryptographic operations would overwhelm any standard logging infrastructure. Meta discloses in the article that roughly 0.05% of CPU cycles across the fleet are spent on X25519 key exchange alone. If every operation produced a log line, the write throughput would be enormous and the storage costs untenable.
The standard answer to this is sampling: only log every Nth operation, increase N until the capacity problem goes away. Meta explicitly rejected this. Sampling means most logs are discarded, which means key overuse goes undetected, weak algorithm call sites are invisible, and the data you need for a migration is incomplete. The monitoring goal required zero omissions.
The solution is a buffered aggregation strategy. Instead of writing a log per operation, FBCrypto maintains an in-memory buffer on the client host. Every cryptographic operation increments a count for its unique event signature — key name, algorithm, method, and whatever other fields the logging schema tracks. A background thread flushes this buffer to Scribe, Meta’s standard logging framework, at a preconfigured interval, exporting each event along with its accumulated count. A single flush might represent millions of operations as a handful of rows. Downstream, data lands in Scuba for short-term warm storage and Hive for long-term cold storage, where analytics and alerting run against it.
Because the aggregation is implemented inside FBCrypto itself, none of the logging logic is visible to the clients of the library. The interface is unchanged. Any service that calls encrypt() gets logging transparently.
Two implementation details are worth examining closely because they address failure modes that would silently degrade the system otherwise.
The first is partially randomized flushing. When a large set of machines restarts at roughly the same time — a common pattern during rollouts and restores — their buffer flush timers would all fire simultaneously, producing a write spike to Scribe followed by a long underutilization period. The fix is a per-host randomized delay applied to the first flush. After that initial jitter, flushes proceed on their normal interval from different phases, spreading load uniformly across time.
The second is derived key aggregation. FBCrypto supports derived crypto, where child keysets are derived from a parent using a key derivation function. Large-scale use cases can generate millions of derived keys. The original logging implementation created a distinct row per derived keyset, which produced enormous log volume and back-end pressure. The fix aggregates all child key operations under the parent key’s name. This is a deliberate approximation — the per-child count becomes a pessimistic upper bound rather than a precise figure — but the security-relevant property (detecting parent key overuse) is preserved while log volume drops dramatically.
The concurrent data structure underlying the client-side buffer is folly::ConcurrentHashMap, which handles heavy writes from multiple threads with atomic guarantees. This choice matters because FBCrypto is called from multithreaded services; a naively locked map would serialize all logging writes and create measurable contention on hot paths.
The operational payoff described in the article spans several dimensions. Key overuse detection becomes proactive rather than reactive — the system can observe trends over time and flag keys approaching their safe usage limits before they become a vulnerability. Algorithm migration is accelerated because every call site for a deprecated algorithm is visible in the data, and teams can be notified with enough lead time to plan the work. Deployment tracking is immediate: because library version is part of the logged schema, engineers can see exactly which clients have picked up a new FBCrypto version in real time rather than waiting for fleet-wide rollout signals. The article specifically notes that this is proving useful for post-quantum cryptography readiness — identifying which services use algorithms that will be vulnerable to quantum attacks and prioritizing their migration.
The reliability angle is less obvious but equally important. Because logs include success rates alongside call counts, anomalous drops become visible during large migrations. A new code path that silently fails on a subset of machines produces a change in the success rate distribution before it produces user-facing errors. The monitoring dataset functions as an early-warning system for cryptographic infrastructure changes, not just a security audit trail.
My take
The architectural insight is that observability over a shared library is most effective when it lives inside the library rather than relying on clients to instrument themselves. Meta benefits here from having a single canonical library that everyone uses — the monitoring only had to be built once and every consumer got it automatically. Organizations with fragmented cryptographic tooling would need to solve the unification problem first before this approach becomes viable.
The tradeoff that stands out is the derived key aggregation. Deliberately trading precision (per-child counts) for capacity (parent-level aggregation) is the right call, but it requires you to know in advance that the security-relevant property — overuse detection — survives the approximation. That reasoning needs to be explicit and revisited whenever the use of derived keys changes. Aggregation strategies that seem safe under one usage pattern can become blind spots under another.
Where this is useful
- Any team responsible for migrating a fleet away from deprecated cryptographic primitives, where knowing all call sites before starting is a hard requirement
- Shared library maintainers who need operational visibility into how clients use their library without modifying client code
- Post-quantum readiness programs where the first step is an accurate inventory of vulnerable algorithm usage
- Not useful as a substitute for cryptographic correctness — the monitoring catches overuse and algorithm choices, not logic errors in how primitives are composed
Apply this: Build logging into the library, not into the callers — a single buffered aggregation inside FBCrypto gave Meta zero-loss fleet-wide visibility that no client-instrumentation approach could have matched.
What I want to remember
Remember: Zero-loss logging at high frequency is achievable without sampling — aggregate events in memory by their unique signature and flush counts, not individual records, so you retain full coverage while controlling write throughput.