Netflix's Distributed Counter Abstraction
Netflix built a counting service on top of their TimeSeries Abstraction that handles 75K requests per second globally by replacing real-time aggregation with an immutable-window event log and a continuous background rollup pipeline.
The first instinct when counting distributed events accurately is to serialize the writes — locks, Compare-And-Swap, a single authoritative row. Netflix’s counter system never serializes at all. It just makes sure you never aggregate a window that’s still receiving writes.
Source
Netflix’s Distributed Counter Abstraction
By: Rajiv Shringi, Oleksii Tkachuk, Kartik Sathyanarayanan
Published: Nov 12 2024
Type: Engineering article
The post is unusually honest about discarded approaches before describing the one they shipped. The progression through four candidate designs — single row, per-instance aggregation, Kafka pipelines, raw event log — makes the eventual architecture legible in a way that jumping straight to the solution rarely does.
Immutable windows and the event log
The core tension in distributed counting is that the operations you want to support — increment, decrement, reset, idempotent retry — are hard to combine. A single Cassandra row per counter handles reads cheaply but requires CAS under concurrent writes, and hot-key contention degrades fast. Per-instance memory aggregation removes contention but introduces data loss on restart and makes counter resets nearly impossible to coordinate reliably across replicas. Kafka-based pre-aggregation adds durability but loses the per-increment audit trail and creates partition rebalancing problems as counter cardinality grows.
Netflix’s eventually consistent counter sidesteps these by treating every increment as an immutable timestamped event stored in their TimeSeries Abstraction, which runs on Cassandra. The write path records each AddCount as an event row keyed on event_time + event_id + event_item_key. That composite is also the idempotency key — clients can retry with the same token and the unique key prevents double-counting. Wide partitions are broken up by time_bucket and event_bucket columns. The individual increments are never mutated after they land; the event log is append-only by design.
The clever piece is the aggregation boundary. The TimeSeries layer has an acceptLimit parameter that rejects events timestamped too close to now. This creates a moving boundary behind which the data is guaranteed to be stable — no new writes will arrive in that window. The rollup process only aggregates within that immutable region, so it can never race with an in-flight write. This is the insight the whole system hinges on: you do not need distributed locking to aggregate correctly if you only ever aggregate windows that have closed.
A background Rollup server runs a pipeline of in-memory queues. Whenever any API operation touches a counter — add, get, or clear — it sends a lightweight event to the Rollup server containing only the namespace and counter name, not the delta. This signal tells the rollup pipeline which counters are active, preventing full table scans across millions of stored events. XXHash routes each counter to the same queue instance, reducing duplicate aggregation work without requiring explicit locks. The pipeline drains rollup signals into a set before processing, so a counter that receives 500 writes in one coalesce window is only aggregated once. Each batch queries the TimeSeries store in parallel, reads all events since the last rollup checkpoint, sums them, and writes the new lastRollupCount and lastRollupTs to a Rollup store — also on Cassandra. That result is then cached in EVCache for fast point reads.
The read path returns the cached rollup value immediately and fires a background rollup to advance the timestamp. A stale read self-remediates on the next access; no external operator is needed to detect or correct drift. The tradeoff is acknowledged plainly: the counter value lags its most recent write by a configurable margin, typically seconds. For the use cases this serves — A/B experiment metrics, feature impression counts, interaction aggregations — that lag is acceptable.
The best-effort type (also part of the abstraction) is an entirely different animal. It maps directly onto EVCache’s native incr and decr operations. No event log, no rollup pipeline, no idempotency. It runs at sub-millisecond latency for experiments that need approximate counts immediately and can tolerate cache eviction. The control plane lets teams pick between types per namespace without touching client code.
The experimental accurate counter is a thin extension of the eventual type: at read time, it computes lastRollupCount + delta, where delta is a live aggregation of events since the last rollup timestamp. The cost is proportional to how many events and time partitions that scan touches. For frequently accessed counters the window stays narrow and the scan is cheap; for cold counters the scan cost climbs. This is honest about its constraints: it works well for counters that are accessed often enough to keep the delta window small.
The part most teams underestimate when building similar systems is the convergence logic for infrequently accessed counters. A counter that only receives writes occasionally never gets a rollup trigger from the read path. Netflix handles this with the lastWriteTs column, updated on every write using Cassandra’s USING TIMESTAMP for predictable LWW semantics. The Rollup server inspects this to determine whether a counter still needs to be circulated, continuing aggregation until the rollup timestamp catches up to the last known write time.
My take
The mental model shift here is treating counting as a read-time aggregation problem rather than a write-time consistency problem. Every approach that tries to maintain an accurate count in real-time ends up fighting write contention. Netflix steps outside that frame by making the write path purely additive — events only ever land — and moving all the hard work into a background process that operates on data the writers have already finished touching. The immutable window is the key that unlocks this: once you have a principled boundary, aggregation becomes a sequential scan, not a concurrent modification problem.
Where this is useful
- High-cardinality counters where hot-key contention on a single row creates a bottleneck under concurrent write load
- Systems that need per-increment auditability or the ability to recount a historical window after the fact
- When NOT applicable: when the application genuinely requires the current count before deciding whether to proceed with a write — low-latency inventory reservation or rate limiting need a different approach where reads block until the write is confirmed
Apply this: Before reaching for distributed locking or CAS on counter updates, ask whether the read latency tolerance is measured in milliseconds or seconds — if it’s seconds, an event log with a background rollup unlocks much simpler write semantics.
What I want to remember
Remember: A counter you aggregate asynchronously within an immutable window has none of the contention problems of a counter you keep accurate in real-time — the correctness guarantee shifts from write-time serialization to read-time convergence, which is a much cheaper property to maintain at scale.