Logarithm: A Logging Engine for AI Training Workflows and Services

You can’t reproduce a failed GPU training run deterministically, and you can’t afford to. By the time you realize something went wrong, the job may have been running for days and the state that would explain it is gone — unless you logged it continuously. The problem is doing that at scale without making logging so expensive that engineers stop doing it.

Source

Logarithm: A logging engine for AI training workflows and services

By: Partha Kanuparthy, Raunak Rungta, Shyam Sundar Chandrasekaran, Junnan Wan
Published: Mar 18 2024
Type: Engineering article

Saved because the metadata model and the SPMD debugging framing resolve a real problem I’ve seen repeatedly: distributed job failures where the symptom is spread across hundreds of interleaved log files and the root cause requires correlating rank identity with log lines that were never stamped with it.

Logging at the Scale of GPU Training

Logarithm is Meta’s hosted, serverless, multitenant logging service. It indexes 100+ GB/s of logs in real time, handles thousands of queries per second, and enforces service-level guarantees on freshness, completeness, durability, and query latency. Users write logs using any logging library — the default is glog — and query using regular expressions, metadata filters, or combinations of both. The implementation is C++20 with coroutines and async execution throughout, which the team credits for both throughput ceiling and development speed (three years from start to production).

The data model is intentionally minimal. A log stream is a named, time-ordered sequence of immutable unstructured text lines, one stream per log file per process. A process emits multiple streams — stdout, stderr, custom log files. Each line can carry zero or more typed key-value metadata pairs. Metadata is attached either at write time via a typed API or extracted from line content at ingest time using regex-based parse-and-extract rules. Typed structures are integers, floats, and strings; the extracted pairs are indexed alongside the text.

That metadata API is where the design pays off for AI training. Distributed training follows a single-program, multiple-data (SPMD) pattern: every GPU rank runs identical code on different data, with synchronization barriers between iterations. When a job fails, the failure shows up in one rank’s output while every other rank looks normal. Without per-line rank identity, debugging means opening hundreds of interleaved log streams and manually reconstructing which lines belong to the rank that broke. With Logarithm’s metadata API, rank ID is set once in thread-local context — the API attaches a glog handler that stamps every subsequent log line with it automatically. This works on third-party code and framework libraries without modifying a single logging call site.

Two debugging primitives sit on top of this model. Filter-by-call-site suppresses known-noisy logging sites: training code emits a large volume of expected high-frequency output, and collapsing it lets you walk a log stream without it dominating the view. Side-by-side rank comparison exploits the SPMD property directly. Because healthy ranks running the same code on different data should produce structurally identical log output, any divergence — extra lines, missing lines, different values at the same iteration — is a signal. The comparison view surfaces those deviations as insertions and deletions, which is usually enough to identify whether the failure is a data issue, a code issue, or a hardware event on one specific rank.

Model telemetry rides the same infrastructure. During training, model layers emit scalar metrics — learning rates, loss values, gradient norms, neuron activations — as typed key-value pairs via the metadata API. Logarithm stores these as iteration-based tensor timeseries with dimension labels: architecture, module name, neuron ID. This feeds live monitoring dashboards including an internal TensorBoard deployment and gives engineers notebook access to raw telemetry for debugging gradient explosions and convergence failures. Systems logs (PyTorch runtime, checkpointing, hardware errors) and model telemetry share the same index, which means cross-layer debugging — correlating a hardware error on rank 3 two iterations before a gradient explosion in a specific layer — is a single query rather than a time-windowed join across two separate systems.

The architectural trade-off Logarithm makes is intentional: it collocates high-throughput unstructured text with typed timeseries in a unified index. Traditional observability stacks separate these into distinct systems — Elasticsearch or Loki for logs, Prometheus or InfluxDB for metrics — requiring a shared trace ID or approximate time-window join to correlate across them. Logarithm makes correlation structural. Rank ID, iteration number, and model layer name are all indexed fields on every log line and every telemetry point. Cross-layer queries are first-class operations, not reconstructions.

One implementation detail worth retaining: the C++20 coroutine-based design isn’t an optimization applied after the fact. At 100+ GB/s ingest, any blocking operation anywhere in the pipeline accumulates enough latency to violate the freshness guarantees. Making the constraint structural — no API surface that allows blocking — means it can’t accidentally be introduced as the codebase grows.

What I want to remember

Remember: In SPMD training jobs, rank identity attached as metadata at the thread level — rather than at every logging call site — is the difference between readable logs and hundreds of interleaved streams you have to diff by hand.