Indexing Code at Scale with Glean
Meta open-sources Glean, a schema-first code indexing system that treats code intelligence as a queryable knowledge graph rather than a fixed-output pipeline.
Most code intelligence tools work by examining code as it loads. At a few hundred files that’s fine. At millions of lines across dozens of languages, go-to-definition works for the files the IDE has loaded and misses the million it hasn’t touched. Find-references drops hits silently. IDE cold starts stretch into minutes. Indexing isn’t a solved problem — it’s just invisible when the scale is small enough to paper over it.
Source
Indexing code at scale with Glean
By: Meta Engineering
Published: Dec 19 2024
Type: Engineering article
Saved because the design rationale behind Glean — schema-per-language, a single query language, stacked immutable databases — is rare thinking about what code intelligence infrastructure actually needs at scale, and the incremental indexing model solves a problem most teams treat as unsolvable.
A query layer for code knowledge
The standard mental model for code tools is a pipeline: parse, extract symbols, dump into a format, wire up an editor. The prevailing format is LSIF (Language Server Index Format), and most IDEs lean on it. Glean starts from a different question: instead of asking “what format should code intelligence data live in,” it asks “what queries do developers actually need to run against code?”
That shift matters more than it sounds. LSIF is built around one specific use case — IDE code navigation — so it exports what IDEs need and nothing else. Glean deliberately avoids prescribing what data gets stored. Each language gets its own schema. C++ stores #include dependency graphs, using statement resolution chains, and function declarations with full type signatures. Python stores something different. A schema abstraction layer — similar in spirit to SQL views — then sits on top of all the language-specific schemas and provides a language-neutral surface. A query like “give me all declarations in this file” works regardless of whether the target is C++ or Rust. You don’t have to choose between detailed language-specific data and a lowest-common-denominator view; the derivations that power the abstraction layer produce both from the same underlying store.
The query language, Angle (an anagram of “Glean”), is declarative and logic-based. A query for folly::parseJson in C++ resolves in about a millisecond against a RocksDB-backed store. Angle can derive information either ahead of time or on the fly, which is the mechanism powering the abstraction layer. Schema definitions look like type definitions with record fields and their types. Predicates are the things you can query; queries return facts.
The harder technical problem Glean solves is incremental indexing. A monorepo with millions of lines changes constantly. If re-indexing the whole repository on every commit takes hours, the index is perpetually stale — sometimes many hours behind. The cost of indexing has to be O(changes), not O(repository). Glean achieves this by stacking immutable databases. A base layer holds the full repository at some revision. When code changes, a new layer is pushed on top that non-destructively adds new facts and hides stale ones from below. The stacked result looks, from a client’s perspective, like a single coherent database at the latest revision. This design also means the index can be queried at multiple revisions simultaneously without storing multiple full-sized copies of the data — each layer is a delta.
Figuring out which files to re-index on a given change is the “fanout” problem. For C++, fanout is calculated by querying #include chains: find all files that include one of the changed files, then recurse until no new files appear. That computation is itself a Glean query — the system uses its own query language to determine the scope of its own re-indexing work.
The user-facing layer on top of all this is Glass, an open-source symbol server that abstracts Glean into a clean API. A code browser needs one call — documentSymbols(repo, path, revision) — to get every definition, reference, and span in a source file. Glass handles the Glean complexity. Beyond the browser, Glass assigns every symbol a stable symbol ID (e.g., REPO/cpp/folly/Singleton) that doesn’t change even when the definition moves. That stability makes URL-per-symbol possible without links rotting on refactors.
One application that seems minor but is genuinely powerful: indexing diffs. When a change is posted for review, Glean indexes the changeset and produces a “diff sketch” — a machine-readable summary of what changed at the semantic level: new classes, removed methods, new calls to a function. That sketch drives code navigation inside the code review tool, giving reviewers accurate go-to-definition on the code being reviewed, not just on the committed repo. It also powers semantic search over commit history — connecting a production stack trace to the specific commits that modified the affected functions changes root-cause analysis from guesswork to a database query.
A non-obvious constraint: Glean’s power comes from schema quality. For cross-language navigation to work — jumping from an RPC call site to the service implementation in another language — someone has to define the schema for that relationship. The system is general, but generality doesn’t mean automatic coverage. Every new language or cross-language bridge requires a schema author to invest in the abstraction.
The common misunderstanding about Glean is treating it as “a better code search backend.” The diff sketch use case shows why that framing is too narrow. It’s closer to a queryable knowledge graph of your repository, where new use cases — dead code detection, build dependency analysis, semantic commit search — drop in by adding schemas and queries rather than building new extraction pipelines from scratch. The generality was intentional from the start, and it’s what lets the system grow beyond the original code navigation use case.
My take
The insight is that code intelligence is really a database design problem. Once you accept that — and build accordingly — the tools become composable instead of siloed. Most teams never get here because they build point solutions: a dead code tool, a code review integration, a doc generator, each with its own extraction pass. Glean runs one indexed pass and serves all of them from the same store.
Where this is useful
- Multi-language monorepos that need accurate cross-language navigation and cross-repo reference finding
- Codebases where dead code detection and API usage analysis directly affect development velocity
- Code review tooling that needs real semantic context — hover types, go-to-definition on diffs — rather than just text diffs
- When NOT applicable: small single-language repos where an LSP server loads in under a second; the schema authorship and infrastructure overhead isn’t worth it until scale makes the pain obvious
Apply this: Before building a new code intelligence tool, ask whether it’s really a schema and a query over data you already have. If it is, invest in the data layer once rather than writing another one-off extraction script.
What I want to remember
Remember: Incremental indexing works by stacking immutable database layers, not by mutating a single store — so you can query multiple revisions cheaply without duplicating the full dataset.