Gemini 1.5: Google's Million-Token Long-Context Breakthrough
Google's Gemini 1.5 announcement explains how mixture-of-experts efficiency and a million-token context window changed what models can process in one go.
The assumed path to a better model in early 2024 was to make it bigger. Google’s mid-size Gemini 1.5 Pro matched its largest model while using less compute — and the real breakthrough wasn’t raw intelligence, but how much the model could hold in working memory at once.
Source
Our next-generation model: Gemini 1.5
By: Google (Sundar Pichai, CEO of Google and Alphabet, and Demis Hassabis, CEO of Google DeepMind)
Published: Feb 15 2024
Type: Product announcement
This is the announcement that pushed long context into the mainstream, and it stays worth reading because it explains two connected ideas better than most coverage did: why a mixture-of-experts architecture lets a mid-size model match a giant one, and what a million tokens of context actually let you do with whole documents, films, and codebases.
How Gemini 1.5 made the context window the story
In early 2024, if your question spanned more than a few thousand words — a long contract, a codebase, a stack of transcripts — you didn’t hand it to a model directly. You chopped the material into chunks, retrieved the pieces that looked relevant, and hoped the right parts survived into the prompt. Google had shipped Gemini 1.0 Ultra, its most capable model, the week before this post. The surprise of Gemini 1.5 was that its first release, the mid-size 1.5 Pro, performed at a broadly similar level to 1.0 Ultra while using less compute — and it carried a much stranger claim: a context window of up to 1 million tokens, the longest of any large-scale foundation model at the time, offered as a limited preview.
A context window is a model’s working memory: everything it can look at while producing one answer. It is measured in tokens, the building blocks models process — chunks of words, but also pieces of images, stretches of audio and video, and lines of code. Gemini 1.0 worked with 32,000 of them. Gemini 1.5 Pro came with a standard window of 128,000, a private preview at 1 million, and a research result of 10 million successfully tested.
What made a mid-size model able to carry that is the post’s second idea: a Mixture-of-Experts (MoE) architecture. A traditional transformer is one large neural network that lights up fully for every input. An MoE model is split into smaller “expert” networks, and for each piece of input it selectively activates only the most relevant expert pathways. Picture replacing one person who does every job with a panel of specialists, where a dispatcher routes each task to the few who know it. That specialization is what made Gemini 1.5 cheaper to train and serve — and, in Google’s telling, faster to iterate on. This was no overnight bet: the post traces the lineage through Google’s own Sparsely-Gated MoE, GShard-Transformer, and Switch-Transformer research.
What does a million tokens hold? Google’s equivalents: one hour of video, eleven hours of audio, over 700,000 words, or a codebase of more than 30,000 lines — in a single prompt. The demos are chosen to stress that “single prompt” part. Given the 402 pages of Apollo 11 mission transcripts, 1.5 Pro could reason about conversations, events, and details spread across the whole document. Given a 44-minute silent Buster Keaton film, it analyzed plot points and picked up small details a viewer could easily miss — reasoning over video, not just text. Given over 100,000 lines of code, it suggested modifications and explained how different parts worked together. My own example of the practical shape this takes: pointing the model at an entire repository and asking where session refresh happens and what breaks if you change it — no embedding pipeline, no chunking, no retrieval step to get wrong.
The performance claims come with numbers, and the numbers come with fine print. 1.5 Pro beat 1.0 Pro on 87% of the benchmarks Google uses for developing its large language models, while landing broadly level with the much larger 1.0 Ultra. On the Needle In A Haystack evaluation — a small fact deliberately buried somewhere inside a huge block of text — it found the needle 99% of the time even at 1 million tokens. It’s worth being precise about what that measures: spotting a planted fact, not deeply weighing every token at once. Google more or less says so itself, noting it is “continuously developing new evaluations” for capabilities no large-scale model had before. The strangest evaluation deserves a second look: on the Machine Translation from One Book benchmark, the model was given a grammar manual for Kalamang, a language with fewer than 200 speakers, and learned to translate English into it from that manual alone — no fine-tuning — at a level comparable to a person learning from the same material. That is in-context learning: the skill gets acquired from the prompt itself.
The safety section reads differently once you notice what it is testing. The checks Google describes — content safety, representational harms, red-teaming — were built for shorter inputs, and the post admits new tests are being developed specifically for long-context risks. My observation: when the input can be an entire private archive in one prompt, the evaluation surface grows with the window.
Availability closed the loop with the honest tradeoffs. The million-token window was a limited preview through AI Studio and Vertex AI, free during testing, with longer latency explicitly promised alongside “significant improvements in speed” still to come. The wider release would carry the standard 128K window, with pricing tiers planned from 128K up to 1 million as the model improved. The misunderstanding to avoid is treating the bigger window as a bigger notepad. The difference between retrieving fragments of a document and attending over the whole thing in one pass changes what you can build — but at launch you paid for the ceiling in latency, and the deepest claims rested on benchmarks that were still being invented.
My take
Read together, the two headline features are one economic story. Mixture-of-experts and long context sit in separate sections, but the efficiency is what makes the window affordable: serving a million-token prompt with a giant dense model would have been economically absurd, so the capability arrived only when the cost of attention dropped. That’s the pattern worth generalizing — new model abilities usually wait on serving economics, not on ideas. The second thing the post reveals without saying it: every demo has the same shape. One whole artifact in context — one mission’s transcripts, one film, one grammar book. The thing being quietly attacked isn’t a rival model; it’s the retrieval infrastructure everyone had just finished building. Retrieval didn’t die, but from this announcement on, “does it fit in the context window?” became the first architecture question, and the pipeline became the fallback.
Where this is useful
- Choosing between long context and a retrieval pipeline for document-heavy features — whole-contract review, whole-repository questions, long-media analysis.
- Working with video and audio corpora, where transcribe-then-chunk pipelines throw away structure that a single-pass multimodal window keeps.
- Budgeting realistically: the top of the window costs latency and money, and the standard tier is what most workloads should target.
- Less applicable to short-context chat products — a big window buys nothing there and may add cost.
Apply this: Pick one task where you currently chunk and retrieve documents. Estimate its total size in tokens (by Google’s own arithmetic, roughly 700 words per 1,000 tokens), and if the whole thing fits inside a window you can afford, run it as one prompt — then compare the answer against your retrieval pipeline.
What I want to remember
Remember: A context window is working memory, not storage. The win isn’t holding more files — it’s letting the model see the whole problem in one pass. And the architecture that makes the window cheap matters as much as the window itself.