Cloudflare Outage on February 20, 2026
A 6-hour BYOIP outage caused by an API parameter parsed as an empty string instead of a presence flag, which led an automated cleanup task to treat every customer BGP prefix as pending deletion.
Automation is supposed to reduce human error — but on February 20, 2026, a new automated cleanup task didn’t reduce the blast radius of the manual process it replaced; it amplified it by orders of magnitude and sustained it for six hours before full recovery.
Source
Cloudflare outage on February 20, 2026
By: Cloudflare (David Tuber, Dzevad Trumic)
Published: Feb 21 2026
Type: Engineering article
This postmortem is notable for how precisely it traces a six-hour BGP incident back to a single empty string in a Go HTTP query parameter — and for its honest accounting of why recovery took so long: three distinct customer impact states, no state snapshots, and no automated rollback path. The section on the Addressing API architecture makes the structural risk visible in a way that applies far beyond BYOIP.
When an Empty String Becomes a Delete-Everything Command
On February 20, 2026, at 17:56 UTC, Cloudflare began withdrawing BYOIP customer prefixes from its global BGP advertisements — not intentionally, and not because of any attack. An automated sub-task had been deployed two weeks earlier as part of an ongoing reliability initiative called Code Orange: Fail Small. Its purpose was to replace a risky, manual process for removing BYOIP prefixes from Cloudflare’s network when customers requested it. The irony at the center of this postmortem is that the task introduced as part of a reliability initiative caused the reliability incident.
The sub-task queried the Addressing API to find prefixes that were pending deletion, using this request:
resp, err := d.doRequest(ctx, http.MethodGet, `/v1/prefixes?pending_delete`, nil)On the server side, the handler checked for the flag like this:
if v := req.URL.Query().Get("pending_delete"); v != "" {
// fetch only prefixes pending deletion
}The bug lives in the gap between these two. In HTTP, a query parameter appended with no value — ?pending_delete — returns an empty string when read with Query().Get(). The server’s condition, v != "", evaluates to false for an empty string. The handler falls through to its default path: return all BYOIP prefixes. The sub-task received the full customer list, interpreted every prefix as pending deletion, and began deleting them along with their associated service bindings — systematically, in a loop, until an engineer identified and terminated the process at 18:46 UTC.
The deletion ran for 50 minutes. During that window, 1,100 prefixes out of 4,306 total BYOIP prefixes were withdrawn from Cloudflare’s global BGP advertisements. The change was iterative, not instantaneous — the sub-task processed the list progressively — which is the only reason the full customer base wasn’t affected. But 25% of BYOIP prefixes being withdrawn is a major incident regardless: the customers whose prefixes were withdrawn had their services and applications become unreachable from the Internet. Connections entered a BGP Path Hunting loop, traversing networks searching for a route that no longer existed, until each connection timed out.
The recovery story is where the six-hour duration comes from, and it is as instructive as the cause. Affected customers were not all in the same state. Some had only their prefix advertisements withdrawn; they could restore service themselves by toggling re-advertisement in the Cloudflare dashboard, and were directed to do so at 19:19 UTC. Others had prefixes withdrawn and some service bindings removed — partial self-remediation was possible, but not complete. A third group had prefixes withdrawn and all service bindings deleted from the edge; these customers could not use the dashboard at all, because there was no longer a service bound to their prefix to re-advertise. Restoring service for this group required a global configuration update that had to push reconstructed state back to every machine in Cloudflare’s edge fleet. That completed at 23:03 UTC, more than five hours after the sub-task was shut down.
The absence of state snapshots made slow recovery almost inevitable. Cloudflare’s Addressing API stores customer configuration in an authoritative database, and changes propagate immediately to the edge. There was no rollback mechanism. There were no snapshots of pre-incident state to restore from. Engineers had to reconstruct correct configuration from database records and push it out through a manual global deployment. The Code Orange initiative was building exactly this kind of snapshot-based rollback system, but it was not yet in production.
Two distinct failure modes compound here. The first is the API ambiguity: ?pending_delete without a value is syntactically valid but semantically different from ?pending_delete=true. The server code had no way to distinguish a flag with presence-implies-true semantics from a flag with no value — both would match v != "" only if the value was a non-empty string, which neither form provides. The fix is to define the parameter as a typed, required value with an explicit valid set, so the client and server share an unambiguous contract.
The second failure mode is the absence of a circuit breaker on the deletion process itself. Withdrawing more than a small number of BGP prefixes in a short window is an unusual operation under any circumstances. A rate limiter or threshold check — “if we are about to delete more than N prefixes, abort and page an engineer” — would have stopped the damage after the first few deletions. No such guard existed. The sub-task’s scope was bounded only by the size of the result set returned by the API query, which, due to the bug, was the entire customer list.
Staging tests passed because the mock data didn’t reflect production at the scale and shape needed to expose the query bug. Code review focused on the customer-initiated removal path — the happy path — rather than the ambient behavior of the background sub-task executing independently against real production data. Both are common blind spots: testing the intended flow while underexamining what a background process does when its inputs are wrong.
My take
This outage shows that the blast radius of automation is not inherently smaller than the blast radius of the manual process it replaces — it is often larger, because automation operates faster and doesn’t pause to question unusual inputs the way a human operator would. The guard rails that made the original manual process safe (an engineer making a deliberate decision for a specific customer, one at a time) were not carried into the automated version. Moving a dangerous operation to automation only reduces risk if the automation includes equivalent safety constraints.
Where this is useful
- Reviewing automated tasks that perform bulk deletions or withdrawals against production data — ask whether they have explicit rate limits, deletion thresholds, and dry-run modes before any destructive step
- Auditing APIs that use presence-as-flag query parameters, which are inherently ambiguous when the calling code doesn’t control whether a value is appended
- Evaluating whether your operational state can be rolled back independently of rerunning a deployment — if the answer requires reconstructing from a database backup, that gap is the highest reliability risk to address
- Not when: the automation is append-only or read-only within a bounded scope; the automation risk profile is very different when writes are idempotent and easily reversible
Apply this: Add a maximum-change threshold to any automated task that deletes or withdraws production records — abort and page if the planned change count exceeds a safe ceiling.
What I want to remember
Remember: Automating a risky manual process inherits the full risk unless you explicitly transfer the human judgment that was bounding it.