High Availability

So you're chasing those nines. But how do you actually know you have them? A system can run clean for a year and still be one bad afternoon from its first real outage. You're on a good cloud, sure. Maybe you even run across zones or datacenters. But are you fine when DNS goes sideways? When BGP quietly routes you into a black hole? When the provider's status page is the last to admit it?

Those are the questions you ask of an always-on service, and they are the ones we ask of ourselves first. latency is the thing that watches for everyone else's outages, so it cannot afford to have its own. High availability here is not a feature we bolted on, it is the design constraint the whole system is built around, practiced continuously and deeply through every layer of the infrastructure. The rest of this page is how.

BGP

Routing is the layer where outages hide in plain sight. A prefix gets withdrawn, a peer leaks a more specific route, an upstream quietly null-routes you, and from the inside everything still looks fine. Your servers are up. Your health check, running in the same place as your service, is green. The traffic just is not arriving.

The only way to see that is from the outside, from many places at once. We watch every target from a spread of vantage points that sit on different networks and in different regions, so a routing problem shows up the way it actually behaves in the wild: as disagreement. When one set of vantage points can reach a target and another cannot, that split is the signal. A single blind spot cannot hide a blackhole while a dozen other eyes are still looking.

We hold our own front door to the same standard. It is not one address in one place. It is spread across regions and answered by whichever cell is nearest and healthy, so no single route, if it goes dark, takes the service with it. If a region drops out of the routing table, resolvers are steered to the next closest one and the remaining cells keep answering. The thing that watches for everyone else's routing failures is built so it cannot be quietly cut off by one of its own.

Consistency: speed versus accuracy

Every distributed system eventually meets CAP: when the network partitions, you get consistency or availability, not both. For most products that is an abstract trade-off. For a monitor it is the entire job description. The one system that has to keep working while everything else, including parts of itself, is failing cannot stop to take a vote. So on the path that matters, we choose to stay available and reconcile the truth afterward.

But "reconcile afterward" is exactly wrong for some things. You do not want to be paged twice for one outage, charged for an SMS that never should have gone out, or shown two different "canonical" incidents for the same rule. Those facts have to be right, not eventually right.

So we don't pick one consistency model and live with it everywhere. We split the platform into two planes and tune where each fact lives.

Durability in depth

Nothing here is trusted not to fail. Durability is built in layers, and every layer assumes the one below it will eventually have a bad day.

• In-region failover. The systems of record run as small clusters, not single instances. When the writer fails, a standby is promoted automatically and traffic follows it within seconds. A lost node is a reconnect, not an outage.
• Cross-region replication. The data that matters is not merely backed up, it is live in more than one region at once. Measurements replicate across regions as a single logical store that keeps a full local copy everywhere, so reads are always local and a whole region can vanish without taking its data with it. The plane of record streams continuously to a standby region ready to take over.
• Quorum where agreement matters. The small set of facts that must be exactly right, and the coordination behind them, is replicated with a quorum that survives losing a member. No single region's failure can corrupt it or stall it.
• Off-region backups. On top of all that, the systems of record are backed up to storage that lives in a different region from the data it protects, because a backup that shares fate with its source is not a backup.
• Continuous verification. None of the above is assumed to work. It is proven, on a schedule, by breaking things on purpose. See below.

The result is a system with no single point whose loss is fatal and, just as importantly, no recovery path that has never actually been run.

Chaos Availability Networking

A recovery path you have never run is a recovery path you do not have. So we run them, continuously, on purpose.

Once a cycle, latency picks one of its cells at random and pokes and prods until we destroy it, piece by piece. This involves real disruption such as killing processes, cutting network links and completely deleting the cluster and its nodes. Don't worry your data is and always will be safe and monitoring and ingestion is rerouted before things go down. Once destroyed, we rebuild that same cell from the ground up and wait until it has rejoined the mesh, caught its data back up, and started serving again. The cell under test is chosen at random and announced at the root of our API before anything is touched, so there is never any doubt about what is being exercised.

The guardrails matter as much as the chaos:

• The hub (our lighthouse) is never a target. Everything else is fair game.
• A cycle refuses to start chaos until the rest of the fleet is already healthy first.
• The blast radius is exactly one cell, and the rebuild draws from the same desired state that provisions every cell, so recovery is the ordinary path, not a special one.

Each cycle is heavily audited to help us learn how our systems behave even on their worst days. The goal is to prepare for that eventuality so when we experience an outage for real, it will not be the first time we have dealt with it.

Note: These features are in beta. Cells and schedules may vary more than usual in unexpected circumstances.