High Availability

High availability here is not a bolted-on feature; it is the design constraint our whole system is built around, practiced continuously through every layer of the infrastructure. The rest of this page covers how.

Consistency: speed versus accuracy

Every distributed system eventually meets CAP: during a network partition, consistency and availability cannot both hold. For most products that trade-off stays abstract. For a monitor, the trade-off is the entire job description. A system that must keep working while everything else, including parts of itself, is failing cannot stop to take a vote. So on the path that matters, our platform stays available and reconciles the truth afterward.

"Reconcile afterward" is exactly wrong for some facts, though. Nobody should be paged twice for one outage, charged for an SMS that never should have gone out, or shown two different "canonical" incidents for the same rule. Those facts must be right, not eventually right.

So instead of one consistency model everywhere, the platform splits into two planes, with each fact placed deliberately.

Durability in depth

Nothing here is trusted not to fail. Durability comes in layers, and every layer assumes the one below will eventually have a bad day.

• In-region failover. Systems of record run as small clusters, not single instances. When the writer fails, a standby gets promoted automatically and traffic follows within seconds. A lost node means a reconnect, not an outage.
• Cross-region replication. Important data is not merely backed up; the data is live in more than one region at once. Measurements replicate across regions as a single logical store with a full local copy everywhere, so reads stay local and a whole region can vanish without losing data. The plane of record streams continuously to a standby region ready to take over.
• Quorum where agreement matters. The small set of facts that must be exactly right, and the coordination behind those facts, replicates with a quorum that survives losing a member. No single region's failure can corrupt or stall that set.
• Off-region backups. On top of all that, systems of record get backed up to storage in a different region from the protected data.
• Continuous verification. None of the above is assumed. Each layer is proven on a schedule by breaking things on purpose. See below.

The result: no single point whose loss is fatal and, just as important, no recovery path that has never actually run.

Chaos Availability Networking

A recovery path that has never run is a recovery path that does not exist. So recovery paths run continuously, on purpose.

Once a cycle, our scheduler picks one cell at random and takes the cell apart, piece by piece. The disruption is real: killed processes, cut network links, and full deletion of the cluster and nodes. Customer data stays safe throughout, and monitoring and ingestion reroute before anything goes down which is the only difference. Once destroyed, the same cell gets rebuilt from the ground up, then watched until the cell has rejoined the mesh, and caught up on data and begun serving. The cell under test is chosen at random and announced at the root of our API before anything is touched, so there is never doubt about what is being exercised.

The guardrails matter as much as the chaos:

• The hub (our configuration lighthouse) is never a target. Everything else is fair game.
• A cycle refuses to start until the rest of the fleet is already healthy.
• The blast radius is exactly one cell, and the rebuild draws from the same desired state that provisions every cell, so recovery is the ordinary path, not a special one.

Each cycle is heavily audited to show how our systems behave even on their worst days. The goal: when a real outage arrives, the response is already practiced.

Note: These features are in beta. Cells and schedules may vary more than usual in unexpected circumstances.