Appearance
Architecture
latency runs on a multi-homed, cell-based network spread across the planet. No single provider, region, or machine is load-bearing. The system is built so that the thing watching for outages doesn't have outages of its own.
We deliberately mix cloud, bare metal, colocation, and the occasional VPS. Any one of them can disappear without taking the platform with it, and probing your services from genuinely independent networks is the whole point.
Cells
The unit of deployment is a cell: a self-contained cluster in one location that can run checks, evaluate rules, self-verify, and raise alerts entirely on its own. A cell holds everything it needs: workers, the rules engine, local state, and NATS.
Cells are independent and autonomous by design but also able to sync together as a combined system. Cells are connection aware and able to degrade to "local only" rather than going completely offline should the majority of the system experience failures or a severe outage. When full connectivity returns, services will automatically begin to re-sync.
New regions are added by standing up another cell, not by scaling a central system. Capacity and geographic coverage grow the same way, one cell at a time.
The NATS communication
Cells are stitched together by NATS, a lightweight messaging system, arranged as a super-cluster of leaf nodes. The latency mesh follows three main concepts:
- Leaf nodes. Each cell runs a NATS server that connects outward to the core super-cluster as a leaf node. The cell never needs an inbound port open to the world, and it holds its own subject space locally. A probe VPS in another country participates in the mesh without us trusting it with anything more than the messages it's meant to see.
- Subject-based routing. Work and events flow over hierarchical subjects, not point-to-point connections. A job published to
worker.<region>is picked up by a worker (group) in that region; results and incidents (incident.opened,incident.closed) flow back over their own subjects along with other metadata. - Queue groups. Multiple workers subscribe to the same subject as a queue group, so a job is delivered to exactly one of them. That's how a cell load-balances checks across its workers, and how the platform survives a worker (or a whole cell) dropping out: the remaining subscribers simply pick up the slack.
Because the streams are subject-addressed and leaf-connected, adding a cell is almost free: it dials into the super-cluster connecting to the known subjects, announces its own for location aware jobs, and starts pulling work. No central registry to update, no routing table to manage.
State and consistency
Every cell keeps its own state: a local key-value store and NATS JetStream for work and orchestration, ClickHouse for the check samples it records, and a link back to the authoritative stores at the core, where identity and the catalog's service graph live in Postgres.
That split is deliberate. Measurement and detection are local and eventually consistent, so a cell keeps working, and keeps its data, even when it is cut off from the rest of the mesh. The records that have to be exact, your account, your quotas, the single canonical incident for a rule, are strongly consistent at the core. It is why a cell can drop to "local only" and reconcile later without losing a check or paging you twice.
We treat that speed-versus-accuracy trade as a dial we tune per feature, not a single global setting. See High Availability for how we choose, and where.
Compute
Applications run on virtualized or bare-metal servers depending on load and availability. Untrusted workloads, including user-defined functions, are isolated with gVisor, and everything is subject to aggressive, continuous security scanning.
Security
Every endpoint connected to our platform is secured by our mesh using Nebula. Cross-cell traffic travels over authenticated VPN connections, and probe nodes at the edge are handed only the narrowly-scoped credentials they need for the work they do, never the platform's primary keys.