Appearance
Architecture
latency runs on a multi-homed, cell-based network spread across the planet. No single provider, region or machine is load-bearing. The system watching for outages is built to avoid outages of its own.
The fleet deliberately mixes cloud, bare metal, colocation and the occasional VPS. Any one provider can disappear without taking the platform down, and probing services from genuinely independent networks is the whole point.
Lighthouse
The configuration that boots this entire network comes from a single authoritative source that we call the Lighthouse. This is a special service in our network responsible for deploying changes to our platform but push based and operates completely independent of the cells described below.
Cells
Our unit of deployment is a cell: a self-contained cluster in one region that handles ingest, runs checks, evaluates rules, self-verifies and raises alerts independently if needed. Each cell holds everything needed for that work: workers, the rules engine, local state and NATS.
Cells are independent and autonomous by design, and also sync together as one combined system. Cells are connection aware and degrade to "local only" instead of going offline when most of the system suffers failures or a severe outage. When full connectivity returns, services re-sync automatically.
New regions are added by standing up another cell, not by scaling a central system. Capacity and geographic coverage grow the same way, one cell at a time.
The NATS communication
Cells are stitched together by NATS, a lightweight messaging system, and our event queue of choice, arranged as a super-cluster. The latency mesh follows three main concepts:
- NATS Nodes. Each cell runs a NATS server that connects outward to the core super-cluster. No cell needs an inbound port open to the world, and each cell holds a local subject space. A probe VPS in another country participates in the mesh with trust limited to the messages meant for that probe.
- Subject-based routing. Work and events flow over hierarchical subjects, not point-to-point connections. A job published to
worker.<region>is picked up by a worker (group) in that region; results and incidents (incident.opened,incident.closed) flow back over their own subjects along with other metadata. - Queue groups. Multiple workers subscribe to the same subject as a queue group, so each job reaches exactly one subscriber over the JetStream durable queue. Queue groups load-balance checks across a cell's workers in that region, and keep the platform running when a worker (or a whole cell) drops out: the remaining subscribers or newly provisioned ones pick up the slack.
Because streams are subject-addressed, adding a cell is almost free: the new cell dials into the super-cluster, connects to the known subjects, announces location-specific subjects for regional jobs, and starts pulling work.
State and consistency
Every cell keeps local working copy: NATS key-value store and JetStream for event work and orchestration, ClickHouse for recorded check samples and telemetry, Postgres for users and configuration.
Our services are engineered to use the fastest connections (replicas) for reads while serving data. While this is not safely possible with writes we back these up by leaning on JetStream for heavy or important write operations that make its way to the currently active primary.
The speed-versus-accuracy trade is a dial tuned per feature, not a single global setting. High Availability covers how each choice gets made, and where.
Compute
Applications run on virtualized or bare-metal servers depending on load and availability. Untrusted workloads like user-defined functions are isolated with gVisor, and everything is subject to aggressive security scanning.
Security
Every endpoint connected to our platform is secured by our mesh using Nebula. Cross-cell traffic travels over authenticated VPN connections, and probe nodes at the edge are handed only narrowly scoped credentials for the work assigned, never the platform's primary keys.