Run control

Admin only

/v1/runcontrol is the platform control plane. Every call requires a platform admin credential (see Authentication). It is not part of the public API and is documented here as an operations runbook.

Run control is how an operator changes the running platform without a deploy: kill switches, feature flags, and bounded overrides of things the platform derives. It is a typed allowlist. Only controls that are registered in the code can be set, so the control plane can never reach anything it was not deliberately given (it cannot, for example, disable authentication or billing).

Model

Every control is addressed by three parts:

<entity> / <target> / <control>

• entity is the kind of thing: platform, feature, route, or service.
• target is the specific instance (a service slug, a route prefix, a feature name). Omit it to target every instance of the entity.
• control is the knob: enabled, status.override, notice, and so on.

Each control declares a typed value, whether it must expire (requireTtl), whether it is shown in public API output (visible), and a fail-safe fallback used when the store is unreachable.

Resolution is most-specific-wins: a control set on a specific target beats one set globally, which beats the registered default. Expiry is evaluated on read, so an expired control stops applying immediately whether or not a cleanup job has run.

Where controls take effect

The write plane (this endpoint) lives in the API, but resolution is shared: both the API and the worker read the same control set (cached behind NATS KV), so a control can be enforced wherever it belongs.

• API edge enforces route/enabled (route kill switch), platform/api.rateLimit (global rate-limit override), and the feature/service controls used by the request handlers.
• Worker / scheduler enforces platform/checks.enabled (pause all probing) and service/ingest.enabled (pause one feed) — controls it could not see when run control lived only in the API.

Enforcement is not instantaneous. Writes invalidate the cache immediately, and readers otherwise refresh on a short interval (a few seconds on the API edge, the scheduler tick on the worker), so a change takes effect within seconds, not on the next request.

Operations

All operations are POST and take a JSON body. Send your admin token in the Authorization header.

List what is controllable

curl https://ws.latency.app/v1/runcontrol/registry \
  -H "Authorization: Bearer $LATENCY_ADMIN_TOKEN"

Returns every registered control with its shape, so tools never hardcode a list that drifts.

List what is currently in force

curl -X POST https://ws.latency.app/v1/runcontrol \
  -H "Authorization: Bearer $LATENCY_ADMIN_TOKEN" \
  -H "content-type: application/json" \
  -d '{ "entity": "service" }'

Set a control

curl -X POST https://ws.latency.app/v1/runcontrol/set \
  -H "Authorization: Bearer $LATENCY_ADMIN_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "entity": "service",
    "target": "cloudflare",
    "control": "notice",
    "value": { "message": "We are re-checking Cloudflare status now.", "level": "info" },
    "reason": "status feed flaked; investigating",
    "ttlMinutes": 120
  }'

reason is always required. ttlMinutes is required for any control the registry marks requireTtl (overrides and notices), so an assertion cannot outlive the situation that prompted it.

Clear a control

curl -X POST https://ws.latency.app/v1/runcontrol/clear \
  -H "Authorization: Bearer $LATENCY_ADMIN_TOKEN" \
  -H "content-type: application/json" \
  -d '{ "entity": "service", "target": "cloudflare", "control": "notice" }'

Runbooks

Post a notice on a host's inspect page

When a provider's status feed is misbehaving and you want visitors to know you are on it:

# Show a message on cloudflare's inspect page for two hours.
curl -X POST .../v1/runcontrol/set -d '{
  "entity": "service", "target": "cloudflare", "control": "notice",
  "value": { "message": "We are working on fixing the connection to this host'\''s status page.", "level": "warning" },
  "reason": "cloudflare feed 500ing", "ttlMinutes": 120
}'

The notice appears on the inspect page (and in /v1/inspect under notices), always labeled as operator-authored, and disappears on its own after the TTL.

Correct a misleading status

A provider's page reads red off component noise with no real incident. Assert the truth for a bounded window (this is disclosed on the page as operator-set, never shown as observed):

curl -X POST .../v1/runcontrol/set -d '{
  "entity": "service", "target": "cloudflare", "control": "status.override",
  "value": { "state": "operational" },
  "reason": "indicator red off PoP noise, no incident", "ttlMinutes": 180
}'

Pause a noisy feed

Stop polling one service without touching its monitors. The scheduler skips dispatching that service's ingest on its next tick; the monitor stays claimed, so it resumes on schedule when you re-enable it:

curl -X POST .../v1/runcontrol/set -d '{
  "entity": "service", "target": "paypal", "control": "ingest.enabled",
  "value": { "enabled": false }, "reason": "feed rate-limiting us"
}'

Turn a feature off for everyone (or one plan)

# Kill a feature platform-wide.
curl -X POST .../v1/runcontrol/set -d '{
  "entity": "feature", "target": "anonymous-reports", "control": "enabled",
  "value": { "enabled": false }, "reason": "abuse spike, pausing"
}'

# Or disable it only for a plan, leaving everyone else on.
curl -X POST .../v1/runcontrol/set -d '{
  "entity": "feature", "target": "some-feature", "control": "enabled",
  "value": { "enabled": false, "plans": ["FREE"] }, "reason": "gating to paid"
}'

Shed load on a route

Disable an endpoint prefix at the API edge. Any request whose path is, or starts with, the target prefix gets a 503 route_disabled before it touches a handler — useful for pulling a misbehaving or expensive endpoint without a deploy:

curl -X POST .../v1/runcontrol/set -d '{
  "entity": "route", "target": "/v1/dns", "control": "enabled",
  "value": { "enabled": false }, "reason": "resolver upstream degraded"
}'

Pause all probing (maintenance)

Stop the scheduler from dispatching any checks — for a maintenance window, or to relieve a struggling worker/ClickHouse tier — without disabling a single monitor. Monitors resume on their normal schedule when you re-enable:

curl -X POST .../v1/runcontrol/set -d '{
  "entity": "platform", "control": "checks.enabled",
  "value": { "enabled": false }, "reason": "clickhouse maintenance"
}'

This is a protective capability, so it fails open: if the control store is unreachable, checks keep running. Clear it (or let no TTL expire — it has none) to resume:

curl -X POST .../v1/runcontrol/clear -d '{
  "entity": "platform", "control": "checks.enabled"
}'

Throttle the API during a slam

Override the global per-client rate limit live. max is requests per window per client IP; set it below the code default to tighten under abuse, or to 0 to fall back to the built-in default:

# Clamp to 120 req/min/client while we ride out a spike.
curl -X POST .../v1/runcontrol/set -d '{
  "entity": "platform", "control": "api.rateLimit",
  "value": { "max": 120 }, "reason": "scraper flood on /v1/inspect"
}'

The API edge reads this from an in-process snapshot refreshed every few seconds, so the change lands within seconds without adding a lookup to the hot path.

Put the platform in read-only mode

curl -X POST .../v1/runcontrol/set -d '{
  "entity": "platform", "control": "writes.enabled",
  "value": { "enabled": false }, "reason": "maintenance window"
}'

Enforcement is per-write-path

writes.enabled is a declared posture with a fail-open default. It is enforced at write call sites, not by a blanket "block every POST" middleware — some read operations are POSTs (a Resource can take its arguments in a JSON body), so a method-based switch would wrongly reject reads. Coverage is being extended one write path at a time; treat this as maintenance signalling, not yet a hard global lock.

Safety properties

• Allowlist. An unregistered (entity, control) pair is a 400. The control plane can only touch what it was deliberately given.
• Fail-safe per control. If the control store is unreachable, protective capabilities (monitoring, alerting) default on; assertions default off. This direction is chosen per control, not globally.
• Everything is audited. Every set and clear is written to the audit log with the actor, reason, value, and expiry.
• Overrides announce themselves. Anything that changes what a user sees (status.override, notice) is always disclosed in the API output, never presented as observed truth.