Restarts and upgrades¶
How each component behaves when it is stopped or replaced. Read the release notes of the version you move to before upgrading, and do it in an environment that looks like production first; the end-to-end stand (testing) is how we check a release.
Relays¶
On SIGTERM a relay answers 503 on /healthz at once, closes its ports after drain_delay (5 s), asks agents to open no new streams, and lets open streams finish until drain_timeout (1 min). Agents reconnect by themselves. With two or more relays behind a load balancer or a Kubernetes Service, replace them one at a time. Set terminationGracePeriodSeconds above drain_delay + drain_timeout. Connection pools keep connections open, so expect the organizations' connections to be re-established after a restart: the tunnel in production.
Agents¶
An agent keeps its certificate in its -state directory and needs no new token after a restart. Run two agents per organization (on two hosts) to replace them without a gap.
The proxy¶
The proxy reads its routing metadata from the cloud database and reloads it after a migration and at least every 30 s. Its TLS certificate files are re-read without a restart. It drains on stop (/healthz answers 503), so take it out of the load balancer first.
The database side¶
- After an upgrade,
lokadb check(andschema-diff/refs-difffor moved organizations) shows whether anything drifted: migrations. Ctrl-CandSIGTERMmakemove-orgclean up after itself. If it is killed hard, its capture triggers stop writing by themselves after 10 minutes without its heartbeat, andlokadb move-cleanupremoves the leftovers.- Moving back and disabling are the way out of a version you do not want: move back.