Deployment
Durable Runner deploys as one durable-server process plus its journal on local disk. No
database, no broker, no cluster — the journal’s one-writer rule is the architecture. Two processes
can never share a journal directory: the second fails at startup on its lock, so “high
availability” means a fast, monitored restart of the same machine, not a second live node.
Filesystem
Section titled “Filesystem”- Local disk only. The journal fsyncs every commit; a network filesystem (NFS, EBS-through-NFS, FUSE mounts) breaks both its latency and its lock guarantees.
- Space: provision journal + snapshots +
--reserve-bytes(64 MiB default) + headroom for--backup-rootif it’s co-located. Dedup records live at most seven days past a run’s end; compaction (--compact-every-secsorPOST /admin/compact) reclaims them. - Permissions: the journal holds submitted inputs verbatim —
0700on the journal directory, owned by the service user.
systemd
Section titled “systemd”[Unit]Description=Durable RunnerAfter=network.target
[Service]User=durableEnvironmentFile=/etc/durable/envExecStart=/usr/local/bin/durable-server --journal /var/lib/durable/journal \ --listen 127.0.0.1:58300 --handler step=http://127.0.0.1:58320/step \ --backup-root /var/lib/durable/backups --compact-every-secs 3600Restart=alwaysRestartSec=1LimitNOFILE=8192
[Install]WantedBy=multi-user.target/etc/durable/env carries DURABLE_API_KEY and DURABLE_READ_KEY (0600, service user).
SIGTERM drains: in-flight requests finish, the engine makes its decided work durable, and it
exits inside --drain-timeout-secs (30 s default) — so systemctl restart is a clean cut, not a
crash drill. Restart=always plus the journal means a crash resumes in place; on boot, replay
restores exactly what was acknowledged.
In front of a proxy
Section titled “In front of a proxy”The server speaks plain HTTP — put a TLS-terminating proxy in front when it serves beyond the host. Two rules:
- The proxy must not flatten the auth surface. Forward
Authorizationand the real client; Durable Runner’s own checks are the boundary, not the proxy’s allowlist. - Never proxy to a server started with
--open-reads. That flag’s safety is “the request really is local”; a proxy makes remote reads look local. With a proxy, reads needDURABLE_READ_KEYlike everything else.
runner.example.com { reverse_proxy 127.0.0.1:58300}Keep --listen 127.0.0.1:58300 — the proxy is the only thing that should reach the port.
Upgrades and restarts
Section titled “Upgrades and restarts”A stop-start on the same journal is the upgrade path: stop (drains clean), swap the binary, start. The journal is forward-readable by newer servers — the changelog calls out any release whose journal format needs attention. Restores from backup are the rollback path when a journal can’t move forward (backups).
What a restart costs, plainly: an upgrade is a restart plus a replay, and unavailability grows
with how many runs are queued. Measured on a fast laptop (release build, spawn → /ready = 200):
| queued runs | journal size | restart-to-ready |
|---|---|---|
| 100k | 161 MiB | 400 ms |
| 300k | 467 MiB | 599 ms |
| 800k | 1.2 GiB | 1,716 ms |
These are development observations — measure your own host the same way before promising a number
(see capacity). While the server is down its socket doesn’t answer at
all; Caddy’s lb_try_duration holds incoming requests through the gap and retries, so clients see
added latency, never an error — and nothing reaches the old process, so even a submit is safe.
runner.example.com { reverse_proxy 127.0.0.1:58300 { lb_try_duration 15s lb_try_interval 250ms }}The deploy kit ships with the product. deploy/ in the repository carries the hardened
systemd unit, the Caddyfile above, cloudflare-ips.sh (feeds Cloudflare’s ranges into both
--trusted-proxy and Caddy’s trusted proxies), install.sh, upgrade.sh, rollback.sh, a
format-guard.sh that refuses to swap in a binary that can’t read the journal, smoke.sh as the
post-install gate, and the daily-backup timer. upgrade.sh and rollback.sh compare
--journal-format — the highest record version a binary reads — against the journal’s before
touching it, so a downgrade that couldn’t parse the journal fails closed instead of eating state.