Runtime resilience: survive OpenBao being unavailable #40

New issue

Closed

opened 2026-10-20 15:03:23 +00:00 by james.oates · 2 comments

james.oates commented

2026-10-20 15:03:23 +00:00

Owner

Execution tracking for the plan at
obsidian/Personal/Projects/edge-router/plans/2026-10-20-runtime-resilience.md
(design: specs/2026-10-20-runtime-resilience-and-error-attribution-design.md, Part 1).

ER currently aborts startup when OpenBao is sealed or unreachable, so on a host
where it owns :443 every unrelated route dies with it and stays dead until a
human notices and restarts it. The rule being implemented: a failed dependency
fetch degrades; a failed local precondition exits.

Phase A — tombstone + #38

A1 CertEntry enum in SniResolver; resolve returns None for a tombstone instead of substituting the fallback cert
A2 a failed stored-cert decrypt tombstones and logs instead of silently downgrading (closes #38)

Phase B — degraded startup and retry

B1 managed_cert::load_all reports per-cert failures instead of aborting
B2 an unreachable OpenBao no longer aborts startup (also fixes the stale BAO_BROKER_SOCKET message, #34)
B3 retry with backoff so an unseal recovers in seconds

Phase C — config reload

C1 a TLS encrypt failure scopes to its domain instead of freezing all reconciliation (closes #39)

Phase D — signalling (do not ship A-C without this)

D1 GET /health liveness, admin host, no auth
D2 GET /ready per-apex availability detail, admin host, authenticated
D3 sd_notify — READY=1 once sockets are bound even when degraded, STATUS= carries the degraded text
D4 docs (CLAUDE.md, README.md, example config)

Acceptance

With an apex tombstoned, openssl s_client -servername <apex> fails the handshake and does not return an edge-router.local certificate — verified against a running binary
ER starts against an unreachable OpenBao, serves unrelated routes, and the apex starts serving within ~60s of OpenBao returning, with no restart
Full suite green, clippy clean, cargo fmt --check, CI green

Part 2 of the spec (error attribution, #37) is a separate plan and issue.

Execution tracking for the plan at `obsidian/Personal/Projects/edge-router/plans/2026-10-20-runtime-resilience.md` (design: `specs/2026-10-20-runtime-resilience-and-error-attribution-design.md`, Part 1). ER currently aborts startup when OpenBao is sealed or unreachable, so on a host where it owns :443 every unrelated route dies with it and stays dead until a human notices and restarts it. The rule being implemented: **a failed dependency *fetch* degrades; a failed *local precondition* exits.** ### Phase A — tombstone + #38 - [ ] A1 `CertEntry` enum in `SniResolver`; `resolve` returns `None` for a tombstone instead of substituting the fallback cert - [ ] A2 a failed stored-cert decrypt tombstones and logs instead of silently downgrading (closes #38) ### Phase B — degraded startup and retry - [ ] B1 `managed_cert::load_all` reports per-cert failures instead of aborting - [ ] B2 an unreachable OpenBao no longer aborts startup (also fixes the stale `BAO_BROKER_SOCKET` message, #34) - [ ] B3 retry with backoff so an unseal recovers in seconds ### Phase C — config reload - [ ] C1 a TLS encrypt failure scopes to its domain instead of freezing all reconciliation (closes #39) ### Phase D — signalling (do not ship A-C without this) - [ ] D1 `GET /health` liveness, admin host, no auth - [ ] D2 `GET /ready` per-apex availability detail, admin host, authenticated - [ ] D3 `sd_notify` — `READY=1` once sockets are bound even when degraded, `STATUS=` carries the degraded text - [ ] D4 docs (`CLAUDE.md`, `README.md`, example config) ### Acceptance - [ ] With an apex tombstoned, `openssl s_client -servername <apex>` **fails the handshake** and does not return an `edge-router.local` certificate — verified against a running binary - [ ] ER starts against an unreachable OpenBao, serves unrelated routes, and the apex starts serving within ~60s of OpenBao returning, with no restart - [ ] Full suite green, clippy clean, `cargo fmt --check`, CI green Part 2 of the spec (error attribution, #37) is a separate plan and issue.

james.oates commented

2026-10-22 16:26:09 +00:00

Author

Owner

Superseded in large part. The TPM, OpenBao and at-rest encryption removal (spec 2026-10-22-remove-tpm-openbao-encryption-design.md, branch feat/per-name-acme-httpreq) deletes the dependency this issue was about, so most of its phases no longer have a reason to exist.

Done, on that branch:

Phase A — the resolver now holds a tombstone and refuses the handshake for a name with no usable certificate, instead of falling through to the parent or the self-signed fallback. Closes #38.
Phase D — /health (admin-host gated, unauthenticated, because a probe carries no key) and /ready (authenticated, per-name JSON with expiry and last error). Pulled forward deliberately: the tombstone without readiness reporting is a regression, since ER would report healthy while refusing handshakes.

No longer applicable:

Phase B (degrade at startup instead of exiting) and Phase C (per-domain reload under a sealed Bao) both existed because OpenBao was a startup dependency. ER now has no startup dependency that can be unavailable.
The sealed-Bao boot coupling itself — the whole premise of this issue.
#39 goes with it: the config reload no longer encrypts anything, so a single failure can no longer abort the reload and freeze L4 routes, static sites and admin entries along with it.

What the resilience thinking turned into instead: a missing httpreq endpoint is not a boot failure but a certificate that quietly fails to renew on a 60-day timescale. So renewal happens at 30 days remaining with jittered capped backoff, a valid certificate keeps serving while its renewal fails, and /ready reports degraded only inside 14 days — failing readiness a month early would be the worse outcome.

Leaving this open until the branch merges, then it closes.

Superseded in large part. The TPM, OpenBao and at-rest encryption removal (spec `2026-10-22-remove-tpm-openbao-encryption-design.md`, branch `feat/per-name-acme-httpreq`) deletes the dependency this issue was about, so most of its phases no longer have a reason to exist. **Done, on that branch:** - **Phase A** — the resolver now holds a tombstone and refuses the handshake for a name with no usable certificate, instead of falling through to the parent or the self-signed fallback. Closes #38. - **Phase D** — `/health` (admin-host gated, unauthenticated, because a probe carries no key) and `/ready` (authenticated, per-name JSON with expiry and last error). Pulled forward deliberately: the tombstone without readiness reporting is a regression, since ER would report healthy while refusing handshakes. **No longer applicable:** - **Phase B** (degrade at startup instead of exiting) and **Phase C** (per-domain reload under a sealed Bao) both existed because OpenBao was a startup dependency. ER now has no startup dependency that can be unavailable. - The sealed-Bao boot coupling itself — the whole premise of this issue. - #39 goes with it: the config reload no longer encrypts anything, so a single failure can no longer abort the reload and freeze L4 routes, static sites and admin entries along with it. **What the resilience thinking turned into instead:** a missing httpreq endpoint is not a boot failure but a certificate that quietly fails to renew on a 60-day timescale. So renewal happens at 30 days remaining with jittered capped backoff, a valid certificate keeps serving while its renewal fails, and `/ready` reports degraded only inside 14 days — failing readiness a month early would be the worse outcome. Leaving this open until the branch merges, then it closes.

james.oates commented

2026-10-26 13:36:23 +00:00

Author

Owner

Overtaken by v2.0.0 — closing, with the one surviving item already tracked elsewhere.

The premise is gone: ER has no OpenBao dependency at startup at all. [[managed_cert]] and the encrypted certificate store were removed, so Phase B (B1–B3, "an unreachable OpenBao no longer aborts startup") has nothing left to degrade. A sealed Bao can no longer affect ER's boot.

Shipped from the rest of the plan:

A1/A2 — CertEntry with tombstones, resolve returning None rather than substituting the fallback. Closed #38. Extended since: a tombstone no longer shadows a covering wildcard (#52).
D1/D2 — GET /health (liveness, unauthenticated) and GET /ready (per-name certificate state, admin, authenticated).
D4 — docs updated across CLAUDE.md, README.md and the example config.
Acceptance 1 — proved against a running binary, and now pinned by an end-to-end rustls test that asserts both directions.

Not done, and carried:

C1 — still open as #39, which is the live issue for it (the failure mode is now a failing config source rather than a TLS encrypt failure, but it is the same all-or-nothing ?).
D3 sd_notify — never implemented, and no longer part of a resilience story since there is no degraded-startup state to signal. Worth its own issue if ER is to run natively under systemd on Kyoshi; not filing one speculatively.

Closing rather than rewriting: the issue tracks a plan whose central phase no longer exists.

Overtaken by v2.0.0 — closing, with the one surviving item already tracked elsewhere. The premise is gone: ER has no OpenBao dependency at startup at all. `[[managed_cert]]` and the encrypted certificate store were removed, so Phase B (B1–B3, "an unreachable OpenBao no longer aborts startup") has nothing left to degrade. A sealed Bao can no longer affect ER's boot. Shipped from the rest of the plan: - **A1/A2** — `CertEntry` with tombstones, `resolve` returning `None` rather than substituting the fallback. Closed #38. Extended since: a tombstone no longer shadows a covering wildcard (#52). - **D1/D2** — `GET /health` (liveness, unauthenticated) and `GET /ready` (per-name certificate state, admin, authenticated). - **D4** — docs updated across `CLAUDE.md`, `README.md` and the example config. - **Acceptance 1** — proved against a running binary, and now pinned by an end-to-end rustls test that asserts both directions. Not done, and carried: - **C1** — still open as **#39**, which is the live issue for it (the failure mode is now a failing config source rather than a TLS encrypt failure, but it is the same all-or-nothing `?`). - **D3** `sd_notify` — never implemented, and no longer part of a resilience story since there is no degraded-startup state to signal. Worth its own issue if ER is to run natively under systemd on Kyoshi; not filing one speculatively. Closing rather than rewriting: the issue tracks a plan whose central phase no longer exists.

james.oates closed this issue

2026-10-26 13:36:33 +00:00

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

IsoHex/edge-router#40

No description provided.

Rows
Columns