A failed ACME key decrypt silently downgrades an apex to the self-signed fallback cert #38

New issue

Closed

opened 2026-10-20 14:03:08 +00:00 by james.oates · 0 comments

james.oates commented

2026-10-20 14:03:08 +00:00

Owner

health.rs:517-523, in reload_routes_config:

if let (Some(cert_pem), Some(key_enc)) = (&stored.cert_pem, &stored.key_pem_enc) {
    if let Ok(key_pem) = crypto.decrypt(key_enc).await {
        if let Ok(ck) = crate::tls::parse_certified_key(cert_pem, &key_pem) {
            tls_resolver.insert(&tls_cfg.apex, ck);
        }
    }
}

Both failures are swallowed. When the decrypt fails, the apex is never inserted
into the resolver — and SniResolver::resolve (tls.rs:81-83) treats an absent
apex as "use the fallback", so the client is served the self-signed
edge-router.local certificate instead of getting a handshake failure.

The operator sees a browser certificate warning on a domain that has a valid
certificate sitting in the database, with nothing in the log to explain it.
Users are trained to click through exactly this warning.

Reachable whenever crypto.decrypt fails for a stored ACME key — most plausibly
ENCRYPTION_ENGINE=openbao with OpenBao sealed or unreachable, where it affects
every [[tls]] apex at once.

Two parts to the fix:

Log the failure at ERROR with the apex named. Silence is the worst part.
Make absence not mean fallback for a configured apex. A configured apex
that could not be loaded should abort the handshake, not substitute a
different certificate. The tombstone approach is described in
specs/2026-10-20-runtime-resilience-and-error-attribution-design.md
("Refusal, not fallback").

The genuine no-SNI case (netboot dialling by IP) keeps using the fallback slot
deliberately — that is unaffected.

`health.rs:517-523`, in `reload_routes_config`: ```rust if let (Some(cert_pem), Some(key_enc)) = (&stored.cert_pem, &stored.key_pem_enc) { if let Ok(key_pem) = crypto.decrypt(key_enc).await { if let Ok(ck) = crate::tls::parse_certified_key(cert_pem, &key_pem) { tls_resolver.insert(&tls_cfg.apex, ck); } } } ``` Both failures are swallowed. When the decrypt fails, the apex is never inserted into the resolver — and `SniResolver::resolve` (`tls.rs:81-83`) treats an absent apex as "use the fallback", so the client is served the self-signed `edge-router.local` certificate instead of getting a handshake failure. The operator sees a browser certificate warning on a domain that has a valid certificate sitting in the database, with nothing in the log to explain it. Users are trained to click through exactly this warning. Reachable whenever `crypto.decrypt` fails for a stored ACME key — most plausibly `ENCRYPTION_ENGINE=openbao` with OpenBao sealed or unreachable, where it affects every `[[tls]]` apex at once. Two parts to the fix: 1. Log the failure at ERROR with the apex named. Silence is the worst part. 2. Make absence not mean fallback for a *configured* apex. A configured apex that could not be loaded should abort the handshake, not substitute a different certificate. The tombstone approach is described in `specs/2026-10-20-runtime-resilience-and-error-attribution-design.md` ("Refusal, not fallback"). The genuine no-SNI case (netboot dialling by IP) keeps using the fallback slot deliberately — that is unaffected.

james.oates referenced this issue

2026-10-20 15:03:23 +00:00

Runtime resilience: survive OpenBao being unavailable #40

james.oates referenced this issue

2026-10-22 16:26:09 +00:00

Runtime resilience: survive OpenBao being unavailable #40

james.oates referenced this issue

2026-10-22 16:27:26 +00:00

WebSocket upgrade to an https upstream returns 501 #45

james.oates referenced this issue

2026-10-23 13:16:18 +00:00

No way to serve a name no public CA will issue for #50