Graceful shutdown holds up to SHUTDOWN_GRACE (20s) on idle HTTP/1.1 keep-alive connections #14

New issue

Open

opened 2026-08-03 11:00:47 +00:00 by james.oates · 0 comments

james.oates commented

2026-08-03 11:00:47 +00:00

Owner

Background

Follow-up from the Network Access Control feature (Task 8's accept-loop generalization, run_binding_loop in src/main.rs). Non-blocking — filed for later.

Problem

run_binding_loop tracks per-connection tasks in a tokio::task::JoinSet and, on shutdown, drains them under a tokio::time::timeout(SHUTDOWN_GRACE, …) where SHUTDOWN_GRACE = 20s. This awaits whole connection tasks, not individual in-flight requests. An idle HTTP/1.1 keep-alive connection that's open (but not actively serving a request) at shutdown time will hold the drain until either its own keep-alive timeout or the full 20s grace period elapses. Under real keep-alive traffic, shutdown can therefore routinely take up to 20s — a latency regression versus the pre-feature axum::serve(...).with_graceful_shutdown(...) path, which finished in-flight requests and then closed idle connections promptly.

Proposed fix

Use hyper's per-connection graceful shutdown rather than "await the whole task, bounded by a timeout". The serve_connection_with_upgrades future is a pollable UpgradeableConnection; pinning each connection handle and calling .graceful_shutdown() on the shutdown signal tells the connection to finish any in-flight request and then close, instead of parking on idle keep-alive. This restores the prior prompt-drain behavior within the "never cancel an in-flight request" constraint (it is not force-cancellation). Requires retaining each connection handle and racing it against the shutdown watch channel.

Severity

Operational/availability only — no security or correctness impact. The current behavior is bounded and safe, just slower to shut down than ideal.

## Background Follow-up from the Network Access Control feature (Task 8's accept-loop generalization, `run_binding_loop` in `src/main.rs`). Non-blocking — filed for later. ## Problem `run_binding_loop` tracks per-connection tasks in a `tokio::task::JoinSet` and, on shutdown, drains them under a `tokio::time::timeout(SHUTDOWN_GRACE, …)` where `SHUTDOWN_GRACE = 20s`. This awaits whole *connection* tasks, not individual in-flight *requests*. An idle HTTP/1.1 keep-alive connection that's open (but not actively serving a request) at shutdown time will hold the drain until either its own keep-alive timeout or the full 20s grace period elapses. Under real keep-alive traffic, shutdown can therefore routinely take up to 20s — a latency regression versus the pre-feature `axum::serve(...).with_graceful_shutdown(...)` path, which finished in-flight requests and then closed idle connections promptly. ## Proposed fix Use hyper's per-connection graceful shutdown rather than "await the whole task, bounded by a timeout". The `serve_connection_with_upgrades` future is a pollable `UpgradeableConnection`; pinning each connection handle and calling `.graceful_shutdown()` on the shutdown signal tells the connection to finish any in-flight request and then close, instead of parking on idle keep-alive. This restores the prior prompt-drain behavior **within** the "never cancel an in-flight request" constraint (it is not force-cancellation). Requires retaining each connection handle and racing it against the shutdown watch channel. ## Severity Operational/availability only — no security or correctness impact. The current behavior is bounded and safe, just slower to shut down than ideal.

james.oates referenced this issue

2026-08-03 13:11:17 +00:00

Composable per-listener access control (bindings, listeners, access rules) #3

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

IsoHex/edge-router#14

No description provided.

Rows
Columns