Graceful shutdown holds up to SHUTDOWN_GRACE (20s) on idle HTTP/1.1 keep-alive connections #14
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Background
Follow-up from the Network Access Control feature (Task 8's accept-loop generalization,
run_binding_loopinsrc/main.rs). Non-blocking — filed for later.Problem
run_binding_looptracks per-connection tasks in atokio::task::JoinSetand, on shutdown, drains them under atokio::time::timeout(SHUTDOWN_GRACE, …)whereSHUTDOWN_GRACE = 20s. This awaits whole connection tasks, not individual in-flight requests. An idle HTTP/1.1 keep-alive connection that's open (but not actively serving a request) at shutdown time will hold the drain until either its own keep-alive timeout or the full 20s grace period elapses. Under real keep-alive traffic, shutdown can therefore routinely take up to 20s — a latency regression versus the pre-featureaxum::serve(...).with_graceful_shutdown(...)path, which finished in-flight requests and then closed idle connections promptly.Proposed fix
Use hyper's per-connection graceful shutdown rather than "await the whole task, bounded by a timeout". The
serve_connection_with_upgradesfuture is a pollableUpgradeableConnection; pinning each connection handle and calling.graceful_shutdown()on the shutdown signal tells the connection to finish any in-flight request and then close, instead of parking on idle keep-alive. This restores the prior prompt-drain behavior within the "never cancel an in-flight request" constraint (it is not force-cancellation). Requires retaining each connection handle and racing it against the shutdown watch channel.Severity
Operational/availability only — no security or correctness impact. The current behavior is bounded and safe, just slower to shut down than ideal.
Fixed in
a1ca87fon main.Each connection now serves via a
serve_connection_gracefulhelper that races the hyper connection against a cloned shutdown watch and calls hyper's per-connectiongraceful_shutdown()on the signal — exactly the proposed fix.run_binding_loopstill drains theJoinSetunderSHUTDOWN_GRACE, but that timeout is now a backstop for a genuinely stuck request rather than the common case.Both halves of the contract are covered by regression tests:
Not addressed here (out of scope, unchanged from before): connections still in a pre-serve phase — PROXY header read, TLS handshake, L4 peek — aren't shutdown-aware and still drain under the grace ceiling, but each is separately bounded by its own timeout.