Benchmark musl vs musl+mimalloc vs glibc on an HP G6 host after go-live #46

New issue

Open

opened 2026-10-22 18:59:31 +00:00 by james.oates · 0 comments

james.oates commented

2026-10-22 18:59:31 +00:00

Owner

Why

v2.0.0 changes two things about allocation at once and neither is backed by a measurement taken on this software:

The release image moved from Debian glibc to Alpine static musl (45.9MB from 179MB).
mimalloc was set as the global allocator, because musl's mallocng takes a lock per size class and a tokio multi-thread runtime produces concurrent small allocations constantly.

Point 2 is mitigation against a documented property of mallocng, not against an observed regression. Nobody has run ER on plain musl under load. So the shipped configuration is a reasonable default chosen from first principles, and that is worth either confirming or correcting with numbers.

Raised by Axianthe during the review of the musl switch.

What to measure

Three builds, same commit, same host, same workload:

Build	Allocator serving Rust allocations
A	glibc (Debian image)
B	static musl, mallocng
C	static musl, mimalloc (what ships)

Note override is off in C, so bundled SQLite and ring use musl's malloc in both B and C. If the numbers are close, re-running C with override on is the next variable, not the first.

Conditions

A real HP G6 fleet host, not this Mac and not a VM on it — the point is contended allocation on the production core count, and OrbStack's arm64 is the wrong machine twice over.
After go-live, so the workload is real traffic shape rather than a synthetic guess.
Concurrency high enough to contend: the failure mode is lock contention, so a single-stream benchmark will show nothing whatever the result.

What would make this actionable

p50/p95/p99 proxied-request latency and RSS for each build.
If B ≈ C, mimalloc is cost without benefit here and should be dropped — one less C dependency in the build.
If C ≫ B, the current default is right and this issue records why.
If A ≫ C, the musl switch bought a smaller image at a throughput cost, and that trade needs stating explicitly rather than being inherited.

Not blocking

v2.0.0 ships as C. This is follow-up measurement, deliberately after go-live.

## Why v2.0.0 changes two things about allocation at once and neither is backed by a measurement taken on this software: 1. The release image moved from Debian glibc to Alpine static musl (45.9MB from 179MB). 2. mimalloc was set as the global allocator, because musl's mallocng takes a lock per size class and a tokio multi-thread runtime produces concurrent small allocations constantly. Point 2 is mitigation against a documented property of mallocng, **not** against an observed regression. Nobody has run ER on plain musl under load. So the shipped configuration is a reasonable default chosen from first principles, and that is worth either confirming or correcting with numbers. Raised by Axianthe during the review of the musl switch. ## What to measure Three builds, same commit, same host, same workload: | Build | Allocator serving Rust allocations | |---|---| | A | glibc (Debian image) | | B | static musl, mallocng | | C | static musl, mimalloc *(what ships)* | Note `override` is off in C, so bundled SQLite and ring use musl's malloc in both B and C. If the numbers are close, re-running C with `override` on is the next variable, not the first. ## Conditions - A real HP G6 fleet host, not this Mac and not a VM on it — the point is contended allocation on the production core count, and OrbStack's arm64 is the wrong machine twice over. - After go-live, so the workload is real traffic shape rather than a synthetic guess. - Concurrency high enough to contend: the failure mode is lock contention, so a single-stream benchmark will show nothing whatever the result. ## What would make this actionable - p50/p95/p99 proxied-request latency and RSS for each build. - If B ≈ C, mimalloc is cost without benefit here and should be dropped — one less C dependency in the build. - If C ≫ B, the current default is right and this issue records why. - If A ≫ C, the musl switch bought a smaller image at a throughput cost, and that trade needs stating explicitly rather than being inherited. ## Not blocking v2.0.0 ships as C. This is follow-up measurement, deliberately after go-live.

james.oates referenced this issue

2026-10-26 17:53:55 +00:00

The WebSocket path has no connect timeout, no idle timeout and no connection cap #62

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

IsoHex/edge-router#46

No description provided.

Rows
Columns