feat(build): lower the Linux glibc floor to 2.31 and publish linux-arm64

[MichaelTaylor3d]

What changed

The platform floor (#1736 / #1735 duplicates, #1741) — the MVP half

What raised the floor: the builder image, not the code. build-binaries.yml built linux-x64 on ubuntu-latest, which GitHub repointed at Ubuntu 24.04 (glibc 2.39). A glibc-linked binary runs on its BUILD glibc and anything newer, never anything older — so v0.64.0 began requiring glibc >= 2.38 and stopped starting on Ubuntu 22.04 LTS (2.35), Debian 12 (2.36) and Amazon Linux 2023 (2.34), with no code change and no signal.

The resulting floor is glibc 2.31, and the supported target matrix is now:

Asset	Target	Built on	Floor
linux-x64	x86_64-unknown-linux-gnu	ubuntu-latest + container: debian:11	glibc 2.31
linux-arm64 (new)	aarch64-unknown-linux-gnu	ubuntu-24.04-arm + container: debian:11 (NATIVE)	glibc 2.31
.deb amd64	x86_64-unknown-linux-gnu	same container	glibc 2.31
.deb arm64 (new)	aarch64-unknown-linux-gnu	same container (NATIVE)	glibc 2.31
windows-x64.exe	x86_64-pc-windows-msvc	windows-latest	n/a (MSVC CRT)
macos-arm64 / macos-x64	aarch64/x86_64-apple-darwin	macos-14	n/a (SDK)

2.31 clears Ubuntu 20.04+, Debian 11+, Amazon Linux 2023 and RHEL 9. The -gnu target was kept over musl deliberately: musl would remove the floor entirely but replaces glibc's NSS/resolver, which the node's DNS seeding relies on — a bigger behavioural risk than the problem being fixed. arm64 builds NATIVELY on the free public-repo arm64 runner, which removes the old SPEC 11.3 objection wholesale (that objection was specifically about CROSS-compiling the vendored openssl-sys in the Chia-wallet-SDK graph).

The floor is gated, not claimed. Declared once in .github/actions/setup-linux-build/action.yml, then enforced three ways:

1. the action asserts the container's own glibc equals the declared floor, so the image and the number cannot drift apart;
2. scripts/check-glibc-floor.sh asserts every produced binary — dig-node AND the dign alias, which is a separate link — is at or below the floor, for both the release binaries and the .deb payload (apt is a separate delivery path and could otherwise regress alone);
3. verify-linux-floor EXECUTES both binaries in ubuntu:22.04, debian:12 and amazonlinux:2023 on both architectures.

The gate script is pinned from BOTH sides (at-bound passes, one-minor-over fails) and the release job additionally re-runs it against an impossible 2.0 floor — a gate that cannot fail is a rubber stamp. Its comparison is version-aware, not lexical: glibc 2.4 is older than 2.31, and a lexical sort gets that backwards.

The ldd output is logged unconditionally before each run because the dynamic loader names only the FIRST unsatisfied library, so a partial fix reads as a fresh regression.

PR#118 gate follow-ups

• #1782 — download.rs: PeerAdded is republished on a supersede and fires no PeerRemoved, so the per-peer address list grew one entry per distinct SocketAddr ever seen (measured 5000 -> 5000). Capped at 8 = dig-dht's MAX_ADDRESSES_PER_RECORD. Eviction is family-aware, not a blind truncate: the tail is where the other address family ends up, and the downstream dial ladder is both IPv6-first and very short, so a lone IPv4 fallback would be churned away (5.2 prefers IPv6, it does not exclude IPv4).
• #1783 — pex.rs: pool membership is re-checked after the dial resolves, at a single named gate (should_adopt_dialed_peer). Since dig-gossip 0.17.12 a second connection SUPERSEDES and tears down the live session; pre-0.17.12 it returned DuplicateConnection, which is why ignoring the race used to be safe.
• #1784 — a wildcard/port-0 address (dig-nat's remote for an accepted relayed circuit with no relay endpoint) is rejected at two independent entry points, because they do not share a path: the pool/selector funnel in download.rs::on_pool_event, and the DHT routing feed in peer.rs, which bypasses the pool entirely. Root cause is in dig-nat and is reported, not patched here.
• #1785 — all four: SPEC 7.8 now states the PeerAdded keyed-idempotency contract; the RANGE_WINDOW #1640 "KNOWN CONSTRAINT" is marked RESOLVED (dig-nat 0.13 sender + 0.14 receiver, both fail-closed); dig-node-core moves to 0.24.0 with the reason recorded in the manifest; the "newest leads the dial order" comment is corrected to hold only within an address family.

Blast radius checked

gitnexus is disabled in the loop (it hangs in worktrees), so this was done with ripgrep + direct reads per CLAUDE.md 2.0.

• on_pool_event — callers: peer.rs::spawn_selector_registry_feed (snapshot + churn), lib.rs:7156, peer.rs:1790/1805/2893/3116/3136. All feed PeerAdded/PeerRemoved; behaviour changes only for a wildcard address (dropped) or past 8 addresses (capped).
• dht_addr_from_gossip_addr — retained unchanged; the new dht_contact_from_pool_addr wraps it, and the two DHT-feed call sites moved to the wrapper. No other caller altered.
• is_usable_contact — new symbol, no prior callers.
• build-binaries.yml — consumed by release.yml and nightly-release.yml, both of which download ALL artifacts by pattern, so linux-arm64 flows through with no caller change. package.yml's deb job became a matrix; its artifact name gained an -${arch} suffix (two legs would otherwise collide) and publish flattens by extension, so the rename is transparent.
• Staging moved into a composite action; release_workflow_dign_guard.rs pinned the old literal and CAUGHT it — updated and strengthened to assert both build jobs build the alias and neither names an asset itself.
• Test-helper visibility widened: peer.rs's test module and three of its helpers became pub(crate) (matching download.rs's existing precedent) so the PEX test can drive a REAL pool. Test-only; no shipped surface.

How verified

• Execution on every floor target — release canary run 30420036070: all 6 verify jobs green (linux-x64 and linux-arm64 x ubuntu:22.04, debian:12, amazonlinux:2023), each running --version for both binaries. The arm64/amazonlinux:2023 log resolves the complete dependency set (libc.so.6 => /lib64/libc.so.6, no not found) — Amazon Linux 2023 is the distro that killed the original e2e.
• Container + gate — run 30418600804: container glibc: 2.31, declared floor: 2.31 on both arches; requires glibc up to 2.30 (declared floor 2.31) ... OK; dpkg-deb --info shows Architecture: arm64 / amd64.
• Mutation-verified, executed not reasoned — each guard's test was watched to FAIL with only that guard reverted (committed first, reverted via file copy):
  • remove the cap -> both #1782 tests fail;
  • blind truncate instead of family-aware -> the IPv4-survival test fails while the cap test still passes, so the two tests are independently discriminating;
  • remove the wildcard guard -> both #1784 pool tests fail;
  • should_adopt_dialed_peer -> true -> the #1783 test fails;
  • validate the RAW instead of the MAPPED address -> the DHT-contact test fails while its control passes;
  • the floor script: lexical sort -> the 2.4-vs-2.31 case fails; swallowing GLIBC_PRIVATE -> the private-tag case fails.
• cargo clippy --all-targets --locked -- -D warnings clean; cargo fmt --all --check clean; cargo test --workspace green.

Version

0.66.0 -> 0.67.0 (minor). New published build targets (linux-arm64 binary + arm64 .deb) are added capability, and every code change is compatible. dig-node-core 0.23.0 -> 0.24.0 separately, per #1785c.

Not done here, deliberately

• install.sh's architecture detection (#1741, second deliverable) lives in dig-installer (install.sh:34 hardcodes os_slug="linux-x64" for every Linux, while the macOS branch does check uname -m), which another lane owns. Reported rather than edited.
• dig-nat's unspecified_addr() root cause (#1784) is in dig-nat. Guarded here; reported there.
• dig-download's dial_candidates truncation order (#1782 secondary) is upstream: it sorts IPv6-first then truncates to 4, so >=4 IPv6 candidates can still cut an IPv4 address out of the ladder. dig-node now guarantees the address survives in the POOL; guaranteeing it survives the LADDER is dig-download's to fix.
• #1763 (cold-start serving with no P2P engine) is left: it turns on a real design fork — whether the cold window should fail loudly or fall to the gateway — which belongs to a decider, not this batch.
• The #1783 guard's USE is code-review-verified, not test-verified. The verdict function is tested against a real pool holding a real connected peer, but exercising the guard through a genuinely-superseding successful NAT dial would require seaming out connect_via_nat/adopt_nat_connection on a live P2P path — a larger refactor than a LOW/analytic finding warrants. Stated rather than papered over.

Closes #1736
Closes #1735
Closes #1741
Closes #1782
Closes #1783
Closes #1784
Closes #1785

[MichaelTaylor3d]

VERDICT: CHANGES-REQUIRED (recorded as a comment review - GitHub rejects request-changes on a self-authored PR, HTTP 422).

CHANGES-REQUIRED - 2 gating findings, both small

Strong PR. The floor work is the most thoroughly gated CI change I have reviewed here: the container-glibc assert, the artifact gate, and the execution matrix are three INDEPENDENT gates, and the 2.0 negative control is genuine. I reproduced the mutation claims independently rather than taking them on report - all five discriminate, and none of the four peer-path fixes opens the remotely-inducible-lasting-verdict class.

Two things gate:

1. GATING - there IS a third unguarded path into the DHT. The PR asserts two independent entry points and no other path publishing an address into the pool or the DHT. bring_up_dht (crates/dig-node-core/src/peer.rs:2575) reads pool.connected_pool_peers() - the SAME source as the routing feed you did guard - maps it with a bare dht_addr_from_gossip_addr, and feeds the result into service.bootstrap() via bootstrap_peers_from_pool (seams/dig_peer/dht.rs:456), with no is_usable_contact. Detail on the inline thread.

2. GATING (one sentence) - the musl rejection is not recorded normatively. I judge the rationale SOUND, but it lives only in this PR body, and scripts/check-glibc-floor.sh:41-45 returns OK for a musl/static binary - so a future switch to a *-musl target passes all three gates while removing the property the DNS-seeding argument protects. Detail on the inline thread.

Mutations I executed (mine, not the lane reported ones)

Worktree at 3f59bad; baseline cargo test -p dig-node-core --lib = 461 passed. Reverted by file copy; tree verified clean afterwards.

Mutation	Result
retain_newest_per_family -> addrs.truncate(8)	ipv6_churn_cannot_evict_a_peers_only_ipv4_address FAILS, a_peer_that_supersedes_forever_never_exceeds_the_address_cap PASSES - the two tests discriminate independently, as claimed
remove the cap call + the wildcard guard	all 4 of the #1782/#1784 pool tests FAIL
is_usable_contact(&dht) -> is_usable_contact(&gossip) (raw not mapped)	a_wildcard_or_unshiftable_pool_address_yields_no_dht_contact FAILS on 203.0.113.7:1, its control PASSES
should_adopt_dialed_peer -> true	a_candidate_already_in_the_pool_is_not_adopted_after_its_dial_resolves FAILS
floor gate: sort -V -> sort	the 2.4-vs-2.31 case FAILS
floor gate: widen the grep to swallow GLIBC_PRIVATE	the private-tag case FAILS
floor gate: usage error exit 2 -> exit 0	the misuse case FAILS

Direct verdicts

Are the floor gates real negative controls? YES. The 2.0 control in build-binaries.yml is genuine, and it does more than advertised: because check-glibc-floor.sh:41-45 exits 0 on empty readelf output, a missing or broken readelf would make the gate silently green - and the 2.0 invocation catches exactly that, since it would also return 0. That is the one fail-open branch in the script, and the control covers it. All three script mutations above confirm the unit suite is not vacuous.

Does the #1783 test gap gate? NO. The verdict function is tested against a REAL two-node loopback-mTLS pool with a real connected peer plus an honest unconnected control, and my mutation fails it. The uncovered part is three lines of straight-line code between Ok(conn) and adopt_nat_connection, and - the thing I actually checked - the new return path takes NO penalize/disconnect/backoff action, so a mis-fire costs one redundant connection and nothing durable. Seaming out connect_via_nat for this is disproportionate to a LOW/analytic finding. The honest disclosure is the right treatment.

Was closing #114 correct? YES - and main is a strict SUPERSET, not merely equivalent. Verified item by item: the cascade pins (dig-nat 0.14, dig-dht 0.8, dig-download 0.12, dig-peer 0.7, dig-peer-selector 0.7, plus the dig-gossip rev) and the non_exhaustive DownloadConfig mutation are on main at download.rs:655. The #114 SPEC bracketing paragraph is superseded by the broader Authority-construction note now on main, which additionally governs /health, status and the control-client URL. The #114 stringly browser_authority is superseded by a typed UrlAuthority enum (open.rs:477) with its own regression test at open.rs:645. Config::bind_addr on main returns SocketAddr where #114 returned String, and Config::host is IpAddr, making the defect unrepresentable. KNOWN_VIOLATIONS is empty on main with the cross-crate reach detector present (banned_address_patterns.rs:314) rather than the #114 SIBLING_CRATE_UNDER_SCAN constant - same job, same guarantee. No work was lost.

Light security - none of the four opens the lasting-verdict class. The cap and the eviction are keyed per peer_id, so churn can only evict a peer own addresses (self-harm). The is_usable_contact inputs come from dig-nat own transport bookkeeping, never a peer-supplied wire field, so a remote peer cannot make an honest peer address look unspecified or port-0. The eviction is provably unable to drop index 0 (a majority family of a list of 9 or more has at least 5 members, so rposition never lands on the head), so a LIVE newest address cannot be evicted in favour of a stale one - which was the original defect. And the #1783 guard writes nothing durable.

The SIGPIPE flake is fixed and the sweep is clean. .github/actions/setup-linux-build/action.yml:56-60 reads ldd --version into a variable and slices it with read plus shell expansion - no pipe. I swept .github/, scripts/ and packaging/ for any other pipe-into-head under pipefail: the only remaining occurrence is the comment explaining the fix. The tail -n 1 in check-glibc-floor.sh is not the same hazard, since tail drains its input.

arm64 .deb metadata is correct, not copied. Architecture: ${ARCH} is parameterized, the filename carries the arch, and Depends: libc6 is arch-independent by design (dpkg resolves libc6:arm64). Confirmed at HEAD, not just in the canary: job 90479828149 logs container glibc: 2.31, declared floor: 2.31, check-glibc-floor: ... OK for the aarch64 payload, and Architecture: arm64.

The canary evidence is real. Run 30420036070, all 6 verify legs green. The arm64 amazonlinux:2023 leg resolves the complete set (libc.so.6 => /lib64/libc.so.6, /lib/ld-linux-aarch64.so.1, zero not-found) and both binaries print 0.67.0; ldd is logged unconditionally before each run, as claimed. Note the canary ran at a2f7f34, before the SIGPIPE fix - but the two .deb legs at HEAD exercise the fixed setup-linux-build step and pass, so the head revision of that step is executed evidence.

Beautiful-code gate: PASSES

retain_newest_per_family, is_usable_contact and should_adopt_dialed_peer are each one named concept with a WHY-comment that explains the defect rather than the mechanism, and every new test doc-comment states what it would catch. The expect in the eviction loop is provably unreachable and says so. The composite-action extraction is honest DRY - the naming genuinely could not diverge across two build jobs afterwards, and the guard test was strengthened to assert that.

Non-gating notes follow in a separate review comment; I will resolve those myself so they cannot block the merge.

[MichaelTaylor3d]

GATING - a third path publishes a pool address into the DHT, unguarded.

This guard is right, and so is the one in download.rs::on_pool_event. But the PR claims those are the only two paths, and they are not.

bring_up_dht at crates/dig-node-core/src/peer.rs:2575:

.map(|(peer_id, addr, _outbound)| {
    let mut bytes = [0u8; 32];
    bytes.copy_from_slice(peer_id.as_ref());
    // The pool reports each peer GOSSIP addr; the DHT must dial its DHT/peer-RPC addr (GAP 2).
    (bytes, dht_addr_from_gossip_addr(addr))
})

That feeds bootstrap_peers_from_pool (seams/dig_peer/dht.rs:456), which turns each entry into a BootstrapPeer::direct(...) with no filtering, and the result goes into service.bootstrap(&bootstrap). Same source (pool.connected_pool_peers()), same port shift, same destination (the DHT routing table) - but no is_usable_contact.

Why this is not merely theoretical: the call site own comment says "Bootstrap from the connected pool (+ relay-introducer peers discovered into it)", and a relay-introducer-discovered peer is precisely the accepted-relayed-circuit case dig-nat reports as [::]:0. Your rationale for the guard on this very line - "routing must never seed a lookup from a wildcard contact" - applies verbatim to bootstrap, which seeds the same table a moment earlier.

Fix is one line: use dht_contact_from_pool_addr at peer.rs:2575 and drop the Nones (or filter inside bootstrap_peers_from_pool, arguably the better home, since that is the function owning the pool-to-DHT-contact conversion).

It also needs a test. bootstrap_peers_from_pool_maps_each_pool_peer (dht.rs:946) feeds only 203.0.113.1:9444 and 203.0.113.2:9444 - the BENIGN-FIXTURE shape: the filter would be a no-op on that input, so adding one and then deleting it again leaves the test green either way. Give it a [::]:0 entry and assert the peer count drops.

While you are there, please restate the blast-radius claim in the PR body - "two independent entry points, no third path" is the sentence that sent me looking, and a reviewer who trusted it would have stopped.

[MichaelTaylor3d]

GATING (one sentence) - record the musl rejection here.

I agree with the decision. The musl built-in resolver is not a drop-in for the glibc one: no NSS module support, a much narrower resolv.conf reading, and historically no TCP fallback for a UDP answer over 512 bytes - which is exactly the shape of a DNS-seed record set carrying many A/AAAA entries. Trading a known floor problem for an unknown resolver problem on the node bootstrap path is the wrong trade, and you called it correctly.

The problem is that this reasoning exists ONLY in the PR description, while 11.3 now reads as though the -gnu target plus a 2.31 container is the whole contract. And the gates do not backstop it:

• scripts/check-glibc-floor.sh:41-45 explicitly returns OK for a binary with no versioned glibc symbols - the comment even says "(static or musl)";
• the container assert only compares the container glibc to the declared number, which a musl TARGET inside the same container still satisfies;
• verify-linux-floor would go green, since a musl binary starts everywhere.

So a future "just target x86_64-unknown-linux-musl and delete the container" passes all three gates and silently changes the resolver the node seeds from. That is precisely the drift 11.3 was written to prevent, one layer up.

One sentence closes it, e.g.: "The -gnu target is REQUIRED. A *-musl target would remove the floor but replaces the glibc NSS/resolver, which DNS seeding depends on; switching to musl is a deliberate protocol-level decision, not a build optimisation."

Since the gate script cannot see this, the SPEC sentence is the only durable record - please also add the same one-liner to the check-glibc-floor.sh:41-45 branch comment, which is the exact place someone will read "static or musl - OK" and conclude musl is sanctioned.

[MichaelTaylor3d]

Non-gating notes (I am resolving these myself so they cannot block the merge)

None of these gate. Recording them because they are cheap and sit next to work this PR already touched.

1. packaging/linux/build-deb.sh:47 - Depends: libc6 carries no version constraint, so apt has no way to express the 2.31 floor. On a host below the floor, apt-get install dig-node succeeds and the binary then fails at exec, which is a worse failure shape than a clean dependency refusal. Depends: libc6 (>= 2.31) (or generating it with dpkg-shlibdeps) would make apt do the right thing. The build-time gate you added is the important half; this is the delivery-time half.

2. .github/workflows/package.yml - the two .deb legs run check-glibc-floor.sh but NOT the impossible-2.0 negative control that build-binaries.yml runs. Since the whole point of the control is that a gate which cannot fail is a rubber stamp, and the .deb is a separate delivery path you explicitly gated for exactly that reason, the symmetry is worth having.

3. No guard test covers package.yml. release_workflow_dign_guard.rs now watches build-binaries.yml plus the staging action, and it caught a real drift this round - but the .deb path could silently drop container: debian:11 or the floor-check step with nothing to notice. A third include_str! assertion over package.yml (asserts the container image and the check-glibc-floor.sh invocation) matches the mechanical-guard-over-a-hand-maintained-claim pattern the rest of this PR follows well.

4. The .deb ships only dig-node, not the dign alias, while the raw release assets ship both. Pre-existing and out of scope here; noting it because every_build_job_builds_the_dign_bin asserting exactly 2 occurrences made me check, and an apt user therefore has no dign command. Worth a ticket rather than a change in this PR.

5. crates/dig-node-core/src/download.rs:376 - retain_newest_per_family recomputes the family counts on every iteration, which is quadratic in the overshoot. With a limit of 8 and a single insertion per event the overshoot is always exactly 1, so this is genuinely irrelevant to performance - I mention it only so the next reader does not think it was missed. The straightforward loop is the right call for readability here.