Skip to content

ci: build the standard Linux binaries in a pinned Ubuntu 22.04 container and assert the glibc floor - #2887

Open
DeusData wants to merge 6 commits into
mainfrom
ci/release-ubuntu-2204-container
Open

DeusData wants to merge 6 commits into
mainfrom
ci/release-ubuntu-2204-container

Conversation

@DeusData

@DeusData DeusData commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

What does this PR do?

The standard (dynamic) Linux release binaries were built on the hosted runners' own Ubuntu 24.04 userland, so they required GLIBC_2.38 and GLIBCXX_3.4.32 and refused to start on Ubuntu 22.04 (#2486). This PR moves both of them into a pinned Ubuntu 22.04 build container and checks the resulting glibc floor on every built binary.

  1. Release build (_build.yml): a new build-linux job builds linux-amd64 and linux-arm64 inside ubuntu:22.04@sha256:076e7bf82319619f28dd0aa9c02ab34b055ad37100395b4f49a2ffba04fb5e11 (the multi-arch index digest, one pin for both arches). The distro toolchain (GCC 11.4 against glibc 2.35 headers) is what sets the floor; the hosted runner only hosts the container, so a runner image retirement no longer moves it. build-unix keeps the two macOS legs, the portable (static) legs are unchanged, and select-package needs the new job. Artifact names are unchanged.
  2. Floor assertion (scripts/ci/check-binary-composition.sh, A6-glibc-floor): when a venue declares CBM_GLIBC_FLOOR (and optionally CBM_GLIBCXX_FLOOR), no GLIBC_* / GLIBCXX_* version a dynamic ELF requires may be newer. Versions compare numerically and GLIBC_ABI_DT_RELR counts as 2.36 (the release that added it); other unnumbered names (GLIBC_PRIVATE) and a dynamic binary with no GLIBC_* requirement fail closed. build-linux declares 2.35 / 3.4.30 (what the pinned image's libc and libstdc++ provide), so all three release candidates are checked in prepare-release-candidates.sh. Undeclared, A6 reports n/a, so local builds, the local artifact smoke and the contract stubs on newer distros are unaffected.
  3. PR CI (pr.yml): a report-only glibc-floor leg builds the production binary in the same pinned image and runs the composition gate with the same floor whenever the selection holds smoke-ubuntu. It is not in ci-ok's needs and has no lane, so it never blocks or passes a merge (whether it should gate is a separate decision).
  4. Contracts: test_venue_parity_contract.sh asserts the digest pin, one shared digest for the release and PR legs, both arches, the declared floors, and that no standard Linux leg returns to a runner userland (this assertion follows the shape of the contract test in ci: build Linux amd64 release on Ubuntu 22.04 #2573). test_release_gate_chain_contract.sh lists build-linux among the native candidate producers. test_release_candidate_derivation_contract.sh proves A6 on the stripped stub candidate: RED at floor 1.0 and GREEN at 999.0 on glibc Linux, n/a on Mach-O / PE, fail-closed on a non-glibc Linux, exit 2 on a malformed floor.
  5. Comments: everything that said the standard Linux binary needs glibc 2.38+ now says 2.35+ (install.sh, npm, PyPI, cli.c, Homebrew formula, smoke-test.sh, _smoke.yml, check-glibc-compat.sh, the local glibc-floor venue). The installers keep choosing the -portable archive: a 2.35 floor still excludes Debian 11, RHEL 8 and Ubuntu 20.04.

Refs #2573 #2486

Evidence (local: Colima on Apple silicon, arm64 native, amd64 under Rosetta)

Same commit, headless production build (scripts/build.sh CC=gcc CXX=g++; the shipped composition differs only by swapping src/ui/embedded_stub.c for the generated asset data), the exact apt line of the new job:

build venue toolchain newest GLIBC needed newest GLIBCXX needed A6 (floor 2.35 / 3.4.30) release derivation with floor
ubuntu:22.04 pinned, arm64 GCC 11.4, glibc 2.35, binutils 2.38 GLIBC_2.34 GLIBCXX_3.4.29 PASS (19/19 assertions) 3 candidates published
ubuntu:22.04 pinned, amd64 (Rosetta) GCC 11.4, glibc 2.35, binutils 2.38 GLIBC_2.34 GLIBCXX_3.4.29 PASS (19/19 assertions) not run locally (disk); derivation is arch-independent and was proven on arm64
ubuntu:noble (24.04), arm64, control GCC 13.3, glibc 2.39, binutils 2.42 GLIBC_2.38 GLIBCXX_3.4.32 FAIL on both (exactly the #2486 loader errors) refused, nothing published

Two arm64 builds of the branch in the pinned image were byte-identical (sha256 78e7b712...). The 22.04-built binaries start (--version) and index a small fixture (cli index_repository, list_projects, search_graph) in a plain ubuntu:22.04 container with nothing installed, linking the image's own libc 2.35, libstdc++ 12.3 and zlib.

What only hosted CI and a release dry run can show

  • the container jobs on the hosted ubuntu-latest and ubuntu-24.04-arm runners (checkout, setup-node, attest and upload running inside the container);
  • the with-UI build (npm ci + embed) inside the container, and the frontend integrity scan there;
  • the PR glibc-floor leg on a hosted runner (it runs on this PR, since a workflow change selects every lane);
  • the full release pipeline: a maintainer-dispatched dry run of this branch.

Thank you to @lorenzozanee for #2573: the diagnosis there and its workflow contract test shaped this change, and the contract assertion here follows its shape. Thank you to @shockme for the precise loader errors and the Ubuntu 22.04 lead in #2486.

Checklist

  • Every commit is signed off (git commit -s) -- required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test) -- the affected contract steps were run locally; the full suite runs in PR CI
  • Lint passes (make -f Makefile.cbm lint-ci) -- the only C change is a comment (clang-format clean on the hunk)
  • New behavior is covered by a test (reproduce-first for bug fixes)

Follow-up: the PR leg gates, and the release smoke runs the binary on plain 22.04 (maintainer decisions, 2026-11-02)

Two commits on top of the three above. They supersede the last sentence of item 3: the PR glibc-floor leg is no longer report-only.

  1. glibc-floor is a gating PR lane (ci: make the glibc-floor PR leg a gating lane). scripts/ci/select-lanes.sh selects glibc-floor exactly when it selects smoke-ubuntu, i.e. for every change that can alter the production Linux binary and never for a docs-only or tests-only change. ci-ok needs the job, and require-all-green.sh maps it to its lane: a selected glibc-floor that fails, is skipped or cancelled, or never runs fails ci-ok; an unselected one may skip like every other lane. The lane-selector table, the lane-gate contract (green, failed, skipped, cancelled, missing and unselected cases) and the lane-wiring contract (the job serves the lane, ci-ok needs it, no continue-on-error) pin this.
  2. Runtime start check on plain Ubuntu 22.04 in the release smoke (ci: run the dynamic Linux binary on plain ubuntu:22.04 in the release smoke). New canonical entry scripts/ci/check-glibc-floor-runtime.sh: it runs a binary in a plain container of a digest-pinned image with nothing installed and requires --version, cli index_repository on a tiny C + Python fixture ("status":"indexed") and cli list_projects (lists the fixture) to succeed. In _smoke.yml every smoke-unix Linux leg (amd64 and arm64, each on its own shipped artifact) runs it as a gating step with the same ubuntu:22.04@sha256:076e7bf8... digest that build-linux builds in. test_venue_parity_contract.sh asserts the step exists once, runs for every Linux leg, has no continue-on-error, and pins the same digest as build-linux and the PR leg; the new entry also joins the --help and strict-flag interface probes.

Evidence (local, Colima on Apple silicon; the step's own command line, run with the step's env):

binary built in newest GLIBC / GLIBCXX needed runtime check in plain ubuntu:22.04@sha256:076e7bf8...
linux-arm64 the pinned 22.04 image (GCC 11.4, glibc 2.35) GLIBC_2.34 / GLIBCXX_3.4.29 PASS: starts, indexes the fixture (24 nodes, 27 edges, status indexed), lists it
linux-amd64 (Rosetta) the pinned 22.04 image (GCC 11.4, glibc 2.35) GLIBC_2.34 / GLIBCXX_3.4.29 PASS: same (24 nodes, 27 edges, status indexed, listed); run with --platform linux/amd64 because the host is arm64
linux-arm64, control ubuntu:24.04 (GCC 13.3, glibc 2.39) GLIBC_2.38 / GLIBCXX_3.4.32 FAIL at --version: GLIBC_2.38' not found, GLIBCXX_3.4.32' not found (the #2486 loader errors)

Every new contract assertion was checked RED-on-revert: removing the selector rule, the lane from the universe, the ci-ok job mapping, the ci-ok need, the job's own lane condition, or adding continue-on-error turns the lane contracts red; removing the runtime step, changing its digest, using a tag, limiting it to amd64, or making it continue-on-error turns the venue-parity contract red.

Only the release dry run shows the runtime step on the hosted ubuntu-latest and ubuntu-24.04-arm runners (Docker there pulls the pinned index digest for the runner's own architecture, as the portable legs' old-glibc check already does).

DeusData and others added 5 commits November 2, 2026 04:02
check-binary-composition.sh gains A6-glibc-floor. When a venue declares
CBM_GLIBC_FLOOR (and optionally CBM_GLIBCXX_FLOOR), no GLIBC_* or
GLIBCXX_* symbol version a dynamic ELF requires may be newer than that
floor. The names come from the dynamic version tables, which the loader
checks at startup and every strip level keeps. Versions compare
numerically (2.4 is older than 2.35); GLIBC_ABI_DT_RELR counts as 2.36.
Other unnumbered names (GLIBC_PRIVATE) and a dynamic binary that
requires no GLIBC_* version at all fail closed. A static ELF passes,
Mach-O and PE report n/a, and with no floor declared A6 reports n/a, so
local builds, the local artifact smoke and contract stubs on newer
distros are unaffected. A malformed floor is a usage error (exit 2).

The release candidate derivation contract proves it on the stripped
stub candidate: floor 1.0 fails and 999.0 passes on glibc Linux, n/a on
Mach-O and PE, fail closed on a non-glibc Linux, exit 2 on a malformed
floor.

Refs #2486

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The standard (dynamic) Linux release binaries were built on the hosted
runners Ubuntu 24.04 userland, so they required GLIBC_2.38 and
GLIBCXX_3.4.32 and refused to start on Ubuntu 22.04 (#2486).

A new build-linux job in _build.yml builds linux-amd64 and linux-arm64
inside ubuntu:22.04 pinned by the digest of its multi-arch index
(sha256:076e7bf8..., resolved 2026-11-02 from Docker Hub, one pin for
both arches) with the distro toolchain (GCC 11.4 against glibc 2.35),
and declares CBM_GLIBC_FLOOR=2.35 / CBM_GLIBCXX_FLOOR=3.4.30 (what the
image provides) for the composition gate, so every release candidate is
checked. The hosted runner only hosts the container, so a runner image
retirement no longer moves the floor. build-unix keeps the two macOS
legs, the portable legs are unchanged, select-package needs the new job
and the artifact names stay the same.

pr.yml gains a report-only glibc-floor leg: the production binary built
in the same image and checked by the composition gate with the same
floor whenever the selection holds smoke-ubuntu. It is not in ci-ok
needs and has no lane, so it never blocks or passes a merge.

The venue parity contract asserts the digest pin, one shared digest for
the release and PR legs, both arches, the declared floors and that no
standard Linux leg returns to a runner userland; the assertion follows
the shape of the contract test in #2573. The release gate chain
contract lists build-linux among the native candidate producers.

Refs #2573 #2486

Co-authored-by: lorenzozanee <wyz0707@proton.me>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Built in the pinned ubuntu:22.04 container, the standard Linux binary
requires at most glibc 2.35 and GLIBCXX_3.4.30. Update the comments
that said 2.38+: install.sh, the npm and PyPI wrappers, cli.c, the
Homebrew formula, smoke-test.sh, _smoke.yml, check-glibc-compat.sh and
the local glibc-floor venue. The installers keep choosing the -portable
archive, since a 2.35 floor still excludes Debian 11, RHEL 8 and Ubuntu
20.04.

The local smoke-glibc-floor lane keeps its assertion and now says what
it proves: a binary built in the noble test image refusing to start
there shows the venue is older than the build image.

Refs #2486

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The glibc-floor job builds the production binary in the pinned
ubuntu:22.04 release container and holds it to the release glibc floor
(2.35 / GLIBCXX 3.4.30). It ran report-only; it now gates like every
other lane.

- select-lanes.sh: new lane glibc-floor, part of the full set and
  selected whenever smoke-ubuntu is (a FOLLOWS rule), i.e. whenever the
  production Linux binary can change. Docs-only and tests-only changes
  do not select it.
- pr.yml: the glibc-floor job runs when its own lane is selected, and
  ci-ok needs it.
- require-all-green.sh: maps the glibc-floor job to its lane, so a
  selected glibc-floor that fails, is skipped or cancelled, or never
  runs fails ci-ok; when it is not selected it may skip.
- Contracts: the selector table and the FULL set carry the lane, and
  every selector case asserts glibc-floor is selected exactly when
  smoke-ubuntu is; the lane-gate contract adds green, failed, skipped,
  cancelled, missing and unselected cases; the lane-wiring contract
  requires the job to serve the lane, ci-ok to need it, and no
  continue-on-error on it.
- scripts/ci/README.md notes the rule.

Refs #2486

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… smoke

The composition gate asserts the glibc 2.35 floor on the built bytes,
but nothing ran those bytes on a 22.04 userland before a release.

- scripts/ci/check-glibc-floor-runtime.sh (new canonical entry): runs a
  linux binary in a plain container of a digest-pinned image with
  nothing installed and requires --version, cli index_repository on a
  tiny C and Python fixture (status indexed) and cli list_projects
  (lists the fixture) to succeed. A binary built on a newer userland
  fails at --version, with the loader errors in the log.
- _smoke.yml smoke-unix: a gating step runs it on every Linux leg
  (amd64 and arm64, each on its own artifact) with the same
  ubuntu:22.04 digest that build-linux builds in.
- test_venue_parity_contract.sh: the step must exist once on
  smoke-unix, run for every Linux leg, gate (no continue-on-error) and
  pin the same digest as build-linux and the PR glibc-floor leg; the
  core smoke-unix matrix must hold linux amd64 and arm64; the new entry
  joins the --help and strict-flag interface probes.
- scripts/ci/README.md lists the new entry.

Refs #2486

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

DeusData commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Two follow-up commits on top of the three reviewed ones, both from the maintainer decisions of 2026-11-02. dd4b5491 makes the glibc-floor PR leg a gating lane: select-lanes.sh selects it exactly when it selects smoke-ubuntu (any change that can alter the production Linux binary, never a docs-only or tests-only change), ci-ok needs the job, and require-all-green.sh maps it to its lane, so a selected glibc-floor that fails, is skipped or never runs fails ci-ok, while an unselected one may skip like every other lane. f7045ac6 adds a runtime start check to the release smoke: each Linux smoke leg (amd64 and arm64) runs its shipped binary in a plain container of the same pinned ubuntu:22.04 digest with nothing installed (--version, cli index_repository on a tiny fixture, cli list_projects), and the venue-parity contract holds that step to the build digest. Locally the check passes for a binary built in the pinned image and fails for a noble-built control with exactly the GLIBC_2.38 / GLIBCXX_3.4.32 loader errors from #2486; every new contract assertion goes red when the wiring it guards is removed. The release dry run of this branch comes next.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant