Repository navigation
ci(release): always build aws-lc-sys from source in shipped builds - #796
Conversation
aws-lc-sys 0.46 and later detect an AWS-LC already present on the build host (OPENSSL_DIR, OPENSSL_INCLUDE_DIR/OPENSSL_LIB_DIR or pkg-config) and link it when it validates. The manylinux build container installs pkgconfig, so a shipped binary could end up depending on whatever the host provides. Set AWS_LC_SYS_USE_SYSTEM=0 in the release and nightly workflows, and for the Linux and Windows PR bundle builds in ci.yml, so the vendored copy is always built. The manylinux container does not inherit the workflow environment, so forward the variable into each docker run. The locked aws-lc-sys is still 0.41.0, which ignores the variable; it takes effect once the lockfile moves to 0.46 (#792). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the changes configuring shipped builds to use vendored AWS-LC.
No open issues 🎉
What changed in this PR
This PR makes shipped builds use vendored AWS-LC rather than a host-provided library once aws-lc-sys reaches 0.46, as planned in #792. The lockfile remains unchanged.
Changes:
- Sets
AWS_LC_SYS_USE_SYSTEM=0across release and nightly workflows, and only on PR-bundle build steps in CI. - Explicitly forwards the setting into all three manylinux build containers.
| File | Description |
|---|---|
.github/workflows/release.yml |
Enforces vendored AWS-LC for release builds, including manylinux. |
.github/workflows/nightly.yml |
Applies the same setting to nightly builds and forwards it into manylinux. |
.github/workflows/ci.yml |
Enforces vendored AWS-LC for Windows and Linux PR bundles without changing debug or test builds. |
Review effort: Balanced
juhovainio
left a comment
There was a problem hiding this comment.
No blocking issues. Every shipped build in ci.yml, nightly.yml and release.yml (Linux container builds and the Windows jobs) is covered, and I found no other shipping build that was missed. The PR body matches the diff.
Non-blocking notes:
- Nothing pins the
-e AWS_LC_SYS_USE_SYSTEMforwarding in the container builds, so dropping it would silently revert that build to detection. An optional check inworkflow_contract.rscould guard it (you already flagged this). - nightly.yml sets the variable at workflow level, so it also reaches the self-hosted e2e lanes. The PR text discloses this; it is a reasonable tradeoff versus the per-step scoping in ci.yml.
- The explanatory comment is repeated across the three workflows. Fine for workflow YAML.
This is a no-op until the lockfile reaches aws-lc-sys 0.46 (#792), so I could not verify an actual build with it. CI and the remaining hardware lanes still need to finish.
| -e CARGO_HOME=/host-cargo \ | ||
| -e RUSTUP_HOME=/host-cargo/rustup \ | ||
| -e ROCM_CLI_RELEASE_REF="${{ github.ref_name }}" \ | ||
| -e AWS_LC_SYS_USE_SYSTEM \ |
There was a problem hiding this comment.
Optional: nothing pins this -e AWS_LC_SYS_USE_SYSTEM forwarding. A workflow_contract.rs check would stop it silently regressing.
There was a problem hiding this comment.
Added in 208e86f. shipped_release_builds_pin_the_vendored_aws_lc_opt_out in xtask/src/workflow_contract.rs now fails if a manylinux release build stops forwarding AWS_LC_SYS_USE_SYSTEM (-e or --env), or if any shipped release build, Windows ones included, resolves it to something other than 0 across step, job and workflow env. It also pins the number of shipped builds per workflow, so a new build has to pin the variable too.
I left the other two notes as they are, per your assessment.
Dropping `-e AWS_LC_SYS_USE_SYSTEM` from a manylinux build container, or changing the value a build resolves, would silently revert that build to host AWS-LC detection with no other symptom. For every shipped release `cargo build` step in release.yml, nightly.yml and ci.yml, resolve the variable the way GitHub does (step env, then job env, then workflow env) and require 0, and require the manylinux docker run that builds to forward it with `-e` or `--env`. The expected number of shipped builds is pinned per workflow, so a new build has to pin the variable too. The checker is also exercised on inline YAML, covering each env layer, the accepted forwarding spellings, unnamed steps, flag order and comments. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 208e86f
Review — needs work
Full review of the whole change (base 1186ff11e93cf803408cad95826260d2cc5e2b07…head 208e86f4a46c611c122d1fc7a10b8d76aaabaae0, 4 files, 307 insertions). No ticket named in the PR body as something this PR closes; it references PR #792 only as context for when aws-lc-sys 0.46 lands, which is informational rather than a closure claim.
Blocking
-
PR description is stale relative to #792, which already merged before the current head was pushed — PR description (Why / Verification / Scope sections)
#792 ("build(deps): bump rustls to 0.23.46 with webpki and aws-lc-rs") merged intomainas6ecb30a1at 2026-10-29T14:16:14Z, bumpingaws-lc-systo 0.46.0. That is ~75 minutes before this PR's current head commit (208e86f4, authored 15:31:24) was pushed. The description still frames 0.46 as a future, not-yet-landed dependency — "aws-lc-sys versions before 0.46 ignore the variable... it takes effect once the lockfile reaches 0.46 (#792)", "Not verified: ... because ... the lockfile does not yet pin 0.46." This branch's ownCargo.lockstill pins 0.41.0 (verified:git show 208e86f4:Cargo.lockvsgit show origin/main:Cargo.lock) and was never rebased onto the newmain, so none of this PR's own CI runs — including the run that validated the new contract test — ever exercised a real vendored build of 0.46 underAWS_LC_SYS_USE_SYSTEM=0, which is the one behavior this PR exists to guarantee. A reader of the current description would reasonably but incorrectly conclude this change has no live effect yet and that verification can wait.
Confidence 90 · mechanical · Fix: update the description to reflect that #792 has merged, and rebase ontomainbefore merging so CI exercises an actualAWS_LC_SYS_USE_SYSTEM=0build against 0.46 (mitigating factor: #792's own CI already built 0.46 successfully in the same manylinux2014 and Windows environments under default detection, so the toolchain itself is known to work — only the forced-vendored path under this PR's new env var is unverified). -
"Changes"/"Scope" sections omit the second commit entirely — PR description ("What changes" / "Changes" table / "Scope" section)
The "Changes" table lists onlyrelease.yml,nightly.yml, andci.yml;xtask/src/workflow_contract.rs(286 of the 307 changed lines — the bulk of the diff) is never mentioned. The "Scope" section still says "No test pins the-eforwarding... a check inxtask/src/workflow_contract.rscould pin it if wanted" — but the current head's second commit added exactly that check. The description was written for the first commit (8c7a2772, the one the two earlier reviews covered) and never updated for the second.
Confidence 90 · mechanical · Fix: add a row/paragraph describing the new contract test and drop or correct the now-false "no test pins this" sentence.
Non-blocking
None
Decisions for the author
is_release_cargo_buildcannot see acargo build --releasecommand wrapped across multiple lines — non-blocking-improvement
The docstring already discloses this ("A command wrapped over several lines is not seen"), and no current build command in any of the three workflows is wrapped this way, so nothing is missed today. A future reflow of one of these build lines would silently stop being protected, with only theexpected_stepscount mismatch as a backstop — and a same-count reflow wouldn't even trigger that. Worth a one-line note if a future edit ever wraps one of these commands.
Positive signals
- The new contract test checks both halves of the fix in one assertion path: the resolved env value (step → job → workflow precedence, matching real GitHub Actions resolution order) and the manylinux
-e/--envforwarding, with anexpected_stepssentinel that fails loudly if a shipped build is added, removed, or reshaped beyond whatis_release_cargo_buildrecognizes. aws_lc_opt_out_check_resolves_every_env_layer_and_spellingis an unusually thorough table of edge cases (env-layer overrides, unnamed steps, the e2e-harness exemption, comment-handling, a flag placed after the image name, a comment line that could be mistaken for a job key) — traced these against the parsing logic and all resolve correctly.
Deployment notes
main already carries aws-lc-sys 0.46 (via #792, merged before this PR's current head was pushed). Once this PR merges, the very next release/nightly/PR-bundle build will be the first one to actually exercise AWS_LC_SYS_USE_SYSTEM=0 forcing a from-source build in the manylinux container and on Windows — a path this PR's own CI has not run, since its branch still builds against the old 0.41.0 lockfile where the variable is a no-op. Rebasing before merge would turn this into a tested path rather than one going live in production release infra unverified.
What this covered
Read the full diff in full (all 4 changed files, plus the surrounding job/step context in the three workflow YAML files and the pre-existing helper functions in xtask/src/workflow_contract.rs the new code reuses), and had an independent fan-out worker review the same scope blind — from the code only, with no access to the PR description or either prior review. The diff is one tightly-coupled group (workflow changes plus the single test file guarding them), so one worker was the correct split per the method's coupling rule. Both passes independently traced the new parsing logic against the real content of ci.yml/nightly.yml/release.yml and found no correctness defects in it; this is corroborated by the PR's own green "Test (affected crates)" CI check at the current head SHA, which runs this crate's test suite (xtask/** is in the path filter that selects it). The two Step-9 findings above were verified independently via git log/git show/the GitHub API (commit timestamps, Cargo.lock contents at the relevant commits, and #792's merge state) — not something a diff-only pass could have found.
Did not run: a history/git blame pass beyond confirming the new code's comment and placement style matches the surrounding file's existing conventions; a prior-changes pass (no visibility into review threads on these workflow files beyond this PR's own reviews, which were deliberately not read per the blind-review rule — including a newer, zero-body review posted at this same head commit, which was intentionally left unread here). Design review (Step 7) considered text-pattern-based YAML parsing vs. a real parser, and scoping AWS_LC_SYS_USE_SYSTEM narrowly vs. workflow-wide; both are sound, pre-existing-pattern-consistent tradeoffs already disclosed by the author, so neither produced a blocking or decisions-for-the-author item beyond what's listed above. Not reconciled against the discussion on this PR.
Why
aws-lc-sys 0.46 added automatic detection of an AWS-LC already present on the build host: it looks at
OPENSSL_DIR,OPENSSL_INCLUDE_DIR/OPENSSL_LIB_DIRand pkg-config, and links the library when it validates as AWS-LC. The manylinux container that builds the release, nightly and PR-bundle binaries installspkgconfig, so the probe can find a host-provided library, and a shipped binary would then depend on whatever the build environment happens to have. SettingAWS_LC_SYS_USE_SYSTEM=0skips detection and always builds the vendored copy, as documented in the aws-lc-sys README.What changes
AWS_LC_SYS_USE_SYSTEM=0is set for every build whose output is shipped or downloadable:release.ymlandnightly.ymlset it at workflow level, which covers the host-side steps and the Windows jobs.ci.ymlsets it on the two PR-bundle build steps only (Linux and Windows), so the debug and test builds keep the default behaviour.-e AWS_LC_SYS_USE_SYSTEM.aws-lc-sys versions before 0.46 ignore the variable, so on a lockfile older than that the change is a no-op; it takes effect once the lockfile reaches 0.46 (#792).
Verification
0skips detection and always builds from source, and the only conflicting setting isAWS_LC_SYS_SYSTEM_DIR, which no workflow sets.docker run -e AWS_LC_SYS_USE_SYSTEMpasses the host value into the container when it is set and leaves it unset otherwise, checked locally with a container.windows-latestrunner.Scope
The lockfile is not touched. No test pins the
-eforwarding, so removing it from a container build would silently return that build to default detection; a check inxtask/src/workflow_contract.rscould pin it if wanted. The workflow-level setting innightly.ymlalso reaches its self-hosted end-to-end lanes, which are not shipped; a runner that setsAWS_LC_SYS_SYSTEM_DIRitself would fail an aws-lc-sys 0.46 build there, because aws-lc-sys rejects that combination withAWS_LC_SYS_USE_SYSTEM=0.tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (N/A: CI configuration, no ticket and no bug fix in the code.)docs/architecture.md. (N/A)