Skip to content

ci(release): always build aws-lc-sys from source in shipped builds - #796

Merged
jussielo-amd merged 2 commits into
mainfrom
ci/release-aws-lc-sys-build-from-source
Oct 29, 2026
Merged

jussielo-amd merged 2 commits into
mainfrom
ci/release-aws-lc-sys-build-from-source

Conversation

@jussielo-amd

Copy link
Copy Markdown
Collaborator

Why

aws-lc-sys 0.46 added automatic detection of an AWS-LC already present on the build host: it looks at OPENSSL_DIR, OPENSSL_INCLUDE_DIR/OPENSSL_LIB_DIR and pkg-config, and links the library when it validates as AWS-LC. The manylinux container that builds the release, nightly and PR-bundle binaries installs pkgconfig, so the probe can find a host-provided library, and a shipped binary would then depend on whatever the build environment happens to have. Setting AWS_LC_SYS_USE_SYSTEM=0 skips detection and always builds the vendored copy, as documented in the aws-lc-sys README.

What changes

AWS_LC_SYS_USE_SYSTEM=0 is set for every build whose output is shipped or downloadable:

  • release.yml and nightly.yml set it at workflow level, which covers the host-side steps and the Windows jobs.
  • ci.yml sets it on the two PR-bundle build steps only (Linux and Windows), so the debug and test builds keep the default behaviour.
  • The manylinux container does not inherit the workflow environment, so each of the three container builds forwards it with -e AWS_LC_SYS_USE_SYSTEM.

aws-lc-sys versions before 0.46 ignore the variable, so on a lockfile older than that the change is a no-op; it takes effect once the lockfile reaches 0.46 (#792).

Verification

  • The variable name and value were checked against the aws-lc-sys 0.46.0 README and builder: 0 skips detection and always builds from source, and the only conflicting setting is AWS_LC_SYS_SYSTEM_DIR, which no workflow sets.
  • docker run -e AWS_LC_SYS_USE_SYSTEM passes the host value into the container when it is set and leaves it unset otherwise, checked locally with a container.
  • All three workflow files parse, and the prek hooks pass.
  • Not verified: a release, nightly or PR-bundle build against aws-lc-sys 0.46, because the workflows cannot run locally and the lockfile does not yet pin 0.46; and the environment of the windows-latest runner.

Scope

The lockfile is not touched. No test pins the -e forwarding, so removing it from a container build would silently return that build to default detection; a check in xtask/src/workflow_contract.rs could pin it if wanted. The workflow-level setting in nightly.yml also reaches its self-hosted end-to-end lanes, which are not shipped; a runner that sets AWS_LC_SYS_SYSTEM_DIR itself would fail an aws-lc-sys 0.46 build there, because aws-lc-sys rejects that combination with AWS_LC_SYS_USE_SYSTEM=0.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (N/A: CI configuration, no ticket and no bug fix in the code.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (N/A)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. (N/A: no user-facing messages change.)

aws-lc-sys 0.46 and later detect an AWS-LC already present on the build host
(OPENSSL_DIR, OPENSSL_INCLUDE_DIR/OPENSSL_LIB_DIR or pkg-config) and link it
when it validates. The manylinux build container installs pkgconfig, so a
shipped binary could end up depending on whatever the host provides.

Set AWS_LC_SYS_USE_SYSTEM=0 in the release and nightly workflows, and for the
Linux and Windows PR bundle builds in ci.yml, so the vendored copy is always
built. The manylinux container does not inherit the workflow environment, so
forward the variable into each docker run.

The locked aws-lc-sys is still 0.41.0, which ignores the variable; it takes
effect once the lockfile moves to 0.46 (#792).

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the changes configuring shipped builds to use vendored AWS-LC.

No open issues 🎉

What changed in this PR

This PR makes shipped builds use vendored AWS-LC rather than a host-provided library once aws-lc-sys reaches 0.46, as planned in #792. The lockfile remains unchanged.

Changes:

  • Sets AWS_LC_SYS_USE_SYSTEM=0 across release and nightly workflows, and only on PR-bundle build steps in CI.
  • Explicitly forwards the setting into all three manylinux build containers.
File Description
.github/​workflows/​release.yml Enforces vendored AWS-LC for release builds, including manylinux.
.github/​workflows/​nightly.yml Applies the same setting to nightly builds and forwards it into manylinux.
.github/​workflows/​ci.yml Enforces vendored AWS-LC for Windows and Linux PR bundles without changing debug or test builds.

Review effort: Balanced

@jussielo-amd
jussielo-amd marked this pull request as ready for review October 29, 2026 14:03
@jussielo-amd
jussielo-amd requested a review from a team as a code owner October 29, 2026 14:03
@jussielo-amd
jussielo-amd requested a review from fredespi October 29, 2026 14:03

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues. Every shipped build in ci.yml, nightly.yml and release.yml (Linux container builds and the Windows jobs) is covered, and I found no other shipping build that was missed. The PR body matches the diff.

Non-blocking notes:

  • Nothing pins the -e AWS_LC_SYS_USE_SYSTEM forwarding in the container builds, so dropping it would silently revert that build to detection. An optional check in workflow_contract.rs could guard it (you already flagged this).
  • nightly.yml sets the variable at workflow level, so it also reaches the self-hosted e2e lanes. The PR text discloses this; it is a reasonable tradeoff versus the per-step scoping in ci.yml.
  • The explanatory comment is repeated across the three workflows. Fine for workflow YAML.

This is a no-op until the lockfile reaches aws-lc-sys 0.46 (#792), so I could not verify an actual build with it. CI and the remaining hardware lanes still need to finish.

-e CARGO_HOME=/host-cargo \
-e RUSTUP_HOME=/host-cargo/rustup \
-e ROCM_CLI_RELEASE_REF="${{ github.ref_name }}" \
-e AWS_LC_SYS_USE_SYSTEM \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: nothing pins this -e AWS_LC_SYS_USE_SYSTEM forwarding. A workflow_contract.rs check would stop it silently regressing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 208e86f. shipped_release_builds_pin_the_vendored_aws_lc_opt_out in xtask/src/workflow_contract.rs now fails if a manylinux release build stops forwarding AWS_LC_SYS_USE_SYSTEM (-e or --env), or if any shipped release build, Windows ones included, resolves it to something other than 0 across step, job and workflow env. It also pins the number of shipped builds per workflow, so a new build has to pin the variable too.

I left the other two notes as they are, per your assessment.

Dropping `-e AWS_LC_SYS_USE_SYSTEM` from a manylinux build container, or
changing the value a build resolves, would silently revert that build to host
AWS-LC detection with no other symptom.

For every shipped release `cargo build` step in release.yml, nightly.yml and
ci.yml, resolve the variable the way GitHub does (step env, then job env, then
workflow env) and require 0, and require the manylinux docker run that builds
to forward it with `-e` or `--env`. The expected number of shipped builds is
pinned per workflow, so a new build has to pin the variable too. The checker
is also exercised on inline YAML, covering each env layer, the accepted
forwarding spellings, unnamed steps, flag order and comments.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd added this pull request to the merge queue Oct 29, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 208e86f

Review — needs work

Full review of the whole change (base 1186ff11e93cf803408cad95826260d2cc5e2b07…head 208e86f4a46c611c122d1fc7a10b8d76aaabaae0, 4 files, 307 insertions). No ticket named in the PR body as something this PR closes; it references PR #792 only as context for when aws-lc-sys 0.46 lands, which is informational rather than a closure claim.

Blocking

  • PR description is stale relative to #792, which already merged before the current head was pushed — PR description (Why / Verification / Scope sections)
    #792 ("build(deps): bump rustls to 0.23.46 with webpki and aws-lc-rs") merged into main as 6ecb30a1 at 2026-10-29T14:16:14Z, bumping aws-lc-sys to 0.46.0. That is ~75 minutes before this PR's current head commit (208e86f4, authored 15:31:24) was pushed. The description still frames 0.46 as a future, not-yet-landed dependency — "aws-lc-sys versions before 0.46 ignore the variable... it takes effect once the lockfile reaches 0.46 (#792)", "Not verified: ... because ... the lockfile does not yet pin 0.46." This branch's own Cargo.lock still pins 0.41.0 (verified: git show 208e86f4:Cargo.lock vs git show origin/main:Cargo.lock) and was never rebased onto the new main, so none of this PR's own CI runs — including the run that validated the new contract test — ever exercised a real vendored build of 0.46 under AWS_LC_SYS_USE_SYSTEM=0, which is the one behavior this PR exists to guarantee. A reader of the current description would reasonably but incorrectly conclude this change has no live effect yet and that verification can wait.
    Confidence 90 · mechanical · Fix: update the description to reflect that #792 has merged, and rebase onto main before merging so CI exercises an actual AWS_LC_SYS_USE_SYSTEM=0 build against 0.46 (mitigating factor: #792's own CI already built 0.46 successfully in the same manylinux2014 and Windows environments under default detection, so the toolchain itself is known to work — only the forced-vendored path under this PR's new env var is unverified).

  • "Changes"/"Scope" sections omit the second commit entirely — PR description ("What changes" / "Changes" table / "Scope" section)
    The "Changes" table lists only release.yml, nightly.yml, and ci.yml; xtask/src/workflow_contract.rs (286 of the 307 changed lines — the bulk of the diff) is never mentioned. The "Scope" section still says "No test pins the -e forwarding... a check in xtask/src/workflow_contract.rs could pin it if wanted" — but the current head's second commit added exactly that check. The description was written for the first commit (8c7a2772, the one the two earlier reviews covered) and never updated for the second.
    Confidence 90 · mechanical · Fix: add a row/paragraph describing the new contract test and drop or correct the now-false "no test pins this" sentence.

Non-blocking

None

Decisions for the author

  • is_release_cargo_build cannot see a cargo build --release command wrapped across multiple lines — non-blocking-improvement
    The docstring already discloses this ("A command wrapped over several lines is not seen"), and no current build command in any of the three workflows is wrapped this way, so nothing is missed today. A future reflow of one of these build lines would silently stop being protected, with only the expected_steps count mismatch as a backstop — and a same-count reflow wouldn't even trigger that. Worth a one-line note if a future edit ever wraps one of these commands.

Positive signals

  • The new contract test checks both halves of the fix in one assertion path: the resolved env value (step → job → workflow precedence, matching real GitHub Actions resolution order) and the manylinux -e/--env forwarding, with an expected_steps sentinel that fails loudly if a shipped build is added, removed, or reshaped beyond what is_release_cargo_build recognizes.
  • aws_lc_opt_out_check_resolves_every_env_layer_and_spelling is an unusually thorough table of edge cases (env-layer overrides, unnamed steps, the e2e-harness exemption, comment-handling, a flag placed after the image name, a comment line that could be mistaken for a job key) — traced these against the parsing logic and all resolve correctly.

Deployment notes

main already carries aws-lc-sys 0.46 (via #792, merged before this PR's current head was pushed). Once this PR merges, the very next release/nightly/PR-bundle build will be the first one to actually exercise AWS_LC_SYS_USE_SYSTEM=0 forcing a from-source build in the manylinux container and on Windows — a path this PR's own CI has not run, since its branch still builds against the old 0.41.0 lockfile where the variable is a no-op. Rebasing before merge would turn this into a tested path rather than one going live in production release infra unverified.

What this covered

Read the full diff in full (all 4 changed files, plus the surrounding job/step context in the three workflow YAML files and the pre-existing helper functions in xtask/src/workflow_contract.rs the new code reuses), and had an independent fan-out worker review the same scope blind — from the code only, with no access to the PR description or either prior review. The diff is one tightly-coupled group (workflow changes plus the single test file guarding them), so one worker was the correct split per the method's coupling rule. Both passes independently traced the new parsing logic against the real content of ci.yml/nightly.yml/release.yml and found no correctness defects in it; this is corroborated by the PR's own green "Test (affected crates)" CI check at the current head SHA, which runs this crate's test suite (xtask/** is in the path filter that selects it). The two Step-9 findings above were verified independently via git log/git show/the GitHub API (commit timestamps, Cargo.lock contents at the relevant commits, and #792's merge state) — not something a diff-only pass could have found.

Did not run: a history/git blame pass beyond confirming the new code's comment and placement style matches the surrounding file's existing conventions; a prior-changes pass (no visibility into review threads on these workflow files beyond this PR's own reviews, which were deliberately not read per the blind-review rule — including a newer, zero-body review posted at this same head commit, which was intentionally left unread here). Design review (Step 7) considered text-pattern-based YAML parsing vs. a real parser, and scoping AWS_LC_SYS_USE_SYSTEM narrowly vs. workflow-wide; both are sound, pre-existing-pattern-consistent tradeoffs already disclosed by the author, so neither produced a blocking or decisions-for-the-author item beyond what's listed above. Not reconciled against the discussion on this PR.

Merged via the queue into main with commit 3dec0eb Oct 29, 2026
24 checks passed
@jussielo-amd
jussielo-amd deleted the ci/release-aws-lc-sys-build-from-source branch October 29, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants