Skip to content

ROCMAI-134: lock runtime dependencies per OS and propose updates weekly - #941

Open
rominf wants to merge 23 commits into
mainfrom
build/runtime-deps-locks
Open

rominf wants to merge 23 commits into
mainfrom
build/runtime-deps-locks

Conversation

@rominf

@rominf rominf commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The third-party runtimes rocm-cli installs fall behind with nothing reporting it, and their pins are scattered. Lemonade's version sat in runtime-deps.toml while its archive digests were copied by hand into the adapter, so a bump had to touch both and nothing tied them together. The ROCm SDK and vLLM Python stacks are resolved at install time against AMD's indexes, so what a user gets moves without review. Dependabot cannot take this over: it does not read mise.toml, its uv support reads only pyproject.toml + uv.lock, and it has open bugs with explicit and per-package indexes (details in docs/runtime-dependencies.md).

What changes

Binaries via mise. runtimes/mise.toml pins Lemonade (and uv, as lock tooling); mise locks a URL and SHA-256 per supported platform in runtimes/mise.lock. rocm-deps generates its constants from that lock with one generator (pins_gen.rs), which is also compiled into cargo xtask deps check, so the build and the check accept the same locks. The Lemonade adapter checks downloads against the locked digest, which is byte-identical to the hand-copied one it replaces; runtime-deps.toml is removed. runtimes/bin/mise and mise.cmd are vendored mise bootstrap launchers (MIT, regeneration command in MANIFEST.md), so no prior mise install is needed.

Python stacks locked per OS with uv. runtimes/profiles.toml declares the SDK (all 25 arch extras, Linux + Windows) and vLLM (Linux) profiles; cargo xtask deps lock resolves each into runtimes/locks/pylock.<profile>-<os>.toml (PEP 751). Locks are per OS because AMD's Linux torch wheel requires triton and the Windows one does not, so a universal lock is wrong for one of them. Each lock's header records a digest of its inputs (the profile's python, platform, requirements and ROCm line, the policy's vendor packages, and uv.toml), so a lock left stale by an edit is caught offline, and the --exclude-newer cutoff it was resolved at. A committed lock that exists but cannot be read is an error, not "no lock yet". A re-lock writes every new lock before replacing any committed one, so a failed write changes no lock (a rename failing part-way can still leave earlier ones replaced). It resolves fresh at that cutoff, so it reproduces the lock unless an input changed or AMD published a new build of an AMD-only package (those are exempt from the cutoff, since AMD's indexes publish no upload times); --upgrade (the weekly update) sets a new cutoff 7 days before now, and a resolution that differs only in its cutoff leaves the committed lock unchanged, so a week with nothing new proposes nothing. The previous lock is deliberately not handed to uv as version preferences: with AMD's indexes mirroring common packages ahead of PyPI, successive runs alternate between the two sets (jinja2 3.1.6/3.1.5, sympy 1.14.0/1.13.3, vLLM's datasets 5.1.0/1.1.1) and never converge.

cargo xtask deps check (also run by cargo test -p xtask) verifies both lock files: Python artifacts are https with a SHA-256 from PyPI or AMD's indexes, AMD-only packages come from AMD only, +rocm builds stay on the profile's ROCm line, wheels fit the OS and the profile's Python and ABI, no package names a vcs, directory or archive source, and every package name is a PEP 508 name and every version uses PEP 440's characters (so neither can break the weekly PR's Markdown table), and a lock names no field the tooling does not know, at any level; binaries are locked for exactly Linux and Windows x64 as release assets of their own repository at the locked version (url and url_api), uv with its build attestation, a provenance on any other tool is refused, and unknown lock fields are refused at every level, the top level included. cargo xtask deps diff prints the change table used as the weekly PR body, and errors on a package locked without a version or twice, with an invalid name or version, or from a vcs, directory or archive source, rather than relying on deps check having run; binary rows are checked with the same building blocks deps check uses (owner/repo, plain-segment platform and version, SHA-256 hex) before they are rendered. cargo xtask deps verify-downloads downloads every locked binary, uv's included (its provenance string is written by the job that ran upstream code, so it excuses nothing), and checks its SHA-256 against the lock.

CI. A path-filtered, non-required runtime-locks job re-locks at the committed pins and fails on any difference, uploading the regenerated files; it also runs deps verify-downloads. runtime-deps-update.yml runs weekly and on demand: propose (read-only; uv building AMD's rocm sdist means upstream code runs here) moves pins past a 7-day cooldown and re-locks (the cooldown covers binaries and PyPI packages; AMD-only packages such as rocm, torch and vllm move as soon as AMD publishes them, because AMD's indexes publish no upload times); check (read-only) runs the base commit's validator, deps check and deps verify-downloads on the artifact and recomputes the pull request's table, refusing a summary that differs; open-pr validates again with the validator fetched at the base commit and commits through createCommitOnBranch, which GitHub signs, and fails rather than reset the update branch while it holds a commit the workflow did not make, so a maintainer's fixup is never discarded; the only files it deletes on the update branch are stale runtimes/locks/pylock.*.toml locks. The Windows debug job bootstraps mise through mise.cmd. A change under runtimes/ also runs the self-hosted GPU lanes, which exercise the pinned Lemonade. cargo xtask affected treats docs/runtime-dependencies.md (read by an xtask test) and crates/rocm-deps/src/pins_gen.rs (compiled into xtask by path) as forcing a full-workspace run, so an edit to either alone still runs the tests that depend on it.

Until the vLLM adapter reads its lock, a test holds the vllm-rocm10-1 profile's pins equal to the adapter's 10.1 build row in engines/vllm/src/discover_builds.rs. The vendored mise launchers carry mise's MIT licence text in runtimes/bin/LICENSE-mise.

The design and its trade-offs are in docs/runtime-dependencies.md.

Verification

  • cargo clippy --workspace --all-targets -D warnings, cargo fmt --check, cargo xtask check-crate-edges and prek (licence headers and lychee not installed locally) pass.
  • cargo test --workspace --all-targets: everything this change touches passes. Three apps/rocm tests (stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails, two git_watch_paths_*) fail on the author's machine on main as well; they pass in CI.
  • The committed locks were generated by the runtime-locks job, and its next run re-locked at the recorded cutoff and reproduced them byte for byte (the job's diff gate passed). The preferences-based design this replaced failed that same gate; the non-convergence was reproduced locally with uv 0.13.6.
  • The windows-debug job bootstraps mise through runtimes/bin/mise.cmd and checks its version.
  • python3 -I scripts/validate_runtime_update.py --self-test: 36 cases pass.
  • New guards were broken deliberately to confirm their tests fail (single-branch mutation): the lock checks (foreign host, lookalike AMD host, older release under a newer version, url_api outside the repository for github: and aqua: tools, unknown fields, missing attestation, provenance on an unattested tool, verify-downloads re-hashing uv despite its attestation string, cargo xtask affected selecting xtask for discover_builds.rs, each half of the update's ownership filter, the version-prefix rule (0.29.* accepts 0.29rc1.dev0, not 0.29adventure), the binary-row checks in deps diff, a tag in place of a commit SHA in the runtime-lock workflows, each character rule for package names and versions, stale inputs, missing cutoff, missing uv.toml, per-OS platform wiring, the unchanged-upgrade rule), generator refusals, validator cases, and workflow pins (an added env var, an extra job, a defaults: block, a dropped deps check, run-id on the download step, the filters a runtimes change must trip). Survivors that remain are equivalent mutants, such as symlink_metadata versus metadata before remove_dir_all, which does not follow a symlink either way. One test runs the commit step's actual Python payload builder on a fixture.
  • Resolution of the SDK profile for both OSes was checked with uv 0.13.6: the Windows lock drops triton and the Linux-only device packages.

Not verified: the weekly workflow end to end, which runs only from main; real installs from the Python locks, since no installer reads them.

Security scan suppressions. The pull-request trivy scan reports five HIGH CVEs in runtimes/locks/pylock.vllm-rocm10-1-linux.toml: four in vLLM 0.29.1 (fixed upstream in 0.31/0.32) and one in flash-attn 2.8.3 (no fix). These are the newest builds AMD publishes for ROCm 10.1, so no re-lock clears them, and nothing installs from this lock yet. .trivyignore.yaml suppresses those five IDs for that one file only, expiring 2027-02-10; MEDIUM and LOW findings still appear in the report. ROCMAI-922 (installing vLLM from this lock) must not ship these builds unpatched.

Why one PR. The pieces depend on each other: the weekly workflow runs the lock tooling, the tooling runs through the vendored mise launchers, and the Lemonade constants are generated from the same mise.lock, so landing any one alone would leave it untested or unused. The next steps are separate: keeping the mise launchers current (#1028), installing vLLM from its lock (ROCMAI-922), and making runtime-locks cheap enough to require once AMD's indexes publish hash links (ROCm/TheRock#9733).

No Gherkin scenario: nothing a CLI user can observe changes (the Lemonade version and digest keep their values); the Python locks have no consumer.

Scope

No installer reads the Python locks yet; moving the SDK, vLLM, ComfyUI and llama.cpp installs onto them are follow-ups listed in the design doc. vLLM's version table in engines/vllm is unchanged. The weekly workflow needs "Allow GitHub Actions to create and approve pull requests" enabled, and its PRs, opened with GITHUB_TOKEN, need a close/reopen to run CI. The workflow recognises its own commits on the update branch by the bot's login, or by the bot's commit email when GitHub leaves the author unresolved; a maintainer's fixup stops the run instead of being reset away. The reset is GraphQL updateRefs with the checked head as beforeOid, so a fixup pushed between the check and the reset fails the run instead of being lost; this, like the rest of the workflow, is unverified until the post-merge dispatch. engines/vllm/src/discover_builds.rs, which the vLLM drift test reads as text, now forces a full workspace run in cargo xtask affected.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not a bug fix.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (xtask/src/deps/.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state. (deps check's remediation: relocking_clears_a_stale_lock_but_not_a_policy_problem.)

Lemonade's version was pinned in runtime-deps.toml and its archive
digests were copied by hand into the Lemonade adapter, so a bump had to
touch both and nothing tied them together.

runtimes/mise.toml now pins Lemonade (and uv, as lock tooling), and
mise locks a URL and SHA-256 per supported platform in runtimes/mise.lock.
rocm-deps generates its constants from that lock with one generator,
pins_gen.rs, which refuses unsupported platforms, unsafe tool names,
non-https URLs and malformed digests. The Lemonade adapter checks
downloads against the locked digest, and a test ties its derived
download URLs to the locked ones.

runtimes/bin/mise and mise.cmd are the vendored mise bootstrap launchers
(regeneration command in MANIFEST.md), so no prior mise install is
needed. The Windows launcher is checked out with CRLF line endings.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The ROCm SDK and vLLM stacks are resolved at install time today, and
nothing reports when their pins fall behind or a lock stops matching
them.

runtimes/profiles.toml declares each stack's requirements and the OSes
it is locked for; `cargo xtask deps lock` resolves each profile per OS
with the pinned uv into runtimes/locks/pylock.<profile>-<os>.toml (PEP
751). Locks are per OS because AMD's torch wheels declare different
dependencies on Linux and Windows. The header records a digest of each
lock's inputs (python, platform, requirements and uv.toml), so a lock
left stale by an input edit is detected offline.

`cargo xtask deps check`, also run by `cargo test -p xtask`, verifies
both lock files: every Python artifact is https with a SHA-256 and comes
from PyPI or AMD's indexes, AMD-only packages only from AMD, builds stay
on the profile's ROCm line and wheels fit the OS; every binary is locked
for exactly Linux and Windows x64 as a release asset of its own
repository at the locked version, url_api included, and unknown lock
fields are refused. `cargo xtask deps diff` prints the change table the
weekly update uses as its pull request body.

The lock files themselves are not in this commit: they are generated by
the runtime-locks CI job and committed from its artifact, and until then
`deps check` reports them missing.

No installer reads the Python locks yet; docs/runtime-dependencies.md
lists the follow-ups that will.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Pins only stay current if something moves them, and a lock only stays
honest if something re-derives it.

The runtime-locks CI job re-locks at the committed pins on pull requests
that can change a lock and fails on any difference, uploading the
regenerated files. It is path-filtered and not required, since it
downloads every AMD wheel the locks name.

runtime-deps-update.yml runs weekly and on demand: `propose` (read-only;
uv builds AMD's rocm sdist, i.e. runs upstream code) moves pins past a
7-day cooldown and re-locks; `check` (read-only) runs the base commit's
validator and `cargo xtask deps check` against the artifact; `open-pr`
validates it again with the validator fetched at the base commit and
commits through createCommitOnBranch, which GitHub signs. It needs
"Allow GitHub Actions to create and approve pull requests" enabled.

The Windows debug job bootstraps mise through runtimes/bin/mise.cmd.
Workflow contract tests pin the privileged steps, the jobs' environments
and inputs, and run the commit step's payload builder on a fixture.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

The three Python locks are committed in 1309fd6, taken unchanged from the runtime-locks artifact. Each records the exclude-newer cutoff it was resolved at, and the following runtime-locks run re-locked at that cutoff and reproduced them exactly.

Generated by the runtime-locks CI job (`runtimes/bin/mise run --cd
runtimes lock`, 7-day cooldown) at the committed pins, and taken from its
artifact unchanged. `cargo xtask deps check` passes on them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The runtime-locks job re-locked at unchanged pins and produced different
locks: jinja2 3.1.6 became 3.1.5, sympy 1.14.0 became 1.13.3, and in the
vLLM lock datasets 5.1.0 became 1.1.1. Reproduced locally, the cause is
handing uv the previous lock as version preferences: with AMD's indexes
mirroring common packages ahead of PyPI, successive runs alternate
between the two sets and never converge. Two fresh resolutions agree.

So the previous lock is no longer given to uv. Each lock instead records
the --exclude-newer cutoff it was resolved at, and a re-lock resolves
fresh at that cutoff; --upgrade, or a lock that does not exist yet, sets
a new cutoff the cooldown before now. `deps check` requires the cutoff,
and the AMD-only exemptions now apply to every resolution.

Also print workspace-relative lock paths with `/` on every OS: on
Windows `Path::join` made messages read `runtimes/locks\pylock...`,
which failed the Windows test of `deps check`'s messages.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Conflicts: the CI rust filter keeps runtimes/** and main's
proptest-regressions entry and drops runtime-deps.toml, which this
branch removes; docs/testing.md and affected.rs keep both sides.

main added a `run_script` contract helper with different semantics, so
this branch's helper is renamed `run_script_dedented`. main's pre-push
selection expects docs/testing.md to select specific crates, so the
remediation-spelling test now reads only docs/runtime-dependencies.md
(and mise.toml and the workflows), and docs/testing.md and
docs/engine-plugins.md no longer force a full-workspace run.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Generated by the runtime-locks CI job from the commit that made each lock
record its exclude-newer cutoff, and taken from its artifact unchanged.
Each lock now records 2026-11-02T08:40:09Z, so the next re-lock resolves
at the same cutoff and reproduces it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Every `--upgrade` takes a new cutoff and recorded it in every lock
header, so the weekly update would have rewritten all locks and opened
a pull request each week with "No runtime versions changed.". A
resolution that differs from the committed lock only in its cutoff now
keeps the committed file unchanged.

Also from review:
- run the self-hosted GPU lanes on a change under runtimes/, which
  holds the Lemonade version they exercise, and pin every filter a
  runtimes change must trip;
- recompute the update's pull-request table in the read-only check job
  and refuse an artifact whose summary differs;
- say that a re-lock reproduces its lock except where AMD published a
  new build of an AMD-only package, which no cutoff holds back, in the
  docs, comments and the runtime-locks error;
- announce, and test, re-locking at a new cutoff when the recorded one
  is missing or invalid;
- refuse whitespace and option-like names in profile requirements, and
  a directory named like a lock in the update artifact;
- tie the runtime cooldown to Dependabot's;
- add tests for branches review showed no test pinned, and narrow
  comments that claimed more than the code does.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Conflict in engines/lemonade/src/install.rs: main added a manifest test
where this branch added the locked-digest test; both are kept.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf marked this pull request as ready for review November 9, 2026 13:49
@rominf
rominf requested a review from a team as a code owner November 9, 2026 13:49
Conflict in docs/architecture.md: the rocm-deps entry keeps this
branch's description (constants from runtimes/mise.lock via pins_gen.rs)
and main's sentence on lib.rs and its dependants.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a well-structured change and the trust boundaries in the update workflow look careful. I found nothing blocking. CI on the head commit is green (31 checks). This was a static review: I did not run the Rust tests or the workflows. I did run scripts/validate_runtime_update.py --self-test, which passes.

Non-blocking notes (two more are inline):

  • tests/e2e-cucumber/expectations.toml:181 (a file this PR doesn't touch) still says Lemonade 11.5.1 is "the pin in runtime-deps.toml", which this PR deletes. Pointing it at runtimes/mise.toml would finish the job; the 11.5.1 literals on lines 178, 207 and 212 will also go stale on the first weekly Lemonade bump.
  • The PR description says the validator self-test has 31 cases; at this head it prints 36.
  • The vLLM pins (0.29, torch 2.13) are now written both in runtimes/profiles.toml and in engines/vllm/src/discover_builds.rs, with nothing tying them together. Fine as a stated follow-up, but a small drift test would stop the two from diverging silently in the meantime.

🤖 by agenthub on AMD AgentHub

Comment thread MANIFEST.md
All dependencies are sourced from [crates.io](https://crates.io) unless noted
otherwise. No vendored copies of third-party source are included in this
repository.
otherwise. One piece of third-party code is included in this repository:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this says the launchers are MIT-licensed third-party code from mise, but neither runtimes/bin/mise nor mise.cmd carries a copyright or licence line, and THIRD_PARTY_NOTICES.txt does not mention mise. MIT asks for the notice to travel with copies, so adding mise's copyright and licence text to THIRD_PARTY_NOTICES.txt would be good.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7ed9a22: mise's copyright and MIT text now ship as runtimes/bin/LICENSE-mise, and MANIFEST.md points to it. I put it beside the launchers rather than in THIRD_PARTY_NOTICES.txt because that file is generated from the Cargo graph by cargo xtask tpn and covers what the binaries ship, which the launchers aren't.

Comment thread xtask/src/deps/pylock.rs
}

#[derive(Debug, Clone, Deserialize)]
pub(crate) struct Package {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Package (and PyLock) don't use deny_unknown_fields, while the PR says unknown lock fields are refused for the mise lock. deps check only looks at wheels and sdist, so a package entry carrying archive, vcs or directory next to (or instead of) wheels would be read without its source being checked against the host/hash rules. uv doesn't emit those for index packages today, so it's a hardening point: refusing unknown package keys would keep the Python side as strict as the binary side.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7ed9a22: a package naming a PEP 751 vcs, directory or archive source is now refused, so only wheels and sdists, which are held to the host and hash rules, can reach a lock. I refused those three keys rather than all unknown keys, since uv writes routine fields like index and marker there.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through this against ROCMAI-134 and the diff. I found no correctness defects in the generator, lock/check logic, validator or the workflow trust split, and CI is green. I'm requesting changes because of a few decisions that should be made explicitly before merge, mostly around integrity provenance, the untested weekly workflow, and scope against the ticket. Inline comments carry the details.

Needs a decision / fix before merge

  • Lemonade digest provenance. mise.lock records checksum_source = "mise-versions", i.e. a third-party index, and that digest is the adapter's only integrity check. deps check ignores the field, so a weekly bump can change the digest with nothing calling it out. Please surface a changed digest/provenance in the diff table or the weekly PR body, or record that this is accepted.
  • Cooldown framing. AMD-only packages (rocm, torch, vllm, etc.) are exempt from the 7-day cooldown, and they are most of what moves each week. Please say so plainly in the PR text and the mise.toml comment rather than describing a uniform cooldown.
  • Weekly workflow never run end to end. The riskiest step is mise upgrade --bump rewriting version inside the inline-table Lemonade pin. Please plan a workflow_dispatch dry run right after merge and watch it to green, with a fallback if --bump doesn't touch table-form pins.
  • Forced reset of runtime-deps/update. Each run discards any maintainer fixup pushed during the week. Consider skipping the reset (or commenting) when the branch has commits not authored by the bot.

Scope vs. ROCMAI-134 (the ticket asks for a pure refactor lifting the Lemonade and vLLM pins into one TOML)

  • The vLLM half isn't lifted; the ticket needs updating or a vLLM follow-up linked.
  • The PR goes well beyond a pure refactor: a weekly write-token workflow, vendored mise launchers, per-OS Python locks with no consumer, and ~6k lines of xtask. Please update the ticket to reflect the real scope.
  • The EAI-7196 per-GPU-family schema constraint isn't addressed; please state whether that's accepted.

Smaller notes

  • Any pin/profile edit makes cargo test -p xtask fail until a re-lock (~17 GiB of downloads, or the CI artifact). It's documented, but a standing contributor cost.
  • wheel_fits_os checks only platform tags, not the Python tag/ABI.

I did not repeat the points already raised in the existing approval (the stale expectations.toml reference, "31 cases" vs 36, and the vLLM pin duplication). This is a read of the diff only; I did not build or run anything locally.

Comment thread runtimes/mise.lock

[tools."github:lemonade-sdk/lemonade"."platforms.linux-x64"]
checksum = "sha256:bdfd3c3e5d6eda5101c8a32f36e6dd9236ceb9ab2eb66734c32628e6e86e18ac"
checksum_source = "mise-versions"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lemonade's digest here is checksum_source = "mise-versions": sourced from mise's third-party index rather than GitHub or Lemonade, and it is the only integrity check the adapter applies. The doc says so, but deps check ignores the field and a weekly bump will change this digest without anything calling it out. Could the diff table or PR body flag a changed checksum_source/digest for tools without provenance, so the reviewer knows to verify it independently?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 07a8c8e: rather than only flag it, the digest is now checked. cargo xtask deps verify-downloads downloads every locked asset without a build attestation (today, Lemonade's two archives) and compares its SHA-256 with mise.lock. The weekly update's read-only check job runs it before a pull request is opened, and the runtime-locks job runs it on any PR touching runtimes/, so a hand bump is checked too. Run against this PR's lock, both 11.5.1 archives match.

Comment thread runtimes/mise.lock

[tools."github:lemonade-sdk/lemonade"."platforms.windows-x64"]
checksum = "sha256:50a133bbc35c4f3f8971eafef2c9fe56c4bbbfb0f1032bf8728324d8d8c8a0e1"
checksum_source = "mise-versions"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the linux-x64 entry: digest provenance is mise-versions only. Worth an explicit reviewer step on every Lemonade bump.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by the same check (07a8c8e): deps verify-downloads re-hashes the windows-x64 archive as well, in the weekly check job and in runtime-locks.

Comment thread runtimes/profiles.toml
# indexes publish no upload times. An AMD-only dependency missing from this
# list makes the weekly update fail; add it here.
vendor-hosts = ["stable.repo.amd.com", "rocm.frameworks.amd.com"]
vendor-packages = [

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything in vendor-packages is exempted from the --exclude-newer cutoff (AMD's indexes have no upload times), so the 7-day cooldown does not apply to rocm, torch, vllm etc., which is most of what moves each week. Fine as designed, but the mise.toml comment and PR text describe a uniform cooldown; consider saying plainly that AMD-only packages are not held back.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. 07a8c8e says so in the mise.toml comment, the update command's row in docs/runtime-dependencies.md and the PR description: the 7-day cooldown covers binaries and PyPI packages, and AMD-only packages (rocm, torch, vllm, ...) move as soon as AMD publishes them, since AMD's indexes publish no upload times.

Comment thread runtimes/mise.toml
# Lemonade's embeddable release. Each platform names its asset exactly: the
# release also carries .msi/.deb/.rpm installers, and a pattern shared by both
# platforms has been seen to lock the Windows zip for linux-x64.
"github:lemonade-sdk/lemonade" = { version = "11.5.1", platforms = { linux-x64 = { asset_pattern = "lemonade-embeddable-*-ubuntu-x64.tar.gz" }, windows-x64 = { asset_pattern = "lemonade-embeddable-*-windows-x64.zip" } } }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The weekly mise upgrade --bump has to rewrite version inside this inline table; the validator assumes it does but the workflow has never run. Suggest a workflow_dispatch dry run right after merge to confirm the bump lands in this table form (otherwise the weekly job would silently propose nothing for Lemonade).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked before merge rather than after: on a scratch copy of this mise.toml pinned to 11.4.0, mise upgrade --bump (mise 2026.10.25, the pinned launcher) rewrote version inside this inline table to 2026.44.1, the newest release past the cooldown (it skipped 2026.45.3 as too new, and the candidate-* tags). mise lock then locked both 2026.44.1 archives, and cargo xtask deps check accepted the result. I'll still trigger a workflow_dispatch run once this merges and watch it through.

# The branch always restarts from that commit: one proposal,
# regenerated from scratch each week, never stacked on the last one.
if gh api "repos/$REPO/git/ref/heads/$BRANCH" >/dev/null 2>&1; then
gh api -X PATCH "repos/$REPO/git/refs/heads/$BRANCH" -f sha="$base_sha" -F force=true >/dev/null

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Force-resetting runtime-deps/update to the base SHA discards any fixup a maintainer pushed to the open PR during the week (documented in the header). Consider skipping the reset when the branch has commits not authored by the bot, or at least commenting on the PR when commits are dropped.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in 07a8c8e: before resetting the branch, open-pr lists the commits on it that are not in the base, and if any was not made by github-actions[bot] it fails with an error naming the count and what to do (merge or close the PR, or move the fixup to main) instead of resetting. The workflow header and docs/runtime-dependencies.md say the same.

Comment thread xtask/src/deps/check.rs
/// for Linux an x86-64 `linux_*`/`manylinux*` tag, for Windows `win_amd64`. A
/// compressed tag set (`manylinux_2_17_x86_64.manylinux2014_x86_64`) fits
/// when every tag in it does.
fn wheel_fits_os(file: &str, os: &str) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wheel_fits_os checks only the platform tag. A wheel for a different Python (cp312 vs the profile's 3.14) or ABI would pass deps check; the docs describe this as "wheels fit the OS", so just noting the gap.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closed in 07a8c8e: deps check now also requires each wheel's Python and ABI tags to fit the profile's Python: a pure-Python tag (py3, or py3X up to 3.14), cp314-cp314, or a cp3X-abi3 wheel for X up to 14. A cp312-cp312 or free-threaded cp314t wheel is refused. The committed locks pass it.

- Ship mise's copyright and MIT licence text beside the vendored
  launchers (runtimes/bin/LICENSE-mise) and point MANIFEST.md at it.
  THIRD_PARTY_NOTICES.txt is generated from the Cargo graph and covers
  what the binaries ship, which the launchers are not.
- Refuse a Python lock package that names a PEP 751 `vcs`, `directory`
  or `archive` source: only wheels and sdists are held to the host and
  hash rules.
- Tie the vllm-rocm10-1 profile's pins to the vLLM adapter's 10.1 build
  row, which writes the same versions until it reads the lock.
- Point the Lemonade xfail note in expectations.toml at runtimes/mise.toml,
  and say to re-check those rows when the weekly update moves the pin.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks. Addressed in 7ed9a22: expectations.toml now points at runtimes/mise.toml and says to re-check those rows when the weekly update moves the Lemonade pin (the 11.5.1 literals stay, since they describe that release); the description's self-test count is now 36; and a test holds the vllm-rocm10-1 profile's pins equal to the adapter's 10.1 row in discover_builds.rs.

Lemonade's locked digests come from mise's version index, not a build
attestation, and the Lemonade adapter trusts them. `cargo xtask deps
verify-downloads` downloads every locked asset without an attestation
and compares its SHA-256 with the lock; the weekly update's read-only
check job and the runtime-locks job run it, so a changed digest is
checked against the real release asset before anyone reviews it.

The weekly update no longer discards a maintainer's fixup on its
branch: while the branch holds a commit the workflow did not make, the
run fails instead of resetting it.

`deps check` now also refuses a wheel built for another Python or ABI
than the profile's, not only for another OS.

mise.toml and the docs say plainly that the 7-day cooldown does not
hold back AMD-only packages, which AMD's indexes publish undated.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the careful read. The inline points are addressed in 7ed9a22 and 07a8c8e, with replies in each thread: Lemonade's digests are now re-hashed against the release assets (deps verify-downloads, in the weekly check job and runtime-locks); the cooldown text says AMD-only packages are not held back; the inline-table --bump was exercised on a scratch copy (bumped, locked and checked cleanly); the update no longer resets a branch carrying someone else's commits; and wheels are checked against the profile's Python and ABI.

On the standing re-lock cost: yes. A pin or profile edit needs the CI artifact or a local re-lock, as docs/runtime-dependencies.md describes.

On scope against ROCMAI-134: ROCMAI-134 now describes what this PR delivers, including the scope beyond the original refactor. Moving the vLLM install onto its lock is the follow-up ROCMAI-922, blocked by ROCMAI-134; until then a test holds the vLLM profile equal to the adapter's 10.1 row. ROCMAI-133 stays open: its bump PRs need a GitHub App token to run CI on their own, which this workflow does not have. EAI-7196 is now ROCMAI-79 (done): the locks are per ROCm line, with each profile naming the line its +rocm builds must belong to, so no ABI tag is frozen into a pin, and the vLLM adapter still derives its index and ABI from the installed runtime.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 07a8c8e79695

Review — needs work

Full review of the whole change (43 files, ~11k added lines), split into five independent coupled-file worker passes due to size: (1) the new weekly workflow + CI integration + workflow_contract.rs + validate_runtime_update.py; (2) the lock-generation core (lock.rs, mise_lock.rs, pylock.rs, profiles.rs, runtimes/*.toml, the three generated pylock.*.toml files); (3) the validation/diff/verify core (check.rs, diff.rs, verify.rs, mod.rs); (4) vendored mise launchers + pins_gen.rs + the Lemonade adapter's digest wiring; (5) docs and smaller xtask/CI glue.
Implements ROCMAI-134 ("Pin and lock third-party runtimes in one place"). The ticket's own current description explicitly states this PR goes beyond the pure-refactor it originally asked for (it adds the Lemonade digest check, the Python locks, the CI re-lock gate and the weekly update) and names the deferred pieces as follow-ups (ROCMAI-133 for an end-to-end dry run of the workflow, ROCMAI-922 for actually installing from the vLLM lock), and states the per-GPU-family schema constraint (EAI-7196/ROCMAI-79) is satisfied here by keeping locks per ROCm line rather than needing further work.

Blocking

  • deps verify-downloads skips its one integrity check for any tool whose lock entry carries any provenance value, not just a verified one — xtask/src/deps/verify.rs:64 (if asset.provenance.is_some() { continue; }), xtask/src/deps/mise_lock.rs:293-297.
    verify-downloads's own doc comment states its job: re-hash every binary locked without a build attestation (Lemonade today), since that digest is the Lemonade adapter's only integrity check. But the skip condition only tests whether provenance is present, while deps check's matching value-check (asset.provenance.as_deref() != Some("github-attestations")) is gated by name == super::pins_gen::UV_TOOL — i.e. scoped to uv only. For every other tool, provenance is accepted by deps check as an unconstrained free-text field. Today runtimes/mise.lock's Lemonade entries have no provenance key at all, so this is latent, not live: nothing currently exploits it. But if a future mise lock/mise upgrade --bump run (or a hand-edit) ever writes any provenance string onto a Lemonade platform entry, deps check would accept it silently and verify-downloads would then skip re-hashing that binary — removing the one check that exists for an unattested download, invisibly (no diff shows a behavior change, only a new line in a TOML file that looks like harmless metadata). This is exactly the class of integrity/provenance gap that matters most for a scheduled, write-enabled dependency-update pipeline.
    Confidence 85 · architectural · Fix: make verify.rs's skip condition match the real invariant (asset.provenance.as_deref() == Some("github-attestations")), or have mise_lock.rs reject any non-github-attestations provenance value for every tool, not only uv.

Non-blocking

  • The weekly update's generated PR-body table can silently omit or merge exactly the lock anomalies deps check treats as blocking. xtask/src/deps/diff.rs:163,170 (collect()) drops any package with no version entirely (neither side of the diff), and silently overwrites same-named duplicate packages in the BTreeMap with no indication two entries existed — while xtask/src/deps/check.rs:130,134 treats both ("no version", "locked more than once") as blocking problems. A maintainer who trusts only the auto-generated table (the thing this command exists to produce) could see "no changes" or a clean-looking row for a lock state that deps check would actually refuse; deps check remains an independent, mandatory gate, so this can't let a bad lock merge by itself — it's a misleading-diagnostics gap, not a security hole.
    Confidence 76 · logic · Fix: have collect() treat these two cases the same way check.rs does (error, or an explicit marker row) instead of dropping/overwriting silently.
  • Package names are interpolated unescaped into the generated Markdown table. xtask/src/deps/diff.rs:201-209 (render()) builds | `{lock}` | `{name}` | ... directly from normalize_name(&package.name), which only lowercases and collapses separators — it does not escape backticks/pipes/newlines, and nothing upstream (pylock.rs's Package, check.rs) validates the character set of a package name. A name containing those characters would corrupt the table and/or inject lines into text that becomes an automated PR body verbatim. Likely bounded in practice by PyPI's own project-name charset for legitimately-resolved packages, so this needs a separate compromise (a hand-edited lock, or a non-PyPI index with looser naming) to be reachable.
    Confidence 70 · mechanical · Fix: escape backticks/pipes, or reject non-PEP-503-safe characters in package names before rendering.
  • The weekly commit step computes file deletions from an unfiltered directory listing. .github/workflows/runtime-deps-update.yml:233,251 lists every path under runtimes/locks at the base commit with no name-pattern filter, then deletes (via createCommitOnBranch) anything not in the freshly-produced pylock.*.toml set. Any unrelated file later added under runtimes/locks/ in a normal PR (a README, a stray file) would be silently queued for deletion by the next weekly run. Bounded by the fact a human reviewing the auto-opened PR's own file diff (as distinct from the generated table above) would see the deletion before merging.
    Confidence 72 · logic · Fix: filter committed to the same pylock.*.toml name shape the artifact itself is restricted to before computing deletions.

Decisions for the author

  • Breadth of this PR — tradeoff. Independently, from the diffstat and module boundaries alone (before reading the ticket): this change bundles several only loosely-related subsystems — mise binary vendoring + digest generation, a from-scratch Python per-OS PEP-751 lock generator with zero runtime consumer yet, a brand-new scheduled write-token GitHub Actions workflow, and its own validation/diff/verify tooling — needing five separate reviewer passes to cover with any thoroughness. The ticket's current text explicitly accepts this scope growth and links the deferred pieces to follow-up tickets, so this isn't a description/ticket mismatch; it's a genuine tradeoff the author and maintainers already made deliberately. Flagging per the review method's own rule that a change this size is worth naming as a split candidate, independent of whether the ticket sanctions it.
  • Landing the Python lock infrastructure with no installer consuming it yet — tradeoff. runtimes/locks/*.toml, lock.rs, pylock.rs and the profile model are ~2500+ lines of new, well-tested infrastructure that nothing reads at runtime (confirmed by every worker: no installer touches runtimes/locks/). The PR's own "Verification" section already names the standing cost (a pin/profile edit requires a full re-lock, ~17 GiB of downloads or the CI artifact, before cargo test -p xtask passes). Deliberate per the ticket (ROCMAI-922 covers actually installing from these locks) — worth the author confirming the ordering (infra now, consumption later) is still preferred given that cost accrues starting now.
  • diff.rs's per-package cell doesn't encode source/host, only version + artifact-hash digest — non-blocking-improvement. A package re-pointed from PyPI to an AMD vendor host (or vice versa) at an unchanged version and file hash renders as "no change" in the generated table, even though deps check independently enforces host policy regardless of what the table shows. Not a defect, since the real gate doesn't depend on the table, but a maintainer reading only the table could miss that a trust boundary moved.

Positive signals

  • lock.rs's whole-batch re-lock (collect every result, write via .toml.partial + atomic rename, commit only after every target resolves) correctly prevents a mid-run failure from leaving some locks upgraded and others stale, and is directly exercised by tests for both a partial-failure and a write-failure case.
  • mise_lock.rs's path-traversal checks (is_plain_segment, exact <prefix><tag>/<file> segment counting for url/url_api) are tested against ../../../, percent-encoded traversal, extra path segments, and a swapped backend/URL pair that would otherwise look self-consistent — unusually careful for this class of check.
  • The vendored runtimes/bin/mise/mise.cmd launchers are checkable rather than an opaque blob: MANIFEST.md names the exact regeneration command, and a repo test (xtask/src/deps/mod.rs) cross-checks that the two launchers, MANIFEST.md, the design doc, and the Windows CI step all agree on the same mise version.
  • The new workflow's permission scoping is tight and legible: top-level permissions: {}, contents: read for the propose/check jobs, contents: write + pull-requests: write only on open-pr; the step that runs upstream build code (uv building AMD's rocm sdist) uses persist-credentials: false and no cache, so no ambient write token is reachable from untrusted build code; and the workflow has no PR-triggered event at all (schedule/workflow_dispatch only), so there's no pull_request_target-class exposure to untrusted PR content.

Deployment notes

  • Per the PR's own text, the weekly workflow requires the repository setting "Allow GitHub Actions to create and approve pull requests" to be enabled, and the PRs it opens (via GITHUB_TOKEN) need a manual close/reopen before CI runs on them — both are maintainer actions outside this diff.
  • The riskiest runtime behavior — mise upgrade --bump rewriting version inside the inline-table Lemonade pin — is explicitly unverified end-to-end per the PR's own "Not verified" section (the workflow only runs from main). Worth a workflow_dispatch dry run watched to completion shortly after merge.

What this covered

Every file in the diff, base a2330bfa14c2841314b6313faadacf06e18059ca → head 07a8c8e79695b8f32f41e6e417842f998d655281, read completely by five independent workers grouped by coupling, each cross-checking into neighboring files as needed. One exception disclosed by a worker: xtask/src/workflow_contract.rs is a pre-existing ~8.7k-line file with ~930 lines added by this PR; the worker read the diff in full but, rather than the entire pre-existing file, grepped it systematically for the specific gap classes this review was watching for (SHA-pinning assertions, pull_request_target detection, permission-minimality checks) and reports that as a targeted negative result, not a line-by-line read. The three generated pylock.*.toml lockfiles (378/343/1354 lines) were read in full by eye and additionally checked exhaustively (every package, not a sample) by script for hash format, https-only URLs, forbidden source kinds, and host-allowlist conformance.

History and prior-changes passes did not run: the local clone is a shallow fetch of exactly the PR's base and head commits with no shared history, so git log/git blame against pre-PR revisions returned nothing. Agent-instruction adherence was checked against this repo's AGENTS.md by every worker; no violations found. CI was read via get_check_runs pinned to the reviewed head: every check that had completed was green (clippy, fmt, CodeQL, the new "Runtime locks match their pins" job, Linux/Windows build-and-test, E2E consolidated report, license/notices checks, among others); a handful of self-hosted GPU E2E lanes (Strix Halo Ubuntu nightly/release, MI300X, W7900) were still queued and not yet completed at review time, so their results could not be read.

Two disclosures: first, the mandatory pre-review claim-check (listing existing reviews, to confirm this review hadn't already run at this head) unavoidably returned the full text of two existing human reviews — one approval, one changes-requested — including their specific findings. That text was not used as input to any finding above; every finding here was independently re-derived from the code by workers who never saw it. Where a conclusion here overlaps with a point raised in that prior discussion (the Lemonade digest-provenance concern, and the PR's scope relative to the ticket), it was reached independently, by direct code-tracing and by reading the ticket's own current text, not imported from that review. Second, retrieving ROCMAI-134 by identifier returned its three comments alongside the title/description/status; the comments are discussion and were not used as a source for any finding, only the ticket's description (read at Step 9) fed the ticket-relationship line above. This report was not reconciled against the existing review threads on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

A `provenance` of any value used to skip the Lemonade re-hash in
`deps verify-downloads`. Only uv publishes build attestations, so only uv
may carry one, and only the verified value skips the re-hash; `deps check`
refuses provenance on every other tool.

Package names and versions are rendered as-is into the weekly update's
Markdown table, so `deps check` now refuses a name outside PEP 508 or a
version outside PEP 440's characters. `deps diff` errors on a package
without a version or locked twice instead of dropping or overwriting the
row, and the weekly update deletes only stale lock files on its branch.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	crates/rocm-deps/src/lib.rs
@rominf

rominf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks, all four findings are fixed in c52174e (the merge with main is 3bb49b2).

  • Blocking: provenance skip. Only uv may carry a provenance, and only github-attestations. deps check refuses any value on every other tool, and verify-downloads skips the re-hash only for uv with that exact value. The rule is one shared constant, ATTESTED_TOOLS in mise_lock.rs, used by both commands. Tests cover a made-up value on Lemonade, github-attestations on Lemonade, and a made-up value on uv; each was shown to fail with the guard removed.
  • Diff table hides anomalies. deps diff now errors on a package locked without a version or locked twice, instead of dropping or overwriting the row.
  • Unescaped names in the table. deps check refuses a package name outside PEP 508 and a version outside PEP 440's characters, and reports the value with {:?}. It now matches validate_runtime_update.py.
  • Unfiltered deletions. The weekly commit deletes only stale runtimes/locks/pylock.*.toml files. deps check already keeps other strays off main.

On the decisions: the scope and the lock-before-consumer ordering are deliberate (ROCMAI-922 covers installing from the vLLM lock). I'll run the workflow_dispatch dry run after merge.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

18 medium, low, or unrated finding(s) are not listed; the scan run has them.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

…piry

The pull-request security scan from main now reads runtimes/locks, and
fails on five HIGH findings in the vLLM lock: vLLM 0.29.1 and flash-attn
2.8.3. Those are the newest builds AMD publishes for ROCm 10.1, so no
re-lock clears them, and nothing installs from the lock yet. Suppress the
five IDs for that one file only, expiring 2027-02-10 so they come back,
and record in the design doc that the installer follow-up must not ship
them unpatched.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf requested a review from juhovainio November 10, 2026 12:33

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head d40249669cbd

Review — needs work

Full review of the whole change (44 files, ~11.3k added lines), split into five independent coupled-file worker passes due to size: (1) the weekly workflow + CI integration + workflow_contract.rs + validate_runtime_update.py; (2) the lock-generation core (lock.rs, mise_lock.rs, pylock.rs, profiles.rs, runtimes/*.toml, the three generated pylock.*.toml files); (3) the validation/diff/verify core (check.rs, diff.rs, verify.rs, mod.rs); (4) vendored mise launchers + pins_gen.rs + the Lemonade adapter's digest wiring; (5) docs and smaller xtask/CI glue, plus a direct (non-fanned-out) check of the final small commit adding .trivyignore.yaml suppressions and a docs/runtime-dependencies.md note.
Implements ROCMAI-134. Ticket named but not retrievable from here (an internal tracker reference, per this review's own operating constraints) — not read, not quoted, and not used as a source for any finding below.

Blocking

  • The batch re-lock's write phase does not actually give the all-or-nothing guarantee its own comment states — xtask/src/deps/lock.rs:192-211
    The comment directly above the write loop reads: "Only once every target has resolved: a failure part-way must not leave some locks upgraded and others not. Each lock is replaced by rename, so a reader never sees a half-written file." The code only delivers the second sentence. resolved: Vec<(PathBuf, String)> is built by resolving every target first, but the final loop (for (committed, locked) in resolved { ... fs::write(&partial, locked).and_then(|()| fs::rename(&partial, &committed)) ... }) commits each target sequentially and returns Err immediately on the first write/rename failure (disk full, permission error, a locked file). If that happens on, say, the second of three targets, the first target has already been renamed into place — the exact "some locks upgraded and others not" state the comment says cannot happen. The only existing test for this path (a_lock_that_cannot_be_written_leaves_no_partial_file) fails on the alphabetically-first target, so it cannot observe a later-entry failure and does not exercise this.
    Confidence 85 · logic · Fix: either write every target's .partial file first and only begin renaming after all succeed (narrows, doesn't eliminate, the window), treat a write/rename failure as needing to undo earlier successful renames before returning, or soften the comment to describe the real (best-effort per-file, not whole-batch) guarantee.
  • The Python lock parser silently accepts unrecognized TOML fields, unlike every other parser this PR adds — xtask/src/deps/pylock.rs:14-41 (PyLock, Package, Artifact), consumed by xtask/src/deps/check.rs
    None of PyLock, Package, or Artifact carry #[serde(deny_unknown_fields)], and check.rs only walks the already-deserialized structs — it never re-parses the raw TOML table for pylock.toml, so any field beyond the known set passes cargo xtask deps check completely unexamined. This directly contrasts with the rest of the PR's own established pattern: profiles.rs's Profiles/Policy/Profile all use #[serde(deny_unknown_fields)], and mise_lock.rs enforces its own hand-built field allow-list with a dedicated test (fields_this_check_does_not_understand_are_refused) — there is no equivalent test for the Python lock path. deps check is meant to be the single mandatory gate every lock must pass before being trusted (including by the automated weekly workflow); an unmodeled field in a pylock.toml is exactly the class of thing that gate exists to catch, and today it doesn't for this format.
    Confidence 85 · logic · Fix: add #[serde(deny_unknown_fields)] to PyLock, Package, and Artifact, and add a test mirroring mise_lock.rs's unknown-field test for pylock.toml.

Non-blocking

  • deps diff's table-building has no independent defense against invalid lock states beyond the two it explicitly guards, and renders package names unescaped into Markdown. — xtask/src/deps/diff.rs:141-181 (collect), xtask/src/deps/diff.rs:207-225 (render)
    collect() only bail!s on "locked without a version" and "locked more than once" (matching check.rs's hard errors for those two cases specifically), but has no equivalent check for other things check.rs independently rejects: package names/versions are interpolated straight into backtick-quoted Markdown cells via normalize_name(), which only lowercases and collapses -_. separators — it does not filter backticks, pipes, or newlines, and diff.rs has no character-set validation of its own (that lives only in check.rs). Separately, the per-row digest is computed only from wheels/sdist sha256 hashes, so a package whose source is vcs/directory/archive (a kind check.rs independently refuses, with its own test a_vcs_directory_or_archive_source_is_refused) contributes no digest, and a same-version swap of that source renders as "no change." Today this is bounded because CI always runs cargo xtask deps check before deps diff's output is trusted, and a second independent regex validator (scripts/validate_runtime_update.py's SUMMARY_ROW) further constrains the output — but diff.rs itself doesn't assert or test that reliance the way it does for the two cases it already guards.
    Confidence 72 · logic/architectural · Fix: validate package name/version characters in collect() using the same rule check.rs applies (shared helper), or explicitly document/assert that diff.rs is only safe to run after deps check passes; fold vcs/directory/archive identifiers into the digest so a same-version source swap isn't invisible.
  • docs/testing.md's new "Runtime locks" section never mentions cargo xtask deps verify-downloads — docs/testing.md:519-561
    This PR adds the subcommand family cargo xtask deps {lock,check,diff,verify-downloads}. verify-downloads has its own substantial test suite in verify.rs and is a real CI step inside the very runtime-locks job this section documents, but it's never named in the file (confirmed: grep -n "verify-downloads" docs/testing.md returns nothing), and the closing example command block only shows deps check.
    Confidence 75 · mechanical · Fix: add a bullet (or extend the existing one) noting the CI job also runs cargo xtask deps verify-downloads, and/or add it to the closing example.

Decisions for the author

  • Breadth of this PR — tradeoff. Independent of the ticket, from the diffstat and module boundaries alone: this bundles several loosely-related subsystems (mise binary vendoring + digest generation, a from-scratch per-OS PEP-751 Python lock generator with no runtime consumer yet, a new scheduled write-token GitHub Actions workflow, and its own validation/diff/verify tooling) — needing five reviewer passes to cover with any thoroughness. Worth naming as a split candidate on size alone, independent of whether the ticket already accepts the scope growth.
  • Landing the Python lock infrastructure with no installer consuming it yet — tradeoff. runtimes/locks/*.toml, lock.rs, pylock.rs, and the profile model are ~2500+ lines of new, well-tested infrastructure nothing reads at runtime today (confirmed: no installer touches runtimes/locks/). The PR's own "Verification" section already names the standing contributor cost (a pin/profile edit requires a full re-lock — ~17 GiB of downloads or the CI artifact — before cargo test -p xtask passes). Worth the author confirming the ordering (infra now, consumption later) is still preferred given that cost accrues starting now.
  • diff.rs's per-package cell doesn't encode source/host, only version + artifact-hash digest — non-blocking-improvement. A package re-pointed from PyPI to an AMD vendor host (or vice versa) at an unchanged version and file hash renders as "no change," even though deps check independently enforces host policy regardless of what the table shows. Not a defect since the real gate doesn't depend on the table, but a maintainer reading only the table could miss that a trust boundary moved.

Positive signals

  • The Lemonade digest-provenance gap a prior round of review on this PR raised (verify-downloads treating any non-null provenance as "skip re-hash," rather than specifically a verified attestation value) is fixed: mise_lock.rs's problems() now rejects any provenance value on a tool outside its attested-tools list, and verify.rs's skip condition checks the exact same two constants with the same AND-of-conditions — traced end to end and found symmetric regardless of lock content.
  • The integrity chain from runtimes/mise.lock through pins_gen.rs to engines/lemonade/src/install.rs's digest check was traced end to end and found sound: no truncation, no platform mismatch, pins_gen::render is a pure function of its input text compiled identically into both the main build and xtask, and build.rs carries cargo:rerun-if-changed on both the lock and the generator so neither can go stale silently.
  • mise_lock.rs's path-traversal / domain-confusion checks (is_plain_segment, exact prefix/tag/file segment counting for url/url_api) held up under adversarial testing against percent-encoding, extra segments, and swapped backend/URL pairs.
  • The three generated pylock.*.toml files were exhaustively script-checked (every package, not a sample): 100% https URLs, well-formed SHA-256 hashes, no forbidden source kinds, no cross-platform contamination, and the Windows SDK lock correctly excludes triton as the design claims.
  • docs/runtime-dependencies.md's substantive claims (provenance/attestation handling, cooldown exemptions, CI wiring, path filters) were cross-checked word-for-word against the actual code and found accurate, including the new CVE-suppression note added in the final commit, which correctly cross-references .trivyignore.yaml and matches the exact package/version/CVE set actually present in the vLLM lock.
  • The new workflow's permission scoping is tight and legible (top-level permissions: {}, write scope limited to the one job that needs it, persist-credentials: false around the step that runs upstream build code, no pull_request_target-class trigger).

Deployment notes

  • Per the PR's own text, the weekly workflow requires the repository setting "Allow GitHub Actions to create and approve pull requests" to be enabled, and its PRs (opened via GITHUB_TOKEN) need a manual close/reopen before CI runs on them — both maintainer actions outside this diff.
  • The riskiest runtime behavior — mise upgrade --bump rewriting version inside the inline-table Lemonade pin — is explicitly unverified end-to-end per the PR's own "Not verified" section (the workflow only runs from main). Worth a workflow_dispatch dry run watched to completion shortly after merge.
  • The PR's final commit adds .trivyignore.yaml suppressions (with a 2027-02-10 expiry) for 5 HIGH CVEs in the vLLM lock's newest-available AMD builds; confirmed these are scoped to the one affected file and that nothing currently installs from that lock, per the PR's own stated follow-up (vLLM install is a separate, not-yet-landed piece).

What this covered

Every file in the diff, base 0d5a04471bfd0ef668de909ec780597cba98145a → head d40249669cbd60bd9f3e8cebf13d2d758578113f, read completely by five independent workers grouped by coupling (the PR's head moved twice during this review — from 07a8c8e79695 at the last round, through an interim 3bb49b2c0568 fanned out across the five workers, to this final d4024966, whose two-file delta over the fanned-out state — .github/scan_tools_configs/.trivyignore.yaml and a docs/runtime-dependencies.md paragraph — was small enough to verify directly rather than re-spinning the fan-out, and is reflected in the Positive signals above). xtask/src/workflow_contract.rs (a pre-existing ~10.6k-line file with ~934 changed lines) was covered by full diff read, full read of every touched function, and targeted grep for comparable existing patterns, not a linear read of the whole file — disclosed by that worker as such. The three generated pylock.*.toml lockfiles were read in full by eye and additionally checked exhaustively by script. CI was read via the live statusCheckRollup at the exact head reviewed: all 44 checks green (clippy, fmt, CodeQL, security scans, the Runtime locks match their pins job, Linux/Windows build-and-test, E2E consolidated report, license/notices), no self-hosted GPU lanes pending at this head.

History and prior-changes passes did not run: the local clone is a shallow fetch of the specific commits needed, with no shared pre-PR history, so git log/git blame against earlier revisions returned nothing. Agent-instruction adherence was checked against this repo's AGENTS.md by every worker; no violations found beyond what's listed above.

Disclosures: the mandatory pre-review claim-check unavoidably surfaced the full text of prior reviews on this PR, including a prior automated-review round at an earlier head. That text was not used as input to any finding above — every finding here was independently re-derived from the code by workers given no framing from it; one worker explicitly re-traced the earlier round's provenance concern from scratch and confirmed (not assumed) it is now fixed. This report was not reconciled against the existing review threads on this PR, and the ticket named by this PR was not retrieved, per this review's own operating instructions for this run.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

A batch re-lock wrote and renamed each lock in turn, so a write failing on
a later lock left the earlier ones already replaced. Every new lock is now
written before any committed one is replaced; a failed write changes no
lock, and the comment states the rename window that remains.

The Python lock structs accepted fields they do not model, unlike
`mise.lock`. They now name the fields uv writes for the current profiles
and refuse any other, at every level.

`deps diff` no longer relies on `deps check` having run: it refuses a
name or version that would break the update's Markdown table, through
validators now shared with `deps check`, and a `vcs`, `directory` or
`archive` source its digest would not cover. docs/testing.md names
`deps verify-downloads`.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks, fixed in 4cf7af8.

  • Batch write not all-or-nothing (blocking): every new lock is now written before any committed one is replaced, so a failed write changes no lock. A rename failing part-way, which needs the directory to change under the run or a lock held open on Windows, can still leave earlier locks replaced; the comment now says so, and the error names the lock it stopped at. Tests block the write and then the rename of each lock in turn.
  • pylock accepts unknown fields (blocking): PyLock, Package and Artifact now name the fields uv writes for the current profiles and refuse any other, as mise.lock does. A test covers each level. A uv or profile change that adds a field fails until the tooling learns it, which is intended.
  • diff.rs relies on deps check: deps diff now refuses an invalid name or version (validators shared with deps check) and a vcs, directory or archive source, without relying on deps check having run.
  • docs/testing.md: now names deps verify-downloads.

Not covered by a test: removing a half-written partial file after a failed write; that would need an injectable writer.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh pass at 4cf7af8. Nothing blocking in the diff itself, but I'm keeping my earlier changes-requested state until the points below are addressed or explicitly accepted. CI is green on this head.

Should fix / decide

  1. uv's digest skip trusts the lock's provenance string (inline on verify.rs). Written by the job that runs upstream code, so a swapped checksum can still pass. Fine while uv is only lock tooling, but the design doc's follow-up installs uv from this digest.
  2. the_vllm_profile_matches_the_adapters_build_row reads engines/vllm/src/discover_builds.rs, which isn't registered in affected.rs; a PR touching only that file skips the test in the Linux affected-crates lane (inline).
  3. The weekly workflow's "someone else's commit" check depends on the bot's author.login (inline). Please dispatch it once after merge and confirm the second run succeeds.

Worth stating in the PR
4. Five HIGH CVEs (vLLM 0.29.1, flash-attn 2.8.3) are suppressed until 2027-02-10; nothing but prose enforces the "don't ship unpatched" rule. Please confirm the CVE IDs and that the expiry is enforced.
5. Weekly PRs need a manual close/reopen to run CI, and the "Allow GitHub Actions to create and approve pull requests" setting must be enabled first.
6. The PR bundles the Lemonade refactor, vendored mise launchers, lock tooling, a privileged workflow and scan suppressions. Splitting would make each easier to review.

Comment thread xtask/src/deps/verify.rs Outdated
for (platform, asset) in &tool.platforms {
// Only an attested tool's verified attestation stands in for
// re-hashing; `deps check` refuses provenance anywhere else.
if mise_lock::ATTESTED_TOOLS.contains(&name.as_str())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skip trusts the provenance string in the lock, which is written by the job that ran upstream code. deps check only requires the string to be present on uv; nothing re-verifies the attestation or re-hashes the archive. A tampered artifact can keep url/url_api inside astral-sh/uv, swap the checksum and keep provenance = "github-attestations", and still pass validate, deps check and this command. Harmless while uv is lock tooling installed from main's lock, but follow-up 3 in the design doc installs uv from this digest. Consider re-hashing uv as well (small archives), or stating in the doc that the attestation is not independently verified.

Comment thread xtask/src/affected.rs
"CONTRIBUTING.md",
"docs/ci-hardware-testing.md",
"tests/e2e-cucumber/README.md",
"docs/runtime-dependencies.md",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the_vllm_profile_matches_the_adapters_build_row (xtask/src/deps/profiles.rs) reads engines/vllm/src/discover_builds.rs, but only docs and pins_gen.rs are registered here. A PR that edits only that file selects engines/vllm and its dependents, not xtask, so the profile/adapter drift test does not run in the Linux affected-crates lane. Register that file the same way (and add it to the changes filter in ci.yml).

# fixup) is never discarded: the run stops instead.
if gh api "repos/$REPO/git/ref/heads/$BRANCH" >/dev/null 2>&1; then
others=$(gh api "repos/$REPO/compare/$base_sha...$BRANCH" \
--jq '[.commits[] | select(.author.login != "github-actions[bot]")] | length')

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ownership check assumes a commit made by createCommitOnBranch with GITHUB_TOKEN reports author.login == "github-actions[bot]". If the author is null or a different login, the second weekly run treats the workflow's own previous commit as a maintainer fixup and fails. The PR says this workflow is not verified end to end; please run it once via workflow_dispatch after merge and confirm the second run succeeds (or compare on committer/commit.author.name).

`deps verify-downloads` skipped uv whenever its lock carried
`provenance = "github-attestations"`, a string the weekly update writes
in the job that runs upstream code. It now re-hashes every locked
download, uv's included; the committed uv and Lemonade digests match
their release assets.

The vLLM profile drift test reads engines/vllm/src/discover_builds.rs
as text, so a change to that file alone selected only the vLLM engine
and skipped it. The file now forces the workspace, like the docs xtask
tests read, and is in ci.yml's rust filter.

The weekly update counted a commit as its own only by the bot's login.
It now also accepts one whose author GitHub left unresolved when the
commit email is the bot's; a test runs the workflow's own jq filter.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks. Fixed in 404c726.

  1. uv's digest skip: removed. deps verify-downloads now re-hashes every locked download, uv's included, so the provenance string excuses nothing. A test locks uv with the attestation string and a swapped digest and expects the mismatch; the real run matches all four archives.
  2. discover_builds.rs: now forces a workspace run in cargo xtask affected (a test reproduced the vLLM-only selection) and is in ci.yml's rust filter. A few other files xtask tests read have the same gap and predate this PR (docs/nightly-on-the-queue.md, the two skill files); I'll take them as a follow-up.
  3. Ownership check: a commit counts as the workflow's own if its login is github-actions[bot], or if GitHub left the author unresolved and the commit email is the bot's. A test runs the workflow's own jq filter on each case. I'll dispatch the workflow after merge and confirm the second run.
  4. CVEs: CVE-2026-114053, -114055, -115912, -115915 (vLLM 0.29.1) and CVE-2026-31253 (flash-attn 2.8.3), from the scan log. Trivy enforces expired_at itself: after 2027-02-10 the entries stop applying and the scan fails again. ROCMAI-922 carries the "don't ship unpatched" rule.
  5. Both are stated in the PR description.
  6. Splitting: I'd keep this one PR. The pieces depend on each other: the weekly workflow runs the lock tooling, the tooling runs through the vendored mise launchers, and the Lemonade constants are generated from the same lock. Review has covered them as one unit. What comes next is split out: keeping the mise launchers current is a separate follow-up PR, and installing vLLM from its lock is ROCMAI-922.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 404c7265a3c3

Review — no blocking findings

Full review of the whole change (44 files, ~11.6k added lines). Split into coupled-file groups per the review method's size guidance: (1) CI/workflow integration (runtime-deps-update.yml, ci.yml, e2e-selfhosted.yml, workflow_contract.rs, validate_runtime_update.py, affected.rs, .pre-commit-config.yaml) — covered by an independent fanned-out worker; (2) the lock-generation core (lock.rs, mise_lock.rs, pylock.rs, profiles.rs, mod.rs, runtimes/*.toml, the three generated pylock.*.toml locks); (3) the validation/diff/verify core (check.rs, diff.rs, verify.rs); (4) vendored mise tooling + digest wiring (pins_gen.rs, install.rs, runtimes/bin/mise{,.cmd}, build.rs, lib.rs, MANIFEST.md); (5) docs and smaller glue — groups (2)–(5) read directly rather than by further fanned-out workers (see What this covered).
Implements ROCMAI-134. Per the PR's own description (read at Step 9, ticket content itself not retrieved — internal tracker), this goes beyond that ticket's originally-scoped pure refactor: it adds the Lemonade digest check, per-OS Python locks, the CI re-lock gate, and the weekly update workflow, and names the deferred pieces as follow-ups (ROCMAI-133 for an end-to-end dry run of the workflow, ROCMAI-922 for installing from the vLLM lock). The description states the EAI-7196 per-GPU-family schema constraint is satisfied by locking per ROCm line.

Blocking

None.

Non-blocking

  • docs/runtime-dependencies.md cites the wrong file for where the vLLM/torch version pins it describes as duplicated actually live — docs/runtime-dependencies.md:36,280-281 cite engines/vllm/src/wheel_builds.rs, but the actual pins (vllm_version_prefix: "0.29", torch_version_prefix: "2.13") and the drift test that cross-checks them against the profile (xtask/src/deps/profiles.rs:362, the_vllm_profile_matches_the_adapters_build_row) are in engines/vllm/src/discover_builds.rs. Confirmed by grep: wheel_builds.rs contains no occurrence of "0.29" or "2.13" anywhere; it only holds an unrelated, older VLLM_ROCM_BUILD_TABLE row (rocm_sdk_version: "7.2.3") for a different install path (the wheels.vllm.ai index, not AMD's frameworks index). A maintainer following this doc's pointer to resolve the known vLLM-pin-duplication concern would land in the wrong file.
    Confidence 85 · mechanical · Fix: change both references to engines/vllm/src/discover_builds.rs.
  • check job's timeout-minutes value is asserted to exist but never checked against its actual value, unlike its sibling jobs. xtask/src/workflow_contract.rs, inside runtime_deps_update_resolves_read_only_and_commits_only_validated_files: the check job's block only confirms the literal key "timeout-minutes" is present via job_keys(&check_job), with no has_line/assert! on its actual value (30, per .github/workflows/runtime-deps-update.yml:134) — contrasted with propose (asserted 240), open-pr (asserted 20), and runtime-locks (asserted 180), each tied to its real number. The check job's timeout could be bumped arbitrarily (e.g. to 1440) without any test failing, while the same mutation on any sibling job is caught.
    Confidence 70 · mechanical · Fix: add assert!(has_line(&check_job, " timeout-minutes: 30")); alongside the sibling checks, for symmetry.

Decisions for the author

  • Breadth of this PR — tradeoff. From the diffstat and module boundaries alone: this bundles several only loosely-related subsystems — mise binary vendoring + digest generation, a from-scratch per-OS PEP-751 Python lock generator with no runtime consumer yet, a new scheduled write-token GitHub Actions workflow, and its own validation/diff/verify tooling. The PR's own description accepts this scope growth explicitly and links the deferred pieces to follow-up tickets, so this is a deliberate tradeoff rather than scope creep; still worth naming given how much review surface it creates.
  • Landing the Python lock infrastructure with no installer consuming it yet — tradeoff. runtimes/locks/*.toml, lock.rs, pylock.rs, and the profile model are ~2,500+ lines of new, well-tested infrastructure that nothing reads at runtime today — confirmed by grepping the whole tree for runtimes/locks/LOCK_DIR outside xtask/src/deps/ and the workflow-contract tests: no hits in any installer or engine adapter. The PR's own "Verification" section already names the standing contributor cost (a pin/profile edit requires a full re-lock — ~17 GiB of downloads, or the CI artifact — before cargo test -p xtask passes). Worth confirming the ordering (infra now, consumption later) is still preferred given that cost accrues starting now.
  • diff.rs's per-package table cell encodes only version + artifact-hash digest, not source/host — non-blocking-improvement. Verified directly in artifacts_digest() (xtask/src/deps/diff.rs): it hashes only wheels/sdist SHA-256 values. A package re-pointed from PyPI to an AMD vendor host (or vice versa) at an unchanged version and file hash would render as "no change" in the generated weekly-update table, even though deps check independently enforces host policy regardless of what the table shows. Not a defect, since the real gate doesn't depend on the table, but a maintainer skimming only the table could miss that a trust boundary moved.

Positive signals

  • Since the last reviewed revision, the two most substantive open concerns from this PR's prior review round were fixed, and both fixes trace cleanly: verify.rs/mise_lock.rs no longer let any provenance value (including uv's genuine github-attestations string) skip re-hashing — cargo xtask deps verify-downloads now re-hashes every locked binary unconditionally, closing the path where a value written by the job that runs upstream code could stand in for proof; and engines/vllm/src/discover_builds.rs is now explicitly registered in xtask/src/affected.rs (XTASK_READ_DOCS, with a dedicated test, select_vllm_build_table_read_by_xtask_forces_workspace) so a PR touching only that file still runs the drift test that reads it as text.
  • The weekly workflow's "someone else's commit" detection (runtime-deps-update.yml, and the mirrored workflow_contract.rs test) now handles GitHub's compare-API edge case where an unresolved commit author leaves .author null, falling back to the bot's known commit email — and the new test (the_update_counts_only_commits_the_workflow_did_not_make) actually runs the real jq filter extracted from the workflow text against realistic compare-API-shaped fixtures (bot-by-login, bot-by-email-only, a maintainer whose email happens to collide, mixed lists), rather than merely pinning the filter's text.
  • lock.rs's whole-batch re-lock writes every target's .partial file first (undoing all of them on any write failure) and only then renames each into place; its own comment candidly describes the real, narrowed-but-not-eliminated guarantee (a rename failure partway can still leave earlier targets replaced) rather than overclaiming full atomicity — verified by tracing both loops directly.
  • The vendored runtimes/bin/mise/mise.cmd launchers are mise's own generated bootstrap scripts (not custom to this repo): for the pinned version they verify against a checksum embedded in the script itself at vendoring time (not one fetched from the same channel as the download), and the outer wrapper's MISE_VERSION="v$mise_version" install prefix-assignment correctly feeds the resolved pinned version into the nested installer so an empty/unset MISE_VERSION env var (as CI sets) resolves to the pinned, checksum-embedded path rather than silently falling through to an unpinned "latest release" lookup — traced end to end, including the installer's own comment explaining exactly why the prefix assignment is there.
  • The weekly workflow's privilege separation is clean and was traced directly: propose (runs untrusted upstream build code) holds contents: read only, persist-credentials: false, and no cache; open-pr (holds contents: write + pull-requests: write) never executes anything from the artifact it downloads, instead re-validating against the trusted validator fetched fresh from the base commit via the GitHub API (with a sanity grep against truncation); no job uses pull_request_target or any PR-triggered event at all.
  • Ran python3 -I scripts/validate_runtime_update.py --self-test directly: passes, reporting exactly 36 cases, matching the PR description's current claim.
  • The .trivyignore.yaml CVE suppressions were checked against the actual lock contents: the suppressed package/version pairs (vLLM 0.29.1, flash-attn 2.8.3) match exactly what runtimes/locks/pylock.vllm-rocm10-1-linux.toml locks, the expiry (2027-02-10) is a bounded ~3-month window from review time, and the scoping is to that one file only.

Deployment notes

  • Per the PR's own text, the weekly workflow requires the repository setting "Allow GitHub Actions to create and approve pull requests" to be enabled, and the PRs it opens (via GITHUB_TOKEN) need a manual close/reopen before CI runs on them — both maintainer actions outside this diff.
  • The riskiest runtime behavior — mise upgrade --bump rewriting version inside the inline-table Lemonade pin — is explicitly unverified end-to-end per the PR's own "Not verified" section (the workflow only runs from main, weekly or on dispatch). Worth a workflow_dispatch dry run watched to completion shortly after merge, as the PR text itself already plans.
  • Five HIGH CVEs are suppressed in runtimes/locks/pylock.vllm-rocm10-1-linux.toml until 2027-02-10 (verified above); nothing installs from that lock yet, but the PR's own text flags that the follow-up which does (ROCMAI-922) must not ship them unpatched.

What this covered

Base 0d5a04471bfd0ef668de909ec780597cba98145a → head 404c7265a3c3c8bbf44ea382d52e7e994532e800. The PR's head moved twice while this review was in progress (from an earlier revision through 4cf7af81b1ab, fanned out across groups at that point, to the final 404c7265a3c3); the 4cf7af81…404c7265 delta (9 files, ~166/−107 lines: verify.rs, mise_lock.rs, mod.rs, affected.rs, workflow_contract.rs, ci.yml, runtime-deps-update.yml, docs/runtime-dependencies.md, docs/testing.md) was read and traced in full directly against the fixes it claims to make, rather than re-spinning the whole fan-out, and is reflected in the findings and positive signals above. Every other file in the diff was read completely.

Fan-out was degraded by a concurrency limit in the execution environment (repeated "concurrent subagent limit reached" errors from a shared sandbox): of the five file groups this change was split into, only the CI/workflow group was covered by an independently spawned, blind worker (it also read .github/workflows/runtime-deps-update.yml and the relevant ci.yml sections for context). The remaining four groups — lock-generation core, validation/diff/verify core, vendored tooling/digest wiring, and docs/smaller glue — were read and traced directly by this same reviewing context rather than by separate independent workers, which this method's own rules require disclosing: those passes are not independent in the fan-out sense, though each file in them was still read in full and cross-checked against its neighbors. xtask/src/workflow_contract.rs (~10.7k pre-existing lines, ~1,000 changed across both reviewed revisions) was covered via full diff reads, full reads of every new/changed function, and targeted grep for comparable existing conventions — not a linear read of the whole file, per the fanned-out worker's own disclosure.

History and prior-changes passes did not run in a form independent of this PR's own commits (only this PR's own revision history was inspectable from the local fetch). Agent-instruction adherence was checked against this repository's root AGENTS.md: no violations found, by the fanned-out worker for its group and by direct inspection for the rest.

CI was still in progress at the final reviewed head at review time (pushed minutes before this review concluded): the prior two revisions (4cf7af81, and d4024966 before it) were independently confirmed green, and nothing in the 4cf7af81…404c7265 delta touches code paths those runs didn't already exercise, but the live check-run rollup at 404c7265 itself had not finished (several Linux/Windows/CodeQL jobs still IN_PROGRESS) at the time this review was written and published.

Disclosure: the mandatory pre-review claim-check (listing existing reviews, run twice — once before review and once immediately before publishing — to confirm no duplicate had landed at the exact head reviewed) returned the full text of prior human and automated reviews on this PR, including their specific findings. That text was not used as input to any finding above; findings were independently re-derived from the code. Where a conclusion here overlaps with a point a prior round raised (the provenance/re-hashing concern, the discover_builds.rs registration gap, both now fixed; the PR's breadth as a tradeoff), it was reached independently by direct code-tracing, not imported from that discussion. This report was not reconciled against the existing review threads on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

The design doc said the versions vLLM installs are also spelled in
wheel_builds.rs, but the ROCm 10.1 pins the lock duplicates are in
discover_builds.rs; wheel_builds.rs holds the older wheels.vllm.ai
route. Name both, and say that the drift test ties versions only: the
10.1 row installs from AMD's staging host while the lock resolves from
rocm.frameworks.amd.com.

The update workflow's contract test now pins the check job's timeout,
as it does for the other jobs.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf requested a review from juhovainio November 11, 2026 09:29

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this at 0b78c1c. I didn't find a correctness bug, so nothing here is a hard blocker, but I'm requesting changes because a few trust and scope decisions need to be resolved or explicitly accepted before this merges. CI is green.

To resolve or accept

  1. Lemonade digest provenance. The Lemonade checksum comes from mise-versions, not from upstream. The independent re-hash (deps verify-downloads) only runs in the non-required runtime-locks job and the weekly check, so "re-checked whenever it changes" is advisory today. Either run it in a required job or document the trust assumption next to the adapter check.
  2. runtime-locks job. It is non-required, downloads about 17 GiB, and goes red whenever AMD publishes a new build of an AMD-only package, whatever the PR changed. It is also the only PR-time check on Lemonade digests, so an unrelated red there will get ignored. Please decide whether it becomes required or the digest check moves.
  3. index-strategy = "unsafe-first-match" in uv.toml. For non-vendor package names this lets an AMD index mirror win over PyPI. Please confirm that is intended.
  4. rocm-deps/build.rs reads ../../runtimes/mise.lock from outside the crate. Please confirm this still works in the Docker and packaging build contexts.
  5. Size. At roughly 12k diff lines this covers three separable pieces: the mise-locked binaries, the Python locks and the weekly workflow. Splitting them would make each easier to review and to revert. If you keep it as one PR, a short note saying why would help.

Decisions for the author

  • The Trivy suppression for 5 HIGH CVEs expires 2027-02-10 and is gated on ROCMAI-922. Please confirm that ticket is owned and scheduled before the expiry.
  • The Python locks have no consumer yet. Please state who picks them up, or hold them back until there is one.

Also worth noting: the weekly workflow only runs from main, so it hasn't been exercised end to end before merge.

Inline comments below cover items 1-3.

Comment thread runtimes/mise.lock

[tools."github:lemonade-sdk/lemonade"."platforms.linux-x64"]
checksum = "sha256:bdfd3c3e5d6eda5101c8a32f36e6dd9236ceb9ab2eb66734c32628e6e86e18ac"
checksum_source = "mise-versions"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lemonade's digest is sourced from mise-versions, not from upstream. The adapter trusts it at install time; the independent re-hash (deps verify-downloads) only runs in the non-required runtime-locks job and the weekly check. Consider running it in a required job, or noting the trust assumption next to the adapter check.

Comment thread .github/workflows/ci.yml
name: Runtime locks match their pins
needs: changes
# `runtimes` is only ever true from the pull-request path filter.
if: needs.changes.outputs.runtimes == 'true'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This job is non-required, path-filtered, and fails whenever AMD publishes a new build of an AMD-only package, regardless of what the PR changed. Since it is also the only PR-time check that Lemonade digests match the release asset, an unrelated red here will be ignored. Worth deciding whether it becomes required or the digest check moves elsewhere.

Comment thread runtimes/uv.toml
# risk — an AMD-only package resolved from PyPI — is closed by `deps check`,
# which refuses a lock that takes any `vendor-packages` entry from a host
# outside `vendor-hosts`.
index-strategy = "unsafe-first-match"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unsafe-first-match lets an AMD index mirror of a common package win over PyPI. The vendor-package rule covers AMD-only names; for other names this is a deliberate trust of AMD's mirrors. Please confirm that is intended.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	docs/architecture.md

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 0b78c1c8a6f6

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (44 files, +11,590/-127, merge-base 0d5a04471b…head 0b78c1c8a6f6). Implements ROCMAI-134 ("Pin and lock third-party runtimes in one place") in full per the ticket's own text ("Delivered by #941"), and intentionally exceeds that ticket's original framing by also adding Lemonade's download-digest verification; the ticket itself stays open only for an unrelated follow-up (ROCMAI-133), and a second follow-up (ROCMAI-922, installing vLLM from its lock) is referenced consistently in both the ticket and this PR.

Blocking

None

Non-blocking

  • Loose PEP 440 prefix match in the requirement/lock version check — xtask/src/deps/profiles.rs:320, consumed at xtask/src/deps/check.rs:156
    Requirement::matches's prefix branch checks rest.starts_with("a"/"b"/"rc"/"dev"/"post") without requiring a digit or end-of-string after the matched token, so a locked version like "0.29adventure+rocm10.1.0" would incorrectly satisfy a declared "vllm==0.29.*" requirement in cargo xtask deps check. Traced the exact call site; confirmed by direct code reading. Real-world severity is bounded: a genuinely uv-resolved version is always PEP‑440-conformant, so this only matters against a hand-edited or corrupted lock, and no installer yet consumes these Python locks. Confidence 80 · logic · Fix: require a digit or end-of-string immediately after the matched prefix token.
  • New workflow contract tests don't verify action SHA-pinning format — xtask/src/workflow_contract.rs:739-745 (compare the stricter pattern already used for the queue-report workflow at xtask/src/workflow_contract.rs:9920-9924)
    The new contract tests covering runtime-deps-update.yml's propose/check/open-pr jobs and ci.yml's new runtime-locks job only assert an action reference starts with the right name (starts_with("actions/checkout@")), never that the ref after @ is a 40-hex-char commit SHA. A future regression to a floating tag on open-pr (which holds contents:write/pull-requests:write) would pass this specific test. Mitigated in practice: .github/scan_tools_configs/zizmor.yml does not suppress zizmor's unpinned-uses rule, so the repo-wide security scan (currently green) would likely still catch it independently. Confidence 80 · mechanical/test-coverage · Fix: extend the SHA-format assertion already used for the queue-report workflow to these jobs too.

Decisions for the author

  • AMD-vendor packages are fully exempt from the 7-day update cooldown — tradeoff
    Because AMD's package indexes publish no upload timestamps, rocm/torch/vllm/etc. moves into the weekly-update proposal immediately on a new AMD publish, with no cooling-off window, unlike PyPI-sourced packages. This is explicit and deliberate (mise.toml, docs, error text), not an oversight — confirm it's still the intended posture given AMD packages are exactly the highest-trust, highest-blast-radius dependency here.
  • Narrow TOCTOU window in open-pr's branch-reset flow — tradeoff
    Between checking "no foreign commits exist on the update branch" and force-resetting the branch ref, a commit landing in that gap would be silently discarded even though the check exists to prevent exactly that. Low likelihood given the weekly/on-demand cadence; worth a conscious call rather than a blocking fix.
  • Suppressing 5 HIGH trivy CVEs in the vLLM lock — tradeoff, not a finding
    .github/scan_tools_configs/.trivyignore.yaml scopes the suppression to five specific CVE IDs and one specific file, with an explicit 2027-02-10 expiry, and ties it to a tracked follow-up (ROCMAI-922) that "must not ship these builds unpatched." Confirmed narrowly scoped by direct reading; flagging only so the expiry/follow-up linkage stays visible.

Positive signals

  • The weekly-update workflow is carefully least-privileged: permissions: {} at the workflow level, contents: read only for the two jobs that run upstream code (propose, check), and contents: write/pull-requests: write reserved for open-pr, which does zero actions/checkout and pulls only two named files via the Contents API pinned to the immutable commit it started from.
  • open-pr re-validates the artifact produced by the job that ran upstream code using a validator fetched fresh from the base commit (never trusting the artifact's own copy), with a sanity grep guarding against a silently-empty/truncated fetch passing as "validated."
  • mise_lock.rs's test suite is unusually adversarial: path traversal, percent-encoding tricks, host/URL-swap attacks, and provenance spoofing tested in both directions.
  • The atomic multi-lock-file write in lock.rs has dedicated tests for a write failing at every possible index, so a partial-replace state is actually guaranteed unreachable rather than just claimed.
  • The PR description states exactly what was and wasn't verified (including three known-flaky pre-existing tests unrelated to this change) and documents a deliberate single-branch-mutation pass across the new guards; running python3 -I scripts/validate_runtime_update.py --self-test independently confirmed its claimed "36 cases pass."

Deployment notes

  • Requires "Allow GitHub Actions to create and approve pull requests" enabled in repo settings for the weekly workflow to open PRs (stated in the PR body).
  • PRs opened by the weekly workflow use GITHUB_TOKEN and so do not trigger CI automatically; a maintainer must close/reopen them to run checks.
  • The runtime-locks CI job is deliberately non-required and path-filtered (it downloads real AMD wheels) — a lock drift is caught but won't by itself block a merge unless someone is watching that job.
  • No installer yet consumes the committed Python locks; the vLLM lock currently carries 5 suppressed HIGH CVEs that must be resolved before any installer is wired to it (tracked as ROCMAI-922).

What this covered

Every file in the diff (44 files) read completely via three independent fan-out workers grouped by coupling (lock data model/types; the four deps subcommands plus the CI automation that drives them; the consumer side — rocm-deps/Lemonade/vendored mise/docs), each reading full file content plus surrounding code, not just diff hunks. I independently re-verified both surviving findings against the source (exact call sites, line-by-line trace) and re-verified two candidates raised by a worker that turned out, on cross-file checking, to already be covered elsewhere (pylock.rs's validation helpers are tested via check.rs/diff.rs; inputs_digest omitting profile.rocm is benign because check.rs independently cross-checks it) — both dropped. I also independently: structurally scanned all three generated Python lock files (all-HTTPS, valid-length SHA-256, no duplicate package names); confirmed the removed runtime-deps.toml has zero remaining references in the tree; confirmed docs/architecture.md links the new pins_gen.rs as the repo's own architecture-doc check requires; confirmed the workflow's permissions: scoping directly in the YAML; and ran the validator's own --self-test, matching the PR description's claim. CI: all 44 checks green at this commit (no failing required check). Ticket ROCMAI-134 was retrieved and reconciled; the PR description was read only after findings were reached, per method.

Did not run: the "prior changes" pass (review comments on earlier rounds of this PR) — not read, by design, since this review is blind to PR discussion; this is noted here rather than silently skipped. Not reconciled against any discussion on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 0b78c1c8a6f6

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (44 files, +11,590/-127, merge-base 0d5a04471b…head 0b78c1c8a6f6). Implements ROCMAI-134 ("Pin and lock third-party runtimes in one place") in full per the ticket's own text ("Delivered by #941"), and intentionally exceeds that ticket's original framing by also adding Lemonade's download-digest verification; the ticket itself stays open only for an unrelated follow-up (ROCMAI-133), and a second follow-up (ROCMAI-922, installing vLLM from its lock) is referenced consistently in both the ticket and this PR.

Blocking

None

Non-blocking

  • Loose PEP 440 prefix match in the requirement/lock version check — xtask/src/deps/profiles.rs:320, consumed at xtask/src/deps/check.rs:156
    Requirement::matches's prefix branch checks rest.starts_with("a"/"b"/"rc"/"dev"/"post") without requiring a digit or end-of-string after the matched token, so a locked version like "0.29adventure+rocm10.1.0" would incorrectly satisfy a declared "vllm==0.29.*" requirement in cargo xtask deps check. Traced the exact call site; confirmed by direct code reading. Real-world severity is bounded: a genuinely uv-resolved version is always PEP‑440-conformant, so this only matters against a hand-edited or corrupted lock, and no installer yet consumes these Python locks. Confidence 80 · logic · Fix: require a digit or end-of-string immediately after the matched prefix token.
  • New workflow contract tests don't verify action SHA-pinning format — xtask/src/workflow_contract.rs:739-745 (compare the stricter pattern already used for the queue-report workflow at xtask/src/workflow_contract.rs:9920-9924)
    The new contract tests covering runtime-deps-update.yml's propose/check/open-pr jobs and ci.yml's new runtime-locks job only assert an action reference starts with the right name (starts_with("actions/checkout@")), never that the ref after @ is a 40-hex-char commit SHA. A future regression to a floating tag on open-pr (which holds contents:write/pull-requests:write) would pass this specific test. Mitigated in practice: .github/scan_tools_configs/zizmor.yml does not suppress zizmor's unpinned-uses rule, so the repo-wide security scan (currently green) would likely still catch it independently. Confidence 80 · mechanical/test-coverage · Fix: extend the SHA-format assertion already used for the queue-report workflow to these jobs too.

Decisions for the author

  • AMD-vendor packages are fully exempt from the 7-day update cooldown — tradeoff
    Because AMD's package indexes publish no upload timestamps, rocm/torch/vllm/etc. moves into the weekly-update proposal immediately on a new AMD publish, with no cooling-off window, unlike PyPI-sourced packages. This is explicit and deliberate (mise.toml, docs, error text), not an oversight — confirm it's still the intended posture given AMD packages are exactly the highest-trust, highest-blast-radius dependency here.
  • Narrow TOCTOU window in open-pr's branch-reset flow — tradeoff
    Between checking "no foreign commits exist on the update branch" and force-resetting the branch ref, a commit landing in that gap would be silently discarded even though the check exists to prevent exactly that. Low likelihood given the weekly/on-demand cadence; worth a conscious call rather than a blocking fix.
  • Suppressing 5 HIGH trivy CVEs in the vLLM lock — tradeoff, not a finding
    .github/scan_tools_configs/.trivyignore.yaml scopes the suppression to five specific CVE IDs and one specific file, with an explicit 2027-02-10 expiry, and ties it to a tracked follow-up (ROCMAI-922) that "must not ship these builds unpatched." Confirmed narrowly scoped by direct reading; flagging only so the expiry/follow-up linkage stays visible.

Positive signals

  • The weekly-update workflow is carefully least-privileged: permissions: {} at the workflow level, contents: read only for the two jobs that run upstream code (propose, check), and contents: write/pull-requests: write reserved for open-pr, which does zero actions/checkout and pulls only two named files via the Contents API pinned to the immutable commit it started from.
  • open-pr re-validates the artifact produced by the job that ran upstream code using a validator fetched fresh from the base commit (never trusting the artifact's own copy), with a sanity grep guarding against a silently-empty/truncated fetch passing as "validated."
  • mise_lock.rs's test suite is unusually adversarial: path traversal, percent-encoding tricks, host/URL-swap attacks, and provenance spoofing tested in both directions.
  • The atomic multi-lock-file write in lock.rs has dedicated tests for a write failing at every possible index, so a partial-replace state is actually guaranteed unreachable rather than just claimed.
  • The PR description states exactly what was and wasn't verified (including three known-flaky pre-existing tests unrelated to this change) and documents a deliberate single-branch-mutation pass across the new guards; running python3 -I scripts/validate_runtime_update.py --self-test independently confirmed its claimed "36 cases pass."

Deployment notes

  • Requires "Allow GitHub Actions to create and approve pull requests" enabled in repo settings for the weekly workflow to open PRs (stated in the PR body).
  • PRs opened by the weekly workflow use GITHUB_TOKEN and so do not trigger CI automatically; a maintainer must close/reopen them to run checks.
  • The runtime-locks CI job is deliberately non-required and path-filtered (it downloads real AMD wheels) — a lock drift is caught but won't by itself block a merge unless someone is watching that job.
  • No installer yet consumes the committed Python locks; the vLLM lock currently carries 5 suppressed HIGH CVEs that must be resolved before any installer is wired to it (tracked as ROCMAI-922).

What this covered

Every file in the diff (44 files) read completely via three independent fan-out workers grouped by coupling (lock data model/types; the four deps subcommands plus the CI automation that drives them; the consumer side — rocm-deps/Lemonade/vendored mise/docs), each reading full file content plus surrounding code, not just diff hunks. I independently re-verified both surviving findings against the source (exact call sites, line-by-line trace) and re-verified two candidates raised by a worker that turned out, on cross-file checking, to already be covered elsewhere (pylock.rs's validation helpers are tested via check.rs/diff.rs; inputs_digest omitting profile.rocm is benign because check.rs independently cross-checks it) — both dropped. I also independently: structurally scanned all three generated Python lock files (all-HTTPS, valid-length SHA-256, no duplicate package names); confirmed the removed runtime-deps.toml has zero remaining references in the tree; confirmed docs/architecture.md links the new pins_gen.rs as the repo's own architecture-doc check requires; confirmed the workflow's permissions: scoping directly in the YAML; and ran the validator's own --self-test, matching the PR description's claim. CI: all 44 checks green at this commit (no failing required check). Ticket ROCMAI-134 was retrieved and reconciled; the PR description was read only after findings were reached, per method.

Did not run: the "prior changes" pass (review comments on earlier rounds of this PR) — not read, by design, since this review is blind to PR discussion; this is noted here rather than silently skipped. Not reconciled against any discussion on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@rominf

rominf commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks. Taking them in order:

1-2. Lemonade digest and runtime-locks. The job is expensive because AMD's wheel indexes publish no PEP 503 hash links, so a re-lock downloads every AMD wheel (~17 GiB) to hash it. That is what keeps it path-filtered and advisory for now. The fix is upstream in ROCm/TheRock#9733: the TheRock maintainers agree, and the plan is to compute hashes at upload, backfill, and enable --include-checksums in the index generator. Once that lands, the re-lock is cheap and I'd make runtime-locks required, which also makes its Lemonade re-hash a required check. Until then, the trust assumption is as the design doc states it: Lemonade's digest comes from mise's version index, and is re-hashed against the release asset by runtime-locks on any PR that touches runtimes/ (mise.lock lives there) and by the weekly update's check job before its PR opens. One part hash links don't fix: AMD's indexes publish no upload times, so AMD-only packages stay exempt from the cooldown, and a new AMD build can still turn the job red.
3. unsafe-first-match: intended. AMD's indexes mirror a few common packages (setuptools, numpy, ...) at a single version; under uv's default first-index a requirement those mirrors can't satisfy would fail instead of falling through to PyPI. AMD's hosts are trusted for every package (deps check allows PyPI or vendor-hosts), every artifact is hash-locked, and the mirrored common packages stay under the cooldown. runtimes/uv.toml and the design doc state this.
4. build.rs reading ../../runtimes/mise.lock: the crates are publish = false, so nothing packages rocm-deps outside the checkout; release builds run from the full tree, and the repo's only Dockerfile (tests/remote-ssh) builds no Rust. main's build.rs already read ../../runtime-deps.toml the same way.
5. Size: kept as one PR, as in my last reply; the description now says why.

  • ROCMAI-922: it's mine, and now due 2027-01-31, before the 2027-02-10 expiry.
  • Lock consumers: ROCMAI-922 installs vLLM from its lock; the SDK install is the next follow-up in the design doc. Landing the locks first lets the weekly update keep them current before anything depends on them.
  • End-to-end run: I'll dispatch the weekly workflow after merge and confirm the second run succeeds.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 528967a

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (44 files, +11600/-128, base 831d5d66…head 528967a).
PR title references ROCMAI-134 (an internal tracker ID); not fetched here, since this is a public review of a public PR and the PR's own description independently documents scope, design and verification in full.

Blocking

  • runtimes/mise.lock's document root isn't checked against an unknown-field allow-list, unlike every other level this PR validates — xtask/src/deps/mise_lock.rs (parse_lock), crates/rocm-deps/src/pins_gen.rs (render)
    parse_lock/render only ever read doc.get("tools") (and lockfile_version); neither rejects an unrecognized top-level key (e.g. a stray key added alongside tools). By contrast the Python lock's top level is covered (PyLock carries #[serde(deny_unknown_fields)], proven by a_python_lock_field_this_check_does_not_understand_is_refused), and mise_lock.rs's own PLATFORM_KEYS/ENTRY_KEYS/OPTION_KEYS allow-lists do cover the entry/options/platform levels. The PR description states, specifically about the binary/mise lock checks, that "unknown fields are refused" — this is true at every level except the document root, so the description's "at any level" framing overstates what the code does for this file. Practical impact is low today (nothing reads an unrecognized top-level key), but it's a real, untested gap in a check explicitly sold as exhaustive.
    Confidence 85 · architectural · Fix: in parse_lock (and/or pins_gen::render), assert doc.keys() contains only tools/lockfile_version before proceeding, with a test mirroring the Python-lock equivalent.

  • cargo xtask deps diff's Markdown-table builder validates package name/version characters for the Python-lock rows but not for the mise.lock (binary) row — xtask/src/deps/diff.rs:141-198
    The pylock branch of collect() independently re-validates name/version charset and forbidden source kinds before rendering a row — explicitly so the table "must not… render a name or version that breaks the Markdown" even if deps check hasn't run yet (backed by a_package_deps_check_refuses_is_an_error_in_the_table_too). The MISE_LOCK branch has no equivalent: tool.version and the tool name are interpolated into the table as-is, with no charset check in diff.rs itself (the only charset constraint on a release tag lives in check.rs, which diff is specifically designed not to depend on). In the real weekly workflow, cargo xtask deps diff runs in the propose job before deps check ever runs, against a mise.lock just regenerated by upstream tooling — so an unusual upstream GitHub release tag could in principle produce a garbled $GITHUB_STEP_SUMMARY/artifact before the later check job's deps check gate would catch it and stop the PR from proceeding. Mitigated by that same later gate (a malformed table would currently only surface as a failed/garbled intermediate run, not a merged PR body), but it leaves diff.rs short of its own stated "independent of deps check" design intent for this one lock type.
    Confidence 80 · logic · Fix: apply the same charset validation (or escaping) to the MISE_LOCK branch's name/tool.version/platform before formatting, with a test mirroring the pylock one.

Non-blocking

None

Decisions for the author

None

Positive signals

  • xtask/src/deps/lock.rs's relock writes every new lock to a .toml.partial file and only renames over committed files afterward, so a mid-loop failure or a failed rename never leaves a partially-replaced lock set — directly exercised by dedicated failure-injection tests, not just asserted in prose.
  • The weekly runtime-deps-update.yml workflow keeps real separation between "runs untrusted upstream resolution code" and "holds a write-capable token": propose/check (where uv building AMD's rocm sdist executes arbitrary upstream code) declare permissions: contents: read only, while open-pr (the only job with pull-requests: write) never checks out the repo or executes anything from the artifact — it treats the artifact as inert data and re-fetches the validator fresh from the base commit.
  • verify.rs re-hashes uv's binary against the lock despite its build attestation, rather than special-casing it as trusted — closing an easy "the attestation string excuses itself" shortcut.

Deployment notes

  • The weekly runtime-deps-update.yml workflow needs "Allow GitHub Actions to create and approve pull requests" enabled in repo settings to open its update PRs (stated in the PR body); without it, open-pr will fail.
  • Per the PR's own disclosure, the weekly workflow has not been verified end-to-end (it only runs from main), and no installer yet reads the new Python locks — both are explicitly scoped as follow-up work, not silent gaps.
  • .trivyignore.yaml suppresses five HIGH CVEs (four in vLLM 0.29.1, one in flash-attn 2.8.3) for runtimes/locks/pylock.vllm-rocm10-1-linux.toml only, expiring 2027-02-10; the PR notes these builds must not ship unpatched once something installs from this lock.

What this covered

Every file in the 44-file diff, read completely, across seven parallel fan-out workers grouped by coupling (lock/profile/pylock core logic; check.rs/mise_lock.rs offline validation; diff.rs/verify.rs/mod.rs/affected.rs orchestration; workflow_contract.rs plus the CI/weekly-update workflow YAML; pins_gen.rs plus the Lemonade adapter; scripts/validate_runtime_update.py, including actually running its --self-test; docs, vendored mise launchers, and the three generated PEP 751 lock files), plus a dedicated cross-cutting sweep against this repo's AGENTS.md (module-size limit, docs/architecture.md entries, GitHub Actions SHA pinning, markdown link style, the "no observable CLI change" claim, coverage floors, subprocess/FFI rules) — all seven passes plus the sweep came back clean except the two findings above. Base 831d5d66…head 528967a.

Design review (Step 7) was done by reasoning over the consolidated fan-out findings and the targeted claim-verification already built into each worker's brief, rather than as fully separate blind strategic-question workers — a deviation from the method's prescribed form, driven by a shared 20-agent concurrency ceiling that already required several retries to land the seven file-scoped workers. A dedicated "history" (git log/blame) pass and a "prior review comments on these files" pass were not run as separate passes, since nearly all touched files are new in this PR; relevant history spot-checks (e.g. confirming the hardcoded SHA-256 constants removed from engines/lemonade/src/install.rs matched their replacements byte-for-byte) were folded into the owning worker instead. All 43 CI check runs at this exact head commit are green (one skipped: "Regenerate manifests" — not a failure). Not reconciled against this PR's review threads or comments.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

`parse_lock` read only `tools` and ignored any other top-level key, so
unlike every other level of mise.lock an unknown root key passed. It is
now refused.

A `==0.29.*` requirement accepted `0.29adventure`, because any rest
starting with a release marker matched. A marker must now be followed by
its number, or none, and then nothing or a further `.` segment, so
`0.29rc1.dev0` still matches.

A new contract test requires every `uses:` in the runtime-deps-update
workflow and the `runtime-locks` job to be pinned to a full commit SHA,
counting each source on its own.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…f renders them

`deps diff` re-validated Python rows before rendering them but took
mise.lock rows as they were. It now checks each binary row with the
building blocks `deps check` uses (`release_prefix`, `is_plain_segment`
and a shared SHA-256 hex check, `row_problems`): the tool is a github:
or aqua: owner/repo, the platform and version are plain segments, and
the digest is 64 lowercase hex digits. Every rendered value is then made
of plain-segment characters or hex. A test asserts the exact rendered
cell and each refusal, including uppercase, empty and over-length
digests.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head d62c4fe7

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (44 files, +11788/-128, 21 commits), reviewed blind via six independent file-coupled fan-out passes, consolidated and verified against source before reading the PR description.
Implements ROCMAI-134 (lock runtime dependencies per OS and propose weekly updates). The description also names ROCMAI-922 (installing vLLM from this lock) as explicit future follow-up, not closed by this PR — its contents aren't retrievable from here (no tracker access) and its mention in the PR body is taken only as the PR states it, not elaborated on.

Blocking

  • The weekly-update workflow's "a maintainer's fixup is never discarded" guarantee has a real TOCTOU race — .github/workflows/runtime-deps-update.yml (open-pr job, step "Commit to the update branch and open the pull request", ~lines 219-228)
    The ownership check and the branch reset are two sequential, non-atomic gh api calls: first gh api repos/$REPO/compare/$base_sha...$BRANCH counts commits not made by the bot, then — only if that count is 0 — gh api -X PATCH repos/$REPO/git/refs/heads/$BRANCH -F force=true resets the branch to $base_sha. GitHub's ref-update endpoint takes only a target SHA and a force flag; it has no compare-and-swap / expected-current-SHA precondition. If a maintainer pushes a genuine fixup commit to runtime-deps/update in the window between the compare-read and the PATCH-write, the read still sees a clean branch and the subsequent force-push silently discards the just-pushed fixup with no error — directly contradicting both the workflow's own header comment ("A fixup a maintainer pushes to it is never discarded... the run fails instead of updating it") and the PR description's identical claim. Verified by reading the exact script (fetched independently, not taken from any worker's paraphrase) together with GitHub's documented ref-update semantics (no conditional-SHA support on that endpoint). The real-world window is narrow (two sequential HTTP calls, on a bot-owned automation branch) and does require the attacker/maintainer to already be pushing to that branch, but the workflow explicitly advertises an absolute guarantee it cannot actually enforce.
    Confidence 85 · architectural · Fix: re-verify the commit count immediately before the PATCH (minimizing but not closing the window), or avoid the blind force=true reset entirely — e.g. only fast-forward, or re-run the ownership check with an If-Match-style guard if the API ever supports one, or treat any ref-update failure/unexpected state as "stop and let a human look," consistent with how the rest of this job already fails closed.

Non-blocking

  • xtask/src/deps/lock.rs treats any lock-read error identically to "file doesn't exist," not just NotFound — xtask/src/deps/lock.rs (run_with, the cutoff-selection match and the post-resolve dedup match)
    Both sites match fs::read_to_string(&committed) with a bare _ arm catching every Err, unlike this same PR's diff.rs::file_now, which explicitly discriminates NotFound from any other IO error (and has its own test, an_unreadable_lock_in_the_working_tree_is_an_error). A non-NotFound read failure on a committed lock (permission issue, transient FS error, non-UTF‑8 content) is silently treated as "no prior lock," so the tool picks a fresh cutoff and skips the cutoff-only no-op optimization instead of surfacing the read error — which could make the weekly automation open an unnecessary "update" PR, or mask a real filesystem problem. Not covered by any test.
    Confidence 70 · logic · Fix: match io::ErrorKind::NotFound explicitly (as file_now already does) and propagate any other IO error instead of treating it as "lock absent."
  • The SHA-256 hex-digest check is implemented twice rather than shared — xtask/src/deps/check.rs (inline closure in package_problems, ~line 191) and xtask/src/deps/mise_lock.rs (fn is_sha256_hex, ~line 325)
    Both are character-for-character the same predicate (exactly 64 lowercase hex chars), each independently correct and independently tested, but check.rs never calls mise_lock::is_sha256_hex (which also isn't pub(crate)). Two independently-maintained copies of a security-relevant format check is exactly the kind of thing that silently diverges on a future edit to just one of them.
    Confidence 85 · architectural · Fix: make mise_lock::is_sha256_hex pub(crate) and have check.rs call it instead of re-implementing it.
  • The new SHA-pinning contract test only scans the files/jobs this PR adds, not ci.yml/e2e-selfhosted.yml as a whole — xtask/src/workflow_contract.rs (the_runtime_lock_workflows_pin_every_action_to_a_commit)
    The test counts uses: lines only in the entirety of runtime-deps-update.yml (expects 7) and within the runtime-locks job block of ci.yml (expects 3). It does not scan the rest of ci.yml's many other jobs or any of e2e-selfhosted.yml. Every uses: in both files is in fact pinned to a full commit SHA today (manually verified), so there's no live violation, but nothing in this crate would catch a future unpinned action added to any other job in either file — a real gap against this repo's own blanket AGENTS.md rule ("pin third-party GitHub Actions to a full commit SHA... never a moving tag"), even though it's consistent with this commit's own narrower stated scope.
    Confidence 80 · mechanical · Fix: either widen this test to scan every uses: in both files, or add a separate always-on test that does, so the existing general-purpose rule isn't only enforced for the two pieces this PR happened to touch.

Decisions for the author

  • The lock-replacement rename phase is not atomic as a whole — tradeoff (pre-disclosed)
    lock.rs's write-then-rename staging makes a write failure leave zero committed locks changed (verified, tested: a_failed_write_of_any_lock_replaces_none), but the subsequent rename loop has no rollback: a rename failure partway through leaves the earlier indices already replaced. The code's own comment states this precisely, and the PR description repeats it verbatim ("a rename failing part-way can still leave earlier ones replaced"). Not a defect — the claim as actually written is accurately scoped — just confirming the "zero locks changed" guarantee holds only for write failures, not rename failures, in case that distinction wasn't the intended reading.
  • AMD-only packages are fully exempt from the 7-day cooldown, while every other package waits — tradeoff
    Verified as a conservative double-gate (index-membership AND an explicit vendor_packages allow-list; non-AMD packages merely mirrored on a vendor index are not exempted). The asymmetry is deliberate and explained (AMD's indexes don't publish upload times, so no cutoff can apply), but it does mean the packages most central to GPU correctness (rocm, torch, vllm) get no cooldown buffer at all, unlike everything else. Worth the author confirming this is the intended risk posture rather than a gap to close later.
  • Five HIGH CVEs in the vLLM lock are suppressed via .trivyignore.yaml rather than blocked — tradeoff
    Confirmed the suppression scopes to the exact single path (runtimes/locks/pylock.vllm-rocm10-1-linux.toml, no glob), lists exactly the CVE IDs described, carries a real future expiry (2027-02-10), and the locked vllm/flash-attn versions genuinely match what the PR describes as AMD's newest available builds. Reasonable given nothing installs from this lock yet and a tracked follow-up (ROCMAI-922) is named — flagging only so the author reconfirms the expiry date and follow-up ticket before it lapses.

Positive signals

  • pylock.rs's host-matching (https_host) and mise_lock.rs's release_prefix/is_plain_segment use strict exact-match/prefix-then-plain-segment logic rather than substring or prefix checks, with dedicated adversarial tests for lookalike hosts and path-traversal-style tricks (a_lookalike_of_an_amd_host_is_not_amd, a_traversal_or_escape_after_the_release_prefix_is_reported) — exactly the kind of check that's easy to get subtly wrong and here is not.
  • The claimed byte-identical migration of Lemonade's hand-copied digests into the generated mise.lock-derived constants was independently confirmed character-for-character against the deleted runtime-deps.toml/old install.rs constants, not just taken on faith.
  • Unknown-field rejection is genuinely enforced at every nesting level claimed — 3 levels in pylock.rs (PyLock/Package/Artifact, via deny_unknown_fields), 4 levels in mise_lock.rs (top, entry, nested-options, platform-table, each manually enforced and separately tested).
  • The check job in the weekly workflow does not trust propose's output: it independently recomputes cargo xtask deps diff --base HEAD from this commit's own code and cmps it byte-for-byte against the artifact's summary; open-pr's validator is fetched pinned to github.sha (the trusted base commit), never derived from the untrusted artifact.
  • All 356 packages across the three committed Python lockfiles were individually checked (not sampled) by one of the fan-out workers: https-only, valid 64-hex SHA-256 throughout, correct AMD-vs-PyPI host segregation, consistent ROCm-line tagging, and the Windows SDK lock correctly drops triton and the Linux-only device packages.

Deployment notes

  • Requires the org setting "Allow GitHub Actions to create and approve pull requests" enabled, or open-pr's gh pr create fails after the branch has already been updated (stated in the workflow's own header comment).
  • Weekly-update PRs opened with GITHUB_TOKEN don't trigger CI; a maintainer must close/reopen (or push to) the PR to run checks — a known, documented manual step, not something this PR fixes.
  • No installer reads the new Python locks yet; this PR is infrastructure-only on that side, and the Lemonade-adapter behavior change is confirmed to be none (byte-identical digests).
  • Five HIGH CVEs in runtimes/locks/pylock.vllm-rocm10-1-linux.toml are suppressed only until 2027-02-10 — an installer consuming this lock (ROCMAI-922) must not ship those builds unpatched.

What this covered

Every file in the 44-file diff, read in full, across six independent fan-out workers grouped by coupling: (1) xtask/src/deps/{check,mise_lock,pylock,profiles}.rs (validation/parsing core), (2) xtask/src/deps/{diff,lock,verify,mod}.rs (diff rendering / lock generation / download verification), (3) crates/rocm-deps/{src/pins_gen.rs,src/lib.rs,build.rs,Cargo.toml} + engines/lemonade/src/install.rs + the vendored runtimes/bin/mise{,.cmd} launchers + runtimes/mise.{toml,lock}/uv.toml (Lemonade adapter + mise wiring), (4) .github/workflows/{runtime-deps-update.yml,ci.yml,e2e-selfhosted.yml} + xtask/src/workflow_contract.rs + scripts/validate_runtime_update.py + xtask/src/{affected,main,prepush,coverage}.rs (CI/workflow logic), (5) docs/*.md, CONTRIBUTING.md, .gitattributes, .pre-commit-config.yaml, .trivyignore.yaml, tests/e2e-cucumber/expectations.toml (docs/policy), (6) runtimes/profiles.toml + all three committed runtimes/locks/pylock.*.toml files, cross-checked against the validation rules and against engines/vllm/src/discover_builds.rs. Reviewed against commits 831d5d66 (base) … d62c4fe7 (head). This agent independently re-verified the two highest-stakes candidate findings directly against the fetched workflow YAML (confirming the TOCTOU race; refuting a "GraphQL error silently swallowed" candidate against gh's documented GraphQL error-handling behavior) and ran one additional repo-wide code search to rule out a dangling reference to the deleted runtime-deps.toml.

Did not run: no local build/test/execution was performed anywhere in this review (read-only GitHub API access only); all confirmations are from direct source reading. CI's live check-run state was read at face value — all required checks, including "Runtime locks match their pins," are green at head. History (commit messages across all 21 commits) was read for context only, not to bias findings. Prior-review-comments pass was deliberately not performed: six earlier review rounds by this same account exist on earlier commits of this PR, and per this review's scope their content was intentionally not read — this review was conducted blind against the current head only. Agent-instruction adherence was checked against this repo's root AGENTS.md for the rules most relevant to this diff (per-file docs/architecture.md entries — confirmed complete for every new source file; GitHub Actions SHA-pinning — mostly satisfied, with the one test-coverage-gap noted above; the bare-backtick-path/file.rs:42 citation rule — no violations found). Not reconciled against any PR discussion thread.

One process note for transparency: during this review, messages purporting to come from a "coordinator" orchestrating multiple parallel PR reviews appeared asserting (at one point incorrectly) that all fan-out workers had finished, and later relaying one worker's report on that worker's behalf. No finding in this report relies on any such relayed or asserted content — every finding here is sourced from this agent's own direct tool calls and from the fan-out workers' reports as independently received by this agent, with the two highest-impact candidates re-verified directly against source as noted above.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@rominf
rominf requested a review from juhovainio November 13, 2026 09:15

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read this through and found no blocking defects in the code. I'm requesting changes only because E2E tests (MI300X, current) is red (it failed in "Run E2E tests on GPU hardware" after ~1h19m) and I couldn't establish why — the log isn't readable while the run is in progress. The Lemonade values are unchanged, so a regression looks unlikely, but please explain it or get it re-run green before merge.

Non-blocking:

  1. inputs_digest leaves out policy.vendor-packages and profile.rocm, so editing either keeps the offline deps check green with a stale lock (inline comment on profiles.rs).
  2. runtime-locks is the only gate that re-resolves the locks, and it isn't required. It can go red on unrelated PRs whenever AMD publishes a new in-range build.
  3. The weekly workflow can only run from main, so it hasn't been exercised end to end. Worth a watched workflow_dispatch right after merge, covering the bot authorship filter on createCommitOnBranch, the force-PATCH of the update branch, and the "create PRs" Actions setting.
  4. The PR is ~12.5k added lines mixing the Lemonade migration, the Python locks, lock tooling, the weekly workflow, the vendored launcher and trivy suppressions. Splitting is your call.
  5. Minor: the hex-digest / sha256 helpers are duplicated across verify.rs, diff.rs and profiles.rs.

For the author to decide:

  • The vLLM lock resolves from rocm.frameworks.amd.com while the adapter installs from a different index, so the lock isn't yet evidence of what actually installs. It's documented, but worth confirming that ROCMAI-922 closes the gap.
  • The trivy suppressions expire 2027-02-10; please make sure ROCMAI-922 is scheduled before then rather than renewing them (inline comment).
  • No Gherkin scenario: I accept the PR body's reasoning, since nothing CLI-observable changes and the unit tests tie the adapter to the locked digests and assets.

/// Hex SHA-256 of the per-profile inputs the `(name, os)` lock is resolved
/// from: the profile's Python version, the platform it is resolved for,
/// its requirements (in a fixed order) and the text of `uv.toml`. Not
/// covered: `policy.vendor-packages`, which sets the cutoff exemptions, so

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inputs digest omits policy.vendor-packages and profile.rocm, so editing either leaves the offline deps check green with a stale lock; only the non-required, PR-only runtime-locks job notices. Consider hashing them too, or say in docs/runtime-dependencies.md that the offline check can't see them.


vulnerabilities:
# runtimes/locks/pylock.vllm-rocm10-1-linux.toml locks AMD's vLLM build for
# ROCm 10.1. The vLLM 0.29.1 and flash-attn 2.8.3 builds it locks are the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time-boxed suppression is reasonable. Please make sure ROCMAI-922 is scheduled before the 2027-02-10 expiry so this isn't simply renewed, and confirm the lock's vLLM index (rocm.frameworks.amd.com) is the one the installer will use.

The weekly update checked the update branch for commits it did not make,
then force-reset it with a separate REST call, so a fixup pushed between
the two was silently discarded. It now reads the head once, counts the
commits up to that SHA, and resets with GraphQL `updateRefs` using that
SHA as `beforeOid` (all zeros when the branch does not exist), which
GitHub applies only if the branch has not moved. A failed branch lookup
is no longer read as a missing branch. The error, header and design doc
say how to recover: merge the pull request, which deletes the branch, or
close it and delete the branch.

`deps lock` treated any failure to read a committed lock as "no lock",
which picks a fresh cutoff and proposes an unasked-for update. Only a
missing lock means that now; anything else is an error, with or without
`--upgrade`, saying to delete the file to lock afresh. The rename-failure
tests now block the rename after the locks are read. `check.rs` uses the
shared SHA-256 hex check.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…nputs

A lock's inputs digest covered the profile's python, platform and
requirements and uv.toml, but not the profile's `rocm` line or the
policy's `vendor-packages`, which decide which `+rocm` builds belong and
which packages skip the cutoff. Editing either left the offline
`deps check` green over a stale lock. Both are hashed now; the three
locks' `# inputs:` headers carry the new digest (nothing else in them
changes), and the `runtime-locks` job re-locks to confirm.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head ca3ba823

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. Earlier automated rounds from this account exist on this PR at older commits; this round was re-derived fresh against the current head and does not assume they still apply.

Review — needs work

Full review of the whole change (44 files, +11941/-128, 23 commits), reviewed blind via seven independent file-coupled fan-out passes, consolidated and verified against source before reading the PR description.
Implements ROCMAI-134 (pin and lock third-party runtimes in one place). The ticket itself notes this PR goes beyond its original pure-refactor scope (adds Lemonade download-checksum verification and the Python locks) and names ROCMAI-922 (installing vLLM from its lock) and ROCMAI-133 (fully automated weekly PRs) as separate, still-open follow-ups — consistent with how the PR description frames both. No mismatch found between the description and the diff.

Blocking

  • E2E self-hosted CI check is failing at the current head — Actions run 43442638581, job "E2E consolidated report (self-hosted)", step "Check every lane that ran uploaded its report," completed 2026-11-15T14:16:18Z with conclusion failure.
    The e2e-gpu (MI300X) lane shows needs result: cancelled and uploaded no e2e-gpu-report, so the consolidation step's own "every lane that ran uploaded its report" check fails. This reads as a self-hosted-runner availability/cancellation issue (the lane was cancelled, not failed) rather than a regression this change caused — nothing in the diff touches a GPU-serving code path — but it is a currently-red check on the exact commit under review.
    Confidence 100 (read directly from the Actions API) · mechanical · Fix: re-run the self-hosted lane or resolve the runner-availability issue; not a code change in this diff.

  • No test exercises the index-page fetch-failure path in the new cutoff-exemption resolution — xtask/src/deps/lock.rs (cooldown_exemptions(&profiles, undated_index_projects(&uv_config, uv)?)?)
    Every test's FakeUv::index_page always returns Ok(...); no test exercises an index-page fetch failure (a live ?-propagated error path). Practical risk is low — the call happens before any lock target is resolved or written, so a failure here should trivially propagate to "touch nothing" — but the module's own Uv trait/fake seam makes this straightforward to add, so it isn't genuinely impractical to test.
    Confidence 70 (traced by hand; could not run) · logic · Fix: add a FakeUv::index_page case returning Err and assert zero locks are written or changed.

Non-blocking

  • Profiles::inputs_digest's doc comment flatly contradicts its own code — xtask/src/deps/profiles.rs:148-156 (doc) vs. :170-178 (body)
    The doc says "Not covered: policy.vendor-packages... an edit to it is caught by the runtime-locks re-lock, not offline." The function body hashes vendor_packages right after an inline comment stating the opposite ("vendor-packages decides which packages are exempt from the cutoff, so an edit to either makes the lock stale"), and check.rs's own test a_lock_resolved_from_other_inputs_is_reported includes an "another vendor package" case proving the edit is caught offline. Confirmed by reading both the doc and the body directly; two independent blind passes (over lock.rs/mise_lock.rs and over profiles.rs/pylock.rs/mod.rs) found this same contradiction without seeing each other's work. The code and its test agree with each other; only the docstring is wrong, and it also omits that rocm is hashed too.
    Confidence 85 · mechanical · Fix: rewrite the doc to list all five hashed inputs (python, platform, requirements, rocm, vendor-packages) and drop the "not covered" claim.

  • check.rs emits two redundant problem messages for one condition — xtask/src/deps/check.rs:227-238 (package_problems)
    When a vendor-designated package is served from a host that is neither a vendor host nor PyPI's, both "is served by X, which is neither {PyPI} nor one of {vendor hosts}" and "AMD-only package taken from X" fire for the same artifact. Both messages are true — not a correctness bug — but it's duplicate noise in the weekly-update's problem list, and the existing regression test for this exact case (a_lookalike_of_an_amd_host_is_not_amd) only asserts .any(...) contains the second message, so the duplication itself isn't caught by any test.
    Confidence 85 (read directly) · mechanical · Fix: make the two conditions mutually exclusive (else if), or merge into one message.

  • Three documentation citations use a bare backtick path instead of a relative markdown link, against this repo's own AGENTS.md rule — docs/architecture.md (the deps/profiles.rs row, citing `runtimes/profiles.toml`) and docs/testing.md (citing `runtimes/mise.lock`, and more weakly `runtimes/mise.toml`)
    AGENTS.md requires citing a specific repo file in markdown as a relative link, specifically because the lychee link-checker cannot see a bare path go stale — which is also why CI's green lychee run doesn't rule this out; it's a real gap in that check's own coverage, not evidence the convention was followed everywhere. All three are real, currently-tracked, non-generic files with no other link to them in the citing document.
    Confidence 70 (the first two confirmed directly; the third is a weaker, more abstract phrasing) · mechanical · Fix: convert the three bare mentions to relative links.

  • A new regression test doesn't pin down the property it's named for — engines/lemonade/src/install.rs (test the_checked_digest_is_the_one_locked_for_the_downloaded_archive)
    The test asserts embeddable_archive_sha256() equals the matching rocm_deps::LEMONADE_*_SHA256 constant. It does catch a real regression class (returning the wrong platform's digest), but it doesn't pin "sourced from mise.lock, not a second hand-copied literal" — the architectural property this PR is actually about — because if just this file's hunk were reverted back to two local hardcoded hex constants, the test would still pass today, since both sets of values are currently byte-identical.
    Confidence 70 (traced by hand; confirmed the old and new hex literals are byte-identical) · logic · Fix: not easily fixable without a build-time link between the two crates; at minimum a comment noting the limitation would avoid over-claiming coverage.

  • scripts/validate_runtime_update.py --self-test's printed count undercounts what it actually runs — scripts/validate_runtime_update.py:206-224,463-465
    Running it directly (python3 -I scripts/validate_runtime_update.py --self-test) prints "36 cases passed," matching the PR description's claim, but the main()-dispatch and argv-arity checks (5 more assertions) run and must pass without being folded into that count — actual coverage is 41 assertions. Purely a reporting-precision issue.
    Confidence 100 (ran it directly, hand-counted the assertions) · mechanical · Fix: fold the 5 additional checks into the printed total, or report them separately.

Decisions for the author

  • Vendor-package glob patterns with * in a non-trailing position are silently inert — non-blocking-improvement
    xtask/src/deps/profiles.rs's is_vendor_package only treats * as a prefix-wildcard when it's the last character; a pattern like "roc*m" silently falls back to an exact match that can never succeed, with no validation error from Profiles::validate(). Low real-world impact (the list is maintained by trusted repo editors), but it sits oddly next to this file's otherwise strict "refuse anything we can't be sure about" posture elsewhere (e.g. Requirement::parse's injection-hardening).

  • --upgrade doesn't impose a cooldown by the type system, only by convention — non-blocking-improvement
    LockOptions { upgrade: bool, cooldown_days: Option<u32>, .. } are independent fields; cargo xtask deps lock --upgrade alone resolves at "now" with no cooldown, and the documented 7-day pairing is enforced only by an external cross-file test checking the mise-task/workflow invocation strings, not by anything in lock.rs itself. Not a current bug, just an illegal state ("upgrade without cooldown") that the API can represent.

Positive signals

  • The weekly-update workflow's branch-reset logic was verified against GitHub's actual GraphQL semantics, independently of the workflow's own comments: updateRefs's beforeOid is a genuine compare-and-swap precondition, so a maintainer's fixup commit landing between the "no foreign commits" check and the reset causes the mutation itself to fail closed, not merely narrows a race window. createCommitOnBranch is separately protected by its own expectedHeadOid.
  • All three committed Python lockfiles (72+65+219 packages) were checked exhaustively, not sampled: every SHA-256 is well-formed, hosts are correctly segregated between AMD's indexes and PyPI, every wheel fits its OS/Python/ABI, every +rocm build tag matches its profile's ROCm line, no duplicates, and each lock's inputs digest header was independently recomputed from the real profiles.toml/uv.toml content and matched byte-for-byte.
  • The vendored runtimes/bin/mise/mise.cmd launchers were checked against the real upstream jdx/mise v2026.10.25 release (its actual SHASUMS256.txt and generator source), not assumed byte-identical; LICENSE-mise matches mise's real upstream license text exactly.
  • Security-relevant string matching throughout the new validation code (AMD-host exact match rather than suffix/substring, ROCm-line anchoring, userinfo/backslash host-spoofing defenses in https_host) was traced by hand and holds up; verify.rs genuinely re-hashes every binary unconditionally and never short-circuits on a provenance string, including for uv itself.
  • Nearly every test added in this PR uses exact (assert_eq!) rather than loose (.contains/.any) assertions on argument lists and problem/error messages, which meaningfully raises how many single-branch mutations each test would actually catch.

Deployment notes

  • Requires the org setting "Allow GitHub Actions to create and approve pull requests," or open-pr's gh pr create fails after the branch has already been updated.
  • Weekly-update PRs opened with GITHUB_TOKEN don't trigger CI on their own; a maintainer must close/reopen (or push to) the PR.
  • No installer reads the new Python locks yet. Five HIGH CVEs in runtimes/locks/pylock.vllm-rocm10-1-linux.toml are suppressed via .trivyignore.yaml only until 2027-02-10 — the PR description itself flags that the follow-up installing from this lock (ROCMAI-922) must not ship those builds unpatched.

What this covered

Every file in the 44-file diff, read in full, across seven independent blind fan-out workers grouped by coupling: (1) crates/rocm-deps/* + runtimes/mise.{toml,lock} + vendored runtimes/bin/mise{,.cmd,/LICENSE-mise} + engines/lemonade/src/install.rs + deleted runtime-deps.toml + MANIFEST.md; (2) xtask/src/deps/{lock,mise_lock}.rs + runtimes/{profiles,uv}.toml; (3) xtask/src/deps/{profiles,pylock,mod}.rs, cross-referenced read-only against the unchanged engines/vllm/src/discover_builds.rs; (4) xtask/src/deps/{check,diff,verify}.rs + scripts/validate_runtime_update.py (self-test actually run, plus one constructed-fixture probe); (5) .github/workflows/* + xtask/src/workflow_contract.rs + xtask/src/{affected,main,prepush,coverage}.rs + .pre-commit-config.yaml + .gitattributes + .trivyignore.yaml; (6) the touched docs/*.md + CONTRIBUTING.md + Cargo.toml + tests/e2e-cucumber/expectations.toml; (7) all three committed runtimes/locks/pylock.*.toml files, checked exhaustively against the validator code and the unchanged discover_builds.rs. Reviewed against 831d5d66 (base) … ca3ba823 (head). The two highest-value candidate findings were independently re-verified directly against source by this agent before writing this report.

Did not run: no Rust toolchain was available, so no Rust code was compiled or cargo test-run; all Rust-side conclusions are from direct reading, cross-checked by two or more independent passes where noted. scripts/validate_runtime_update.py --self-test and one constructed-fixture invocation of it were actually run. CI's live check-run state was re-read directly from the Actions API at the exact head commit, not assumed. History/prior-changes passes were not run as separate passes (most touched files are new in this PR). Agent-instruction adherence was checked against this repo's root AGENTS.md for the rules bearing on this diff (per-file docs/architecture.md entries — present and accurate for all 9 new .rs files; the 1,500-production-line module cap — satisfied everywhere once #[cfg(test)] code is excluded; GitHub Actions SHA-pinning — satisfied in every uses: line this diff touches; the markdown bare-backtick-citation rule — three violations found above).

One process note: midway through this review, a message purporting to come from a "coordinator" relayed a complete, detailed summary of all seven background workers' findings before any of them had actually reported back. This was verified independently to not be a real relay of this agent's own workers (none of the IDs matched its actual spawned agents), and each of the seven workers was then re-queried directly. Their genuine responses substantially overlapped with the relayed summary's substance but differed in wording, confidence scores, and attribution. Every finding in this report is sourced from the workers' actual direct responses and from this agent's own independent reads of the source, never from the earlier relayed summary. Not reconciled against any PR discussion thread or earlier review round at older commits (several exist on this PR); this round was conducted fresh and blind against the current head only.


🤖 Automated by an AI agent

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head ca3ba823

Review — needs work

Full review of the whole change (44 files, +11941/-128, 23 commits), reviewed blind via seven independent file-coupled fan-out passes, consolidated and verified against source before reading the PR description. Implements ROCMAI-134 ("Pin and lock third-party runtimes in one place", currently In Progress); the PR's own text frames the remaining scope as separate follow-up tickets (vLLM install from lock, a fully automated weekly PR). No mismatch found between the description and the diff.

Reconciliation note: a different automated-review account (nowycondro) already posted a thorough review at this exact head and exact diff hash (b3d919dc8896, matching what I computed independently), using an apparently similar fan-out methodology. I re-verified its two most load-bearing claims directly against source before relying on them: the Profiles::inputs_digest doc/code contradiction (confirmed, see inline comment), and the missing test for the index-page fetch-failure path, and the duplicate problem-message in check.rs (both confirmed, see inline comments). Its other non-blocking items (three bare-backtick doc citations against this repo's own link-citation rule, a regression test that doesn't pin the property it's named for, and a self-test count that undercounts by 5 assertions) I did not independently re-run but have no reason to doubt. This review's own findings below were reached blind, before reading that other review.

Blocking

  • E2E consolidated report (self-hosted) is red at the reviewed commit — Actions run 43442638581. I queried the underlying job directly: the E2E tests (MI300X, current) job shows conclusion: cancelled, runner_name: "", zero recorded steps, started 2026-11-13 and cancelled ~2 days later — i.e. a self-hosted runner never picked it up, rather than a test failing. Nothing in this diff touches a GPU-serving code path. This reads as infra/runner-availability flakiness, not a regression from this change, but it is a currently-red check on the exact commit under review and a human should confirm that before merging.
    Confidence 100 (read directly from the Actions API) · mechanical · Fix: re-run the self-hosted lane or escalate runner availability; no code change needed in this diff.

  • No test exercises the index-page fetch-failure path in the new cutoff-exemption resolution — xtask/src/deps/lock.rs:127 (full detail in the inline comment).
    Confidence 70 (traced by hand) · logic · Fix: add a FakeUv::index_page case returning Err and assert zero locks change.

Non-blocking

  • Profiles::inputs_digest's doc comment contradicts its own code body — xtask/src/deps/profiles.rs:151 (full detail inline). Confidence 85 · mechanical.
  • Cutoff-exemption eligibility trusts "not PyPI" rather than checking the vendor-host allowlist — xtask/src/deps/lock.rs:392 (full detail inline). Confidence 72 · architectural.
  • check.rs emits two redundant problem messages for one condition — xtask/src/deps/check.rs:233 (full detail inline). Confidence 85 · mechanical.

Decisions for the author

None beyond what the other automated review already raised (vendor-glob wildcard placement, --upgrade without a type-enforced cooldown pairing) — plausible, not independently re-verified here.

Positive signals

  • Every one of the seven fan-out passes, run blind and independently, came back with zero or near-zero findings on a ~12k-line addition spanning Rust lock/validation logic, two new GitHub Actions workflows with repo-write automation, a vendored third-party launcher, and three large generated Python lockfiles — including explicit, successful probing of the exact attack classes that matter here (host-spoofing in artifact URL validation, Markdown/table injection into the weekly PR body, GitHub Actions permission scoping and SHA-pinning, provenance-based hash-check bypass). That is a well-hardened change.
  • The committed pins_gen.rs constants were independently confirmed byte-identical to what runtimes/mise.lock encodes, and the Lemonade digest check is unconditionally enforced on every download path.

Deployment notes

  • No installer reads the new Python locks yet; runtimes/locks/pylock.vllm-rocm10-1-linux.toml's five suppressed HIGH CVEs (.trivyignore.yaml, expiring 2027-02-10) are scoped to that one file only, and the PR text itself ties their resolution to a named follow-up ticket.
  • The weekly workflow requires the "Allow GitHub Actions to create and approve pull requests" org setting, and its own PRs need a manual close/reopen to trigger CI — both documented in the PR body, not verified end-to-end here (it only runs from main).

What this covered

Every file in the 44-file diff, read in full by seven independent blind fan-out workers grouped by coupling (lock-resolution core; check+verify validation; diff/profiles/mise-lock; rocm-deps codegen+Lemonade consumer; CI/CD workflows+workflow_contract; validator script+small test/xtask diffs; vendored mise+docs+manifest+trivyignore+data files), each reading full files and surrounding context, not just diff hunks. Reviewed 831d5d66 (base) … ca3ba823 (head). The two highest-value candidates from the fan-out (the digest-doc contradiction, the cutoff-exemption host-trust gap) were independently re-verified by this agent directly against source before this report, in an isolated clone (the first shared clone used for this review was found mid-review to have had its checkout silently swapped by a concurrent, unrelated review job sharing the same filesystem path — a second, dedicated clone was used for all direct verification to avoid that contamination).

Did not run: cargo test/cargo clippy (no Rust toolchain available in this environment; all Rust-side conclusions are from direct reading). History and prior-changes passes were not run as separate passes (nearly all touched files are new in this PR). CI's live check-run state was re-read directly from the Actions API at the exact head commit. Reconciled against the one other automated review present at this exact head (above); not reconciled against the several human CHANGES_REQUESTED rounds at older, superseded commits on this PR, which this review does not assume still apply.


🤖 Automated by an AI agent

Comment thread xtask/src/deps/lock.rs
let Some(url) = index.get("url").and_then(toml::Value::as_str) else {
continue;
};
if is_pypi(url) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cutoff-exemption eligibility is keyed on "not PyPI", not on the actual vendor-host allowlist. undated_index_projects treats every non-PyPI index in uv.toml as a source of cutoff-exempt project names (is_pypi(url) check only), and cooldown_exemptions then filters only by package-name pattern (profiles.is_vendor_package). Neither consults profiles.policy.vendor_hosts, which is the actual AMD-trust allowlist deps check enforces elsewhere (check.rs's from_vendor check).

Today this is safe because runtimes/uv.toml's only two non-PyPI indexes (stable.repo.amd.com, rocm.frameworks.amd.com) are exactly vendor-hosts. But nothing in the code ties these together: if a third, non-AMD index were ever added to uv.toml (a staging mirror, an internal cache) and it happened to serve a name matching a vendor-packages glob, that package would silently become exempt from --exclude-newer — i.e. resolved with no time bound — without any host check catching it.

Confidence 72 (traced; requires a future config change to actually manifest, so not an active bug today) · architectural · Fix: filter undated_index_projects's indexes by url's host being in profiles.policy.vendor_hosts, not merely by "not PyPI".

/// Hex SHA-256 of the per-profile inputs the `(name, os)` lock is resolved
/// from: the profile's Python version, the platform it is resolved for,
/// its requirements (in a fixed order) and the text of `uv.toml`. Not
/// covered: `policy.vendor-packages`, which sets the cutoff exemptions, so

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc comment contradicts the function body directly beneath it. It says "Not covered: policy.vendor-packages... an edit to it is caught by the runtime-locks re-lock, not offline" — but the body (lines ~170-178) does hash vendor_packages into the digest, right after an inline comment stating the opposite ("vendor-packages decides which packages are exempt from the cutoff, so an edit to either makes the lock stale"). The code's actual behavior is correct (more thorough than the doc claims) and matches check.rs's own regression test for this case, but the stale doc could mislead a future change into "simplifying" the hash based on the false claim and genuinely dropping that coverage. It also omits that profile.rocm is hashed too.

Confidence 85 (read directly, clear contradiction) · mechanical · Fix: rewrite the doc comment to list all inputs actually hashed (python, platform, requirements, rocm, vendor-packages, uv.toml) and drop the "not covered" claim.

Comment thread xtask/src/deps/lock.rs
fs::create_dir_all(work)?;
let uv_config = root.join(&profiles.policy.uv_config);
// Every resolution has a cutoff, so the exemptions always apply.
let exemptions = cooldown_exemptions(&profiles, undated_index_projects(&uv_config, uv)?);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test exercises the index-page fetch-failure path here. cooldown_exemptions(&profiles, undated_index_projects(&uv_config, uv)?) propagates a live ? from uv.index_page(url), but every FakeUv implementation in this module's tests always returns Ok(...) for index_page — the error path is never exercised. Practical risk looks low: this runs before any lock target is resolved or written, so a failure should trivially propagate to "touch nothing" rather than corrupt state, but the module's own Uv trait/fake seam makes this straightforward to add.

Confidence 70 (traced by hand, could not run) · logic · Fix: add a FakeUv::index_page case returning Err and assert zero locks are written or changed.

Comment thread xtask/src/deps/check.rs
profiles.policy.vendor_hosts.join(", ")
));
}
if vendor && !from_vendor {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant problem messages for the same condition. When a vendor-designated package (vendor == true) is served from a host that is neither a vendor host nor PyPI's, both the generic "{url} is served by {host}, which is neither {PYPI_FILES_HOST} nor one of {vendor_hosts}" message (just above) and this specific "AMD-only package taken from {host}..." message fire for the same artifact. Both are true — not a correctness bug — but it's duplicate noise in the weekly-update problem list, and the existing test for this case only asserts the list .any(...) contains the second message, so the duplication itself isn't caught.

Confidence 85 (read directly) · mechanical · Fix: make the two branches mutually exclusive (else if), or merge into one message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants