Skip to content

docs(agents): make the quality gate run the e2e scenarios - #372

Open
siloteemu wants to merge 1 commit into
mainfrom
docs-agents-md-runnable-commands
Open

siloteemu wants to merge 1 commit into
mainfrom
docs-agents-md-runnable-commands

Conversation

@siloteemu

@siloteemu siloteemu commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

AGENTS.md requires user-observable behaviour changes to be covered by a Gherkin scenario, but the quality gate it prescribed runs none of them.

What was wrong

  • The gate was cargo test --workspace --all-targets, cargo clippy --workspace --all-targets and cargo xtask smoke. The e2e target of tests/e2e-cucumber sets test = false, so cargo test and cargo nextest skip every scenario and still report success. Only cargo xtask e2e runs them, and the gate never named it.
  • The same setting leaves the harness out of the workspace clippy run, so CI covers it with a second, package-scoped invocation that the gate did not carry.
  • The notices check (cargo xtask tpn --check) is not in the gate. The cargo-tpn hook inside prek skips silently without the pinned cargo-about, where the command fails loudly, so a green local run did not mean what CI means.

What changed

  • AGENTS.md section 8: the gate gains prek run --all-files, the package-scoped e2e harness clippy, cargo xtask tpn --check and cargo xtask e2e. Both clippy lines take --locked and come first, as in CI. The notes under it explain why cargo test is not enough, that a green cargo xtask e2e does not show a gated scenario ran, that a filtered run or one with ROCM_CLI_BINARY set is not the gate, that some prek hooks rewrite files, and that a hook which skips or lacks its tool is a check that did not run (prek shows a skipped hook as passed unless the hook is verbose).
  • AGENTS.md section 3 gains one line pointing at cargo xtask e2e in section 8.
  • CONTRIBUTING.md: the sentence after the test table links section 8 as the minimum quality gate, step 4 of "Making changes" says to run at least that gate, and the cargo-about install comment says the section 8 gate needs the tool.
  • docs/testing.md: Local Verification drops its own copies of the cargo test and cargo clippy commands and points at section 8, and the "CI test selection" text no longer tells readers to keep using cargo test --workspace --all-targets for a full local verification.

Each fact has one owning place: section 8 owns the gate, and the other files point to it.

Test plan

Documentation only, so no scenario applies: no behaviour a CLI user can observe changes.

Checked by reading each claim against the source it describes: the e2e target's test = false, both clippy invocations in the CI workflow, PlatformManifest and e2e_binaries_from for the platform.json and ROCM_CLI_BINARY notes, run in the tpn module for the hook and the direct command, and the .pre-commit-config.yaml hook definitions. The commands were not run on this head; prek (including the link check) and the docs build run in CI.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: documentation only, no code behaviour changes, and expectations.toml is untouched by this PR.

Scope

  • Out of scope: the licence-header and notices walkthroughs, because CONTRIBUTING.md already owns what the tools need and how each hook behaves without them.
  • Out of scope: other e2e documentation drift found while checking, namely the e2e README tag table, a stale "Linux only" comment in tests/e2e-cucumber/Cargo.toml, cargo xtask e2e --help wording and the skip-reason strings.

@siloteemu
siloteemu marked this pull request as ready for review September 9, 2026 08:46
@siloteemu
siloteemu requested a review from a team as a code owner September 9, 2026 08:46

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Docs-only change (AGENTS.md, CONTRIBUTING.md), no runtime/CLI-facing impact.

Correctness: All 11 new factual claims (e2e test = false quirk, exact CI clippy invocation, cargo xtask e2e, license-header hook aborting vs. third-party-notices hook silently no-oping, etc.) were independently verified against the repo — all accurate. Fixes the four gaps it describes; a few other "rule with no named command" spots remain elsewhere in AGENTS.md (§3/§13, §6, §8, §12) — same defect class, out of scope here, good follow-up candidate.

Note: you already flagged it yourself, but the new e2e-cucumber clippy line uses --locked (matching CI) while the adjacent pre-existing workspace clippy line still doesn't, even though CI's version has it too — worth aligning whenever convenient.

UX: minor flow nits only — the new §3 paragraph sits between the topic sentence and the bullets it explains (slightly breaks the antecedent), and the AGENTS.md cross-reference in CONTRIBUTING.md lands mid-setup-section without a transition. Neither is misleading, just placement polish.

No blocking issues, CI green. Approving.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this doc-only change against the actual repo config rather than just the prose, and every claim checks out.

tests/e2e-cucumber/Cargo.toml really does set test = false on the e2e test target, so cargo test --workspace --all-targets silently skips every scenario — confirmed that's the real gap this fixes. CI's workflow runs both cargo clippy --locked --workspace --all-targets and a separate cargo clippy --locked -p e2e-cucumber --test e2e, and the new line added to AGENTS.md's quality gate matches that second invocation byte-for-byte. HAWKEYE_VERSION: v7.0.0 is pinned in ci.yml as stated, and the cargo-tpn pre-commit hook does use --if-available, so a green local hook run genuinely doesn't guarantee the same result CI's --check invocation gives.

CI is green (the E2E lanes correctly no-op since this touches no code path). Approving.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · d77c93b

Review — no blocking findings

Full review of the whole change (two files, 24 lines touched — read as a single pass; no fan-out needed given the size, per the review method's sizing guidance). No ticket named; the PR description is a self-contained bug report/fix for AGENTS.md's own quality-gate section, not tied to an external issue.

Blocking

None

Non-blocking

  • CONTRIBUTING.md's Test commands table still presents bare cargo test as if it covers Rust testing — CONTRIBUTING.md:74
    This PR adds a direct cross-reference from CONTRIBUTING.md:57 to AGENTS.md, whose newly-added text (AGENTS.md:200) explains, correctly (verified against tests/e2e-cucumber/Cargo.toml's test = false on the e2e target), that cargo test --workspace --all-targets silently skips every e2e scenario. The CONTRIBUTING.md table two lines below the new cross-reference still lists the Rust test command as plain cargo test with no such caveat. Pre-existing, not introduced by this diff, and softened by the table being followed by "See docs/testing.md for the full test guide" — but it is the same defect class this PR exists to fix, sitting in a file this PR already touches.
    Confidence 85 · documentation · Fix: add a one-line caveat or a second row for cargo xtask e2e to the table, or point it at AGENTS.md §8.

Decisions for the author

  • Two scope questions the author already raised in "Notes for reviewers" — tradeoff
    (1) The newly added cargo clippy --locked -p e2e-cucumber --test e2e line (AGENTS.md:193) carries --locked, matching CI byte-for-byte, while the adjacent pre-existing cargo clippy --workspace --all-targets line (AGENTS.md:192) does not, even though CI's equivalent does. (2) Whether the e2e-cucumber clippy line belongs in this PR at all, versus being "adjacent" scope creep. Both are already surfaced by the author with a stated preference to keep the change narrow. Independently verified: keeping the second clippy line is well justified (the same test = false root cause this PR documents also excludes the e2e harness from the first clippy invocation), and the --locked mismatch on the untouched line is pre-existing. Neither choice is wrong — confirm the narrow-scope call stands, or fold the alignment into this PR.

Positive signals

  • Every new factual claim is independently checkable against the repo's own configuration, and all of them check out on re-tracing: tests/e2e-cucumber/Cargo.toml's test = false comment, .github/workflows/ci.yml's exact two clippy invocations and HAWKEYE_VERSION pin, .pre-commit-config.yaml's --if-available on the notices hook and the plain system-language license-header hook, and docs/testing.md's explicit warning that -n/--tags bypass the suite's custom expectation/applicability resolver for lifecycle scenarios.
  • The new §8 bullets state why each command is needed rather than just asserting it (e.g., "that same test = false also excludes the e2e harness from cargo clippy --workspace --all-targets"), which is exactly the reasoning an agent-instruction file needs to extend a rule to a case not explicitly enumerated.

Deployment notes

None — documentation-only change, no build, runtime, or CI-configuration impact. CI confirms this: all GPU-dependent E2E lanes correctly report SKIPPED rather than SUCCESS or FAILURE.

What this covered

Full diff read completely: AGENTS.md and CONTRIBUTING.md, both in full (not just the changed hunks), against commits 03c46b58…d77c93bd (merge-base of the PR head with the current main tip, since main has moved on unrelated commits since this PR's base). Cross-checked every command and factual claim the diff adds against its actual source: tests/e2e-cucumber/Cargo.toml, .github/workflows/ci.yml, .pre-commit-config.yaml, docs/testing.md, tests/e2e-cucumber/README.md, and xtask/src/e2e.rs. Reviewed as an AI-instruction-file change per the ai-instruction-quality checklist (no phantom commands, no broken cross-references, examples accurate, reasoning given alongside each rule). Checked live CI via gh pr view --json statusCheckRollup: all required checks SUCCESS, GPU lanes SKIPPED as expected for a docs-only change. Checked commit history for the two files: single, well-formed Conventional-Commits-style commit, no noise to squash.

Did not run: "prior changes" pass — did not read this PR's review discussion or threads (out of scope per review instructions); no other open PRs against these two files were found to cross-check. History pass beyond git log on the two files was not otherwise applicable (docs-only, no code to blame against logic). Not reconciled against any discussion on this PR.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (two files, 23 additions/1 modification). No ticket named; the PR description is a self-contained bug report/fix for AGENTS.md's own quality-gate section.

Blocking

None

Non-blocking

  • CONTRIBUTING.md and docs/testing.md still present bare cargo test/cargo clippy --workspace --all-targets as if they cover Rust testing and linting — CONTRIBUTING.md:74, docs/testing.md:16, docs/testing.md:22
    This PR adds a direct cross-reference from CONTRIBUTING.md:57 to AGENTS.md, whose newly-added §8 text correctly explains, verified against tests/e2e-cucumber/Cargo.toml's test = false on the e2e target, that cargo test --workspace --all-targets silently skips every e2e scenario (and the adjacent cargo clippy --workspace --all-targets misses the harness too). CONTRIBUTING.md's "Test commands" table and docs/testing.md's "Local Verification" section — the two other places a contributor looks for these exact commands — still show the incomplete recipe with no caveat. Pre-existing, not introduced by this diff, but the same defect class this PR exists to fix, now a single cross-reference hop away from where it's named.
    Confidence 85 · mechanical · Fix: add a one-line caveat or cross-reference to AGENTS.md §8 at both locations.

Decisions for the author

  • Two scope questions already raised in "Notes for reviewers" — tradeoff
    (1) The new cargo clippy --locked -p e2e-cucumber --test e2e line matches CI byte-for-byte, while the adjacent pre-existing cargo clippy --workspace --all-targets line lacks --locked even though CI's equivalent has it. (2) Whether the e2e-cucumber clippy line belongs in this PR versus being adjacent scope creep. Independently verified: CI (.github/workflows/ci.yml) does run both clippy invocations exactly as now documented, and the --locked mismatch is on an untouched, pre-existing line. Neither choice is wrong — confirm the narrow-scope call stands, or fold the alignment in.

Positive signals

  • Every new factual claim is independently checkable against the repo's own configuration and holds up: tests/e2e-cucumber/Cargo.toml's test = false, .github/workflows/ci.yml's two exact clippy invocations and HAWKEYE_VERSION pin, .pre-commit-config.yaml's --if-available on the notices hook versus the plain language: system license-header hook, xtask/src/e2e.rs's argument-forwarding to the cucumber CLI, and docs/testing.md's explicit warning that --tags/-n bypass the suite's custom applicability/expectation resolver.
  • New §8 bullets explain why each command is needed rather than just asserting it, which is exactly the reasoning an agent-instruction file needs to extend a rule to an unenumerated case.

Deployment notes

None — documentation-only change, no build, runtime, or CI-configuration impact.

What this covered

Full diff read completely: AGENTS.md and CONTRIBUTING.md, both in full, against 03c46b58…d77c93bd (merge-base of the PR head with current main). Cross-checked every new command and factual claim against its source: tests/e2e-cucumber/Cargo.toml, .github/workflows/ci.yml, .pre-commit-config.yaml, docs/testing.md, xtask/src/e2e.rs. Reviewed as an AI-instruction-file change (no phantom commands, no broken cross-references, reasoning given alongside each rule). Checked live CI via the GitHub API: all required checks SUCCESS, GPU-dependent E2E lanes SKIPPED as expected for a docs-only change. Checked commit history: single well-formed commit, no overlapping recent edits to either file.

Did not run: history/prior-changes passes beyond git log on the two files (docs-only, nothing to blame against logic; no other open PRs found touching these files). Not reconciled against this PR's existing review discussion.


🤖 Automated by an AI agent

Comment thread CONTRIBUTING.md Outdated
cargo install cargo-about@0.9.1 --locked --features cli # optional, for THIRD_PARTY_NOTICES.txt
```

Agents working in this repository additionally follow `AGENTS.md`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, pre-existing: this new cross-reference points at AGENTS.md, whose newly-added §8 text correctly explains that tests/e2e-cucumber's test = false on the e2e target makes cargo test --workspace --all-targets silently skip every e2e scenario (verified in tests/e2e-cucumber/Cargo.toml). The "Test commands" table 17 lines below this (CONTRIBUTING.md:74) still lists the Rust test command as plain cargo test with no such caveat, and docs/testing.md:16/:22 show the same gap for cargo test --workspace --all-targets / cargo clippy --workspace --all-targets -- -D warnings. Same defect class this PR exists to fix, now one cross-reference-hop away. Confidence 85 · mechanical · Fix: add a one-line caveat or cross-reference to AGENTS.md §8 at both locations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in the reworked head (7179396). The sentence after the Test commands table in CONTRIBUTING.md now links AGENTS.md section 8 as the minimum quality gate and says the table lists only some of its commands, and step 4 of "Making changes" points at the same gate. In docs/testing.md, Local Verification no longer carries its own cargo test and cargo clippy commands and points at section 8 instead, and the "CI test selection" text no longer recommends plain cargo test --workspace --all-targets as the full local verification.

AGENTS.md requires user-observable behavior changes to be covered by a
Gherkin scenario under tests/e2e-cucumber/, but the quality gate it
prescribed ran none of them: the e2e target sets `test = false`, so
`cargo test --workspace --all-targets` skips every scenario and still
reports success, and the same setting leaves the harness out of
`cargo clippy --workspace --all-targets`.

AGENTS.md section 8: the gate gains `prek run --all-files`, the
package-scoped e2e harness clippy that CI runs, `cargo xtask tpn --check`
and `cargo xtask e2e`. Both clippy lines take `--locked` and come first,
as in CI. The notes under it say why `cargo test` is not enough; that a
green `cargo xtask e2e` does not show a gated scenario ran (a plain run
lists each skipped scenario with its reason in platform.json) and that a
filtered run, or one with `ROCM_CLI_BINARY` set, is not the gate; that
some `prek` hooks rewrite files, so the lines above it are re-run when it
changed anything; that a hook which skips or lacks its tool is a check
that did not run, and that prek shows a skipped hook as passed unless it
is verbose, as `powershell-script-analyzer` is not; and why the notices
check is its own line: the `cargo-tpn` hook inside prek skips silently
without the pinned `cargo-about`, where the command fails loudly. The
notes link the code behind the platform.json, `ROCM_CLI_BINARY` and
`cargo xtask tpn --check` claims. Section 3 gains one line pointing at
`cargo xtask e2e` in section 8.

CONTRIBUTING.md: the sentence after the test table links section 8 of
AGENTS.md as the minimum quality gate and says the table lists only some
of its commands; step 4 of "Making changes" says to run at least that
gate, plus the suites relevant to the change; the `cargo-about` install
comment and the sentence above it say the section 8 gate needs the tool.
docs/testing.md: the Local Verification section drops its copies of the
`cargo test` and `cargo clippy` commands and points at the gate instead,
and the "CI test selection" section no longer tells readers to keep
using `cargo test --workspace --all-targets` for a full local
verification and says the gate does not use `affected`.

Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com>
@siloteemu
siloteemu force-pushed the docs-agents-md-runnable-commands branch from d77c93b to 7179396 Compare November 17, 2026 19:20
@siloteemu siloteemu changed the title docs(agents): name the commands AGENTS.md's rules require docs(agents): make the quality gate run the e2e scenarios Nov 17, 2026
@siloteemu

Copy link
Copy Markdown
Contributor Author

I reworked this PR since the heads you reviewed (d77c93bd), so a fresh look would help.

  • Rebased onto main and shrunk it to the gate itself. AGENTS.md section 8 is now the one place that lists the minimum quality gate, and CONTRIBUTING.md and docs/testing.md point to it instead of carrying their own command lists.
  • Dropped the licence-header and notices walkthroughs: CONTRIBUTING.md already owns what the tools need and how each hook behaves without them.
  • Added cargo xtask tpn --check to the gate as its own line, because the cargo-tpn hook inside prek skips silently without the pinned cargo-about. Both clippy lines now take --locked and come first, as in CI.
  • Added notes that prek shows a skipped hook as passed unless the hook is verbose, and that a filtered run or one with ROCM_CLI_BINARY set is not the gate.

The PR title and body are updated to match.

@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants