Skip to content

test(ci): exercise metadata edits with a failed source check - #1945

Closed
mldangelo-oai wants to merge 11 commits into
mainfrom
mdangelo/codex/ci-metadata-negative-control
Closed

mldangelo-oai wants to merge 11 commits into
mainfrom
mdangelo/codex/ci-metadata-negative-control

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This disposable draft tests whether pull request edits preserve pending, failed and cancelled source checks. Its source test deliberately fails on this exact draft. Production candidate

Changes

Metadata edits can forward an unbroken source history for the current pull request, head and test merge. A missing, cancelled or ambiguous history calls for full qualification. The candidate checks the executing attempt and the final target, then forwards the result to three required OS checks. This draft also compares the Git parents with the webhook after a target change. Its deliberate assertion is restricted to this repository and PR 1945.

Testing

  • At 349609d1, a separate optional job passed while the required OS checks were missing and passed again once the source had failed. Missing-source edit · Failed-source edit
  • At fa159cd5, both directions of a retarget received a test merge for the previous base. Both guards recorded the expected and checked-out parents and exited with failure. Forward retarget · Restore main
  • At 879ef00e, the pending-source edit waited for the original source failure. A later edit observed that failure in 28 seconds. The original static test reported its intended failure, and the two container builds also reproduced a fixture type error. GitHub marked all nine individual OS results across the three runs required and failed. Source · Pending edit · Settled edit
  • The corrected 8f87be98 head passed the complete SDK and MCP typechecks, 18 workflow-quality tests, the aggregate contract and two Action event tests. All 21 ordinary Node tests passed in five simulated routes. Only this exact draft's route produced the deliberate extra failure. Repository and affected formatting passed.
  • At 17:49:31 UTC the fresh source run was queued, with no required OS checks on its head or test merge. It was cancelled before its static source test completed. The source workflow finished cancelled and the aggregate then reported failure on all three required OS checks. This body edit tests full qualification on the same commit after that cancellation. Cancelled source · Cancelled static test

Risk and rollout

This draft is a hosted negative control and should stay out of main. Compare source and metadata runs at each specific head and target. The draft state and mandatory review make underlying check IDs and job results the signals for this experiment.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the skip-release-notes Omit internal changes from generated release notes label Oct 24, 2026

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — reviewed as the disposable diagnostic draft described here. All ten selector/gate tests pass in each environment; the added test skips local/main/unrelated runs and is the only failure on the control PR. Formatting and workflow lint pass. The shared workflow/tests are identical to the verified candidate in #1944.

The hosted source static check failed as intended while the later metadata run succeeded on the same head. Source aggregation is still pending, so this does not yet establish preservation of the final required checks. Keep this diagnostic branch out of main as planned; the intentional canary should not become permanent test coverage.

@mldangelo-oai
mldangelo-oai changed the base branch from main to mdangelo/codex/ci-metadata-check-result October 24, 2026 16:52
@mldangelo-oai
mldangelo-oai changed the base branch from mdangelo/codex/ci-metadata-check-result to main October 24, 2026 17:00

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — one unintended type-check failure needs attention, as detailed inline; it is also present in #1944. The 20 ordinary selector/verifier tests passed in seven environments, with only this diagnostic PR’s canary failing as intended, and all 18 workflow tests passed. The hosted metadata run was still waiting on its source when inspected, so completed failure forwarding remains unverified. This remains a disposable diagnostic draft to keep out of main as planned.

const classifier = workflow.jobs["classify-changes"]!;
expect(classifier["timeout-minutes"]).toBe(10);
expect(classifier.if).toContain("github.event.changes.base != null");
expect(classifier["timeout-minutes"]).toContain("40 || 10");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Allow expression-valued timeouts in the workflow type

The parsed workflow type above still declares "timeout-minutes"?: number, so this new string assertion makes pnpm --dir sdk/typescript run types fail with TS2769 (Argument of type 'string' is not assignable to parameter of type 'number'). Please allow the expression value in that field, for example number | string, and rerun typechecking. This is independent of the intentional canary: the same failure occurs outside the diagnostic PR environment.

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this owner-level re-review confirms the timeout-type fix. All 442 focused SDK tests, 21 ordinary selector/verifier tests, two Action event tests, SDK/MCP types, formatting and workflow lint pass. The deliberate canary is isolated to this diagnostic PR.

The cancelled source emitted three failing required OS checks. Its next metadata edit selected full qualification and reached the intended canary failure; that run is still finishing, so its final aggregate is not yet verified. No new confirmed code defect. This remains the disposable diagnostic draft described in the PR, to keep out of main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-release-notes Omit internal changes from generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants