test(forge): livegithub live-credentials oracle suite (RIG-2229) - #438
Merged
mattwilkinsonn merged 2 commits intoAug 22, 2026
Merged
Conversation
|
Compass engineering docs preview: https://compass-server-2229-forge-li.compass-eng-docs.pages.dev Deployed from |
rigel-mintaka
force-pushed
the
compass-server-2229-forge-livegithub-oracle
branch
from
August 20, 2026 03:52
af3d155 to
b827d93
Compare
mattwilkinsonn
approved these changes
Aug 21, 2026
Base automatically changed from
compass-server-2228-forge-golden-fixtures
to
main
August 21, 2026 04:19
Leg 2 of the forge integration-testing record (docs/designs/product/compass-forge-integration-testing/design.md, T2): a //go:build livegithub oracle suite that re-runs the T1 golden scenarios against the REAL GitHub and Linear forges using per-identity PATs from the environment, then asserts the live decoded value matches the committed T1 fixture EXCEPT for an explicit volatile-field allowlist (ids, timestamps, URLs, numbers, SHAs, branch names, per-identity account/author, run-id-suffixed title/body). Guarded by the build tag so a bare go test never compiles it; the untagged golden battery stays credential-free. The suite skips (never fails) when its credentials are unset: the GitHub trio (LIVEGITHUB_REPO/AUTHOR_TOKEN/REVIEWER_TOKEN) gates the GitHub legs, LINEAR_FORGE gates the Linear legs independently, each with a stable one-line skip-message literal that T3's CI guard greps from source. Scenarios: the F1 headline (author opens a PR, the author's own APPROVE is rejected 422, the reviewer's APPROVE succeeds), create/comment/get/list issue, create PR, comment on PR, submit REQUEST_CHANGES + COMMENT reviews, and an auth-failure path asserting the 401/403 maps to *StatusError and fires TokenSource.Invalidate(). Linear (co-equal): create/comment/get/list issue plus a locked ErrUnsupported contract on the PR/review family. Hygiene: unique run-id-suffixed artifact names, t.Cleanup teardown, a bounded one-shot backoff on GitHub's secondary rate limit. The live-green run activates when the throwaway testbed is provisioned; until then the suite compiles under the tag and skips cleanly, per the record's staged posture. Refs RIG-2229. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ound-2)
Adversarial review of the //go:build livegithub oracle found five blocking assertion defects that would make the GitHub oracle non-functional or vacuous once the throwaway testbed lands. All fixes are test-only; the shipped provider contract is correct as-is (verified at source, github.go:454 / linear.go).
H1: submit_review + F1 asserted past-tense read-side verdicts (approved/changes_requested/commented), but SubmitReview echoes the write-side input token (SubmittedReview{Verdict: in.Verdict}, github.go:454); assert approve/request_changes/comment. H2: comment_on_issue compared two all-volatile Comment values (vacuous {}=={}); assert the decoded shape directly (ID/URL non-empty, Body echoes the sent run-unique body; Linear identity travels via URL since its comment ids are UUIDs). H3: the PR-family scenarios never seeded a head branch, so CreatePullRequest 422s at setup; add seedHeadBranch (main-ref SHA -> create ref -> diverging commit) before every PR create, and set Draft:true to match the fixture.
M1: drop the dead "SEA" envTeam default and require LINEAR_FORGE_TEAM alongside LINEAR_FORGE (no-team-no-run). M2: archiveLinearIssue archived by team-number, a permanent no-op leaking every created issue; resolve the node UUID via the client's resolveIssueID first. Lows: generalize the bounded one-shot secondary-rate-limit backoff to all GitHub content-creates (still a one-shot select, not a retry loop); find the setup row by run-unique number instead of trusting got[0] ordering; drop the wasted wantBody seeding.
Verified: gofmt clean, go vet -tags livegithub clean, untagged build clean, 14 scenarios SKIP (not silent-pass) with credentials unset; skip-message literals byte-identical so T3's source-derived CI guard still matches.
Refs RIG-2229.
Co-authored-by: Matt Wilkinson <matt@rigel.build>
mattwilkinsonn
force-pushed
the
compass-server-2229-forge-livegithub-oracle
branch
from
August 21, 2026 04:19
b827d93 to
d2b332a
Compare
mattwilkinsonn
approved these changes
Aug 22, 2026
mattwilkinsonn
added a commit
that referenced
this pull request
Aug 22, 2026
…229 T2) (#486) * test(forge): add -update live-capture path for golden fixtures (RIG-2229 T2) Implements the `-update` regeneration lane owed by RIG-2229 T2: a `TestLiveUpdateFixtures` (//go:build livegithub) that re-runs each committed fixture's scenario against the real GitHub/Linear forges and rewrites testdata/<provider>/<name>.json, canonicalizing volatile fields on write and reusing the oracle's volatileFields as the single source of truth (per Matt's ruling). The oracle suite (#438) merged without this capture path; this closes it and unblocks T3's workflow_dispatch regen lane (RIG-2230). Mechanism (untagged golden_capture_test.go, runs credential-free in the normal battery): - canonicalizeWire rewrites a wire-JSON tree, substituting a fixed sentinel for every volatile wire key (matched by name at any depth) while preserving key, structure, and type. A pure per-run change canonicalizes to identical bytes; a real shape change (new/renamed/retyped field) survives and shows in the diff, so a regeneration diffs only on a genuine contract-shape change. - Both fixture halves (the emitted request method/path/query/body AND the decoded Want) are DERIVED from one invoke() replay over the canonicalized responses and canonicalized coordinates — the identical code path golden replay runs. So TestGoldenFixtures' load-bearing invariants (the emitted request matches Request, and marshal(decode(Body)) == Want) hold by construction after any regeneration, and a live per-run coordinate can never leak into a committed request. Because the request is re-derived (not recorded from the live wire), a Linear resolve id (teamId/issueId) canonicalizes consistently on both sides, so string ids are canonicalized for regen stability. - domainToWire ties the wire-key table back to volatileFields; TestUpdateCanonicalizeCoversVolatileFields asserts its keyset equals volatileFields and the two tables agree on wire keys, so a new domain volatile without a wire mapping fails the untagged battery. volatileFields moves from the tagged oracle file to the untagged capture file so one definition is shared by both suites (a tagged build compiles untagged files; stripVolatile still resolves it) — no behavior change to the oracle. .golangci.yml: set goconst.ignore-tests. goconst counts string literals package-wide but the existing exclusion drops goconst reporting on test files; without ignore-tests, a literal repeated in the new test counts toward the package total and tips a production site (linear.go) over the threshold, surfacing a finding there. ignore-tests aligns the counting with the reporting policy; production repetition still gates. Verified: gofmt clean; full untagged forge suite green (incl. all 10 committed fixtures via TestGoldenFixtures — no committed fixture changed); default-lane and livegithub-tagged golangci-lint both 0 issues; tagged `go vet` clean. TestUpdateCanonicalizeStable is a RED->GREEN guard proven to fail if a volatile wire key (incl. displayName, target_url) is dropped or if a request coordinate is taken from the live wire instead of derived. Refs RIG-2229. Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(forge): address review findings on -update capture (RIG-2229 T2) Review round 1 on #486 (0 high, 2 medium, 3 low). Fixes: - medium (composite coverage gap): add TestUpdateCanonicalizeComposite, an untagged credential-free guard for get_pull_request — the only fixture with EXTRA legs. It drives the real assembleFixture -> replayFixture pipeline over a synthetic PR detail (asserted) + reviews/check_runs/statuses (Extra), so the responses[prelude+1:] Extra assembly and canonNode's []any recursion into an array of objects carrying volatile keys (two review Authors) are exercised — the path create_issue cannot reach. Asserts both review authors canonicalize while per-review Verdict/IsBot survive distinctly, target_url + html_url both fold to canonURL, and head.sha canonicalizes consistently across the detail leg and the rolled-up Checks.HeadSHA. RED-proven: disabling canonNode's array recursion fires it on all four nested-volatile assertions. - medium (paginated regen truncation): add an all-consumed exact-count guard to deriveFixtureHalves mirroring replayFixture's invariant. A captured response the client never reaches (e.g. a later pagination page whose rel=next Link header the recorder drops, so HasNext stays false) would otherwise be written into a truncated fixture silently; now it fails loudly AT CAPTURE. RED-proven: an extra unconsumed response fires "replay emitted N requests, want exactly M". The additive multi-page header-capture capability this guard protects is deferred to RIG-2481 (no committed fixture is multi-page today). - low (setup-client clarity): setup clients built the fixture subject through a throwaway recordingRoundTripper whose recording was discarded, implying the recording was load-bearing. Add setupGitHub/setupLinear (non-recording) so only the asserted op is wrapped in a recorder. - low (guard message): reword assembleFixture's insufficient-responses fatal to match the check (at least prelude + 1, not > prelude + 1). Refs RIG-2229. Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <matt@rigel.build>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack containing 2 PRs:
mainLeg 2 of the forge integration-testing record (docs/designs/product/compass-forge-integration-testing/design.md, T2): a //go:build livegithub oracle suite that re-runs the T1 golden scenarios against the REAL GitHub and Linear forges using per-identity PATs from the environment, then asserts the live decoded value matches the committed T1 fixture EXCEPT for an explicit volatile-field allowlist (ids, timestamps, URLs, numbers, SHAs, branch names, per-identity account/author, run-id-suffixed title/body).
Guarded by the build tag so a bare go test never compiles it; the untagged golden battery stays credential-free. The suite skips (never fails) when its credentials are unset: the GitHub trio (LIVEGITHUB_REPO/AUTHOR_TOKEN/REVIEWER_TOKEN) gates the GitHub legs, LINEAR_FORGE gates the Linear legs independently, each with a stable one-line skip-message literal that T3's CI guard greps from source.
Scenarios: the F1 headline (author opens a PR, the author's own APPROVE is rejected 422, the reviewer's APPROVE succeeds), create/comment/get/list issue, create PR, comment on PR, submit REQUEST_CHANGES + COMMENT reviews, and an auth-failure path asserting the 401/403 maps to *StatusError and fires TokenSource.Invalidate(). Linear (co-equal): create/comment/get/list issue plus a locked ErrUnsupported contract on the PR/review family. Hygiene: unique run-id-suffixed artifact names, t.Cleanup teardown, a bounded one-shot backoff on GitHub's secondary rate limit.
The live-green run activates when the throwaway testbed is provisioned; until then the suite compiles under the tag and skips cleanly, per the record's staged posture.
Refs RIG-2229.
Co-authored-by: Matt Wilkinson matt@rigel.build