Repository navigation
Description
Problem
When preDeploy cannot reliably determine deploy checklist state, the pipeline can still run createNewVersion and updateStaging, force-pushing main to staging and triggering a full staging deploy.
Example (2026-04-27)
-
Merge Fix bidirectional duplicate violation cleanup ran preDeploy workflow run 24994278102.
-
In chooseDeployActions job 73188175321, step
isDeployChecklistLockedlogged:No open deploy checklist found, continuing... Error: Unable to find StagingDeployCash issue. -
Immediately after,
createNewVersionandupdateStaging / updateProtectedBranchsucceeded, which pushedstagingand started staging deploy 24994657605. -
A few minutes later, another merge’s preDeploy correctly deferred deploy (
skipDeployran), showing the checklist lock was intended to block merges during QA.
Root cause
isDeployChecklistLocked.ts wraps getDeployChecklist() and on any error sets IS_LOCKED to false and continues. That is fail-open: any failure to load the open StagingDeployCash issue (including “no issue found”, API errors, or parsing failures) is indistinguishable from “QA is not blocking” in preDeploy’s SHOULD_DEPLOY logic for deployers.
getDeployChecklist() currently throws when issues.listForRepo returns an empty list, so the “no open checklist” case and many failure modes share the same catch path.
Expected behavior
- Staging must not be promoted when we cannot confirm checklist / lock state.
- We should only treat the checklist as “absent / not locked” when we have a successful, verified GitHub response (for example: successful API call and an explicit interpretation of results that matches product rules).
- Transient GitHub/API failures should not unlock deploys.
Proposed solution
-
Separate outcomes in code (action and/or
DeployChecklistUtils):- Success + empty list → “no open
StagingDeployCashissue” (only if product agrees this means deploy is allowed; otherwise fail closed). - Success + exactly one issue → read labels; set
IS_LOCKEDfrom🔐 LockCashDeploys 🔐as today. - Success + multiple issues → error / fail closed (already invalid per
getDeployChecklist). - Request / network / 5xx / rate limit / parse errors → not the same as “unlocked”.
- Success + empty list → “no open
-
Retries: for retryable failures only, retry
issues.listForRepo(or the thin wrapper) up to 3 times with backoff sleep (e.g. 2s / 5s / 10s or similar) before giving up. -
Fail closed after retries: if we still do not have a definitive successful read, do not set
IS_LOCKEDtofalseby default. Options (pick one in implementation / review):- set
IS_LOCKEDtotruesoSHOULD_DEPLOYstays false, or - fail the
chooseDeployActionsjob so the workflow is red and nothing merges to staging until someone fixes GitHub / checklist state.
- set
-
Tests: extend
isDeployChecklistLockedTest(and anyDeployChecklistUtilstests) to cover: empty success vs error, retries, and post-retry fail-closed behavior.
Acceptance criteria
- No staging deploy path when checklist state is unknown after retries.
- Retry behavior is logged clearly for supportability.
- Unit tests cover fail-closed and success paths.
Activity
mountiny commentedon May 12, 2026
looking into the pr again today
mountiny commentedon May 14, 2026
Doing a bit of digging
Simplify deploy checklist resolution and cover fake-empty case
mountiny commentedon May 19, 2026
PR in review
Fail closed when deploy checklist lock cannot be resolved
mountiny commentedon May 20, 2026
Waiting for QA and deploy
[-]Bug: Deploy checklist lookup failure is treated as unlocked and staging can deploy[/-][+][Due for payment 2026-05-29] Bug: Deploy checklist lookup failure is treated as unlocked and staging can deploy[/+]melvin-bot commentedon May 22, 2026
Reviewinglabel has been removed, please complete the "BugZero Checklist".melvin-bot commentedon May 22, 2026
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.3.79-4 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2026-05-29. 🎊
mountiny commentedon May 25, 2026
PR is in production we can close