Skip to content

[Due for payment 2026-05-29] Bug: Deploy checklist lookup failure is treated as unlocked and staging can deploy #88930

Closed

Description

@mountiny

Problem

When preDeploy cannot reliably determine deploy checklist state, the pipeline can still run createNewVersion and updateStaging, force-pushing main to staging and triggering a full staging deploy.

Example (2026-04-27)

Root cause

isDeployChecklistLocked.ts wraps getDeployChecklist() and on any error sets IS_LOCKED to false and continues. That is fail-open: any failure to load the open StagingDeployCash issue (including “no issue found”, API errors, or parsing failures) is indistinguishable from “QA is not blocking” in preDeploy’s SHOULD_DEPLOY logic for deployers.

getDeployChecklist() currently throws when issues.listForRepo returns an empty list, so the “no open checklist” case and many failure modes share the same catch path.

Expected behavior

  • Staging must not be promoted when we cannot confirm checklist / lock state.
  • We should only treat the checklist as “absent / not locked” when we have a successful, verified GitHub response (for example: successful API call and an explicit interpretation of results that matches product rules).
  • Transient GitHub/API failures should not unlock deploys.

Proposed solution

  1. Separate outcomes in code (action and/or DeployChecklistUtils):

    • Success + empty list → “no open StagingDeployCash issue” (only if product agrees this means deploy is allowed; otherwise fail closed).
    • Success + exactly one issue → read labels; set IS_LOCKED from 🔐 LockCashDeploys 🔐 as today.
    • Success + multiple issues → error / fail closed (already invalid per getDeployChecklist).
    • Request / network / 5xx / rate limit / parse errors → not the same as “unlocked”.
  2. Retries: for retryable failures only, retry issues.listForRepo (or the thin wrapper) up to 3 times with backoff sleep (e.g. 2s / 5s / 10s or similar) before giving up.

  3. Fail closed after retries: if we still do not have a definitive successful read, do not set IS_LOCKED to false by default. Options (pick one in implementation / review):

    • set IS_LOCKED to true so SHOULD_DEPLOY stays false, or
    • fail the chooseDeployActions job so the workflow is red and nothing merges to staging until someone fixes GitHub / checklist state.
  4. Tests: extend isDeployChecklistLockedTest (and any DeployChecklistUtils tests) to cover: empty success vs error, retries, and post-retry fail-closed behavior.

Acceptance criteria

  • No staging deploy path when checklist state is unknown after retries.
  • Retry behavior is logged clearly for supportability.
  • Unit tests cover fail-closed and success paths.
Issue OwnerCurrent Issue Owner: @mountiny

Activity

  1. self-assigned this
    on Apr 27, 2026
  2. mountiny commented on May 12, 2026

    @mountiny
    ContributorAuthor

    looking into the pr again today

  3. mountiny commented on May 14, 2026

    @mountiny
    ContributorAuthor

    Doing a bit of digging

  4. mountiny commented on May 19, 2026

    @mountiny
    ContributorAuthor

    PR in review

  5. mountiny commented on May 20, 2026

    @mountiny
    ContributorAuthor

    Waiting for QA and deploy

  6. changed the title [-]Bug: Deploy checklist lookup failure is treated as unlocked and staging can deploy[/-] [+][Due for payment 2026-05-29] Bug: Deploy checklist lookup failure is treated as unlocked and staging can deploy[/+] on May 22, 2026
  7. melvin-bot commented on May 22, 2026

    @melvin-bot

    Reviewing label has been removed, please complete the "BugZero Checklist".

  8. melvin-bot commented on May 22, 2026

    @melvin-bot

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.3.79-4 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2026-05-29. 🎊

  9. mountiny commented on May 25, 2026

    @mountiny
    ContributorAuthor

    PR is in production we can close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

DailyKSv2Fire-CleanupSomething needs to be done after a fireTask

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions