Skip to content

fix(redshift): stabilize program upgrade coordination - #354

Merged
bzawisto merged 2 commits into
mainfrom
fix/program-upgrade-coordination
Nov 16, 2026
Merged

bzawisto merged 2 commits into
mainfrom
fix/program-upgrade-coordination

Conversation

@bzawisto

Copy link
Copy Markdown
Collaborator

What changed

Coordinate upgrade delivery through separate hot/cold proxies and an observed cold HTTP fetch, so a missed WebSocket notification cannot stall the test. Finish warm progress before starting cold loads while preserving version, rollback, and convergence checks.

Closes #353

@bzawisto bzawisto added the bug Something isn't working label Nov 16, 2026
@bzawisto bzawisto self-assigned this Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The program upgrade consistency scenario now uses separate proxies and fault scopes for the hot and cold ERs. It verifies that the cold ER has not materialized the program, then waits for a stalled cold account fetch before releasing the configured faults.

Changes: Program upgrade consistency

Layer / File(s) Summary
Separate proxy and fault scopes
redshift/src/scenarios/chainlink/program_upgrade_consistency.rs
The scenario creates and configures separate proxies and fault scopes for the hot and cold ERs. Invocations and hot upgrade holds use the hot fault scope. Cleanup resets both scopes and drops both proxies.
Cold-fetch fault coordination
redshift/src/scenarios/chainlink/program_upgrade_consistency.rs
After the upgrade, the scenario checks that the cold ER has not materialized the program. It stalls the cold program-account HTTP response during three cold account fetches and releases the hot holds and cold stall after the cold response stall is observed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other · Severity of issue fixed: Low

Suggested reviewers: bmuddha

Merge Risk: Minimal · up to 66a27

This change makes the program upgrade consistency test scenario more reliable by coordinating on an observed cold fetch instead of a one-off notification. It only affects test code, and no merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "fix(redshift): stabilize program upgrade coordination" matches the main change. The change gives the hot and cold ERs separate proxies and fault scopes, and it waits for an observed cold fe…
Description check Passed The description states that upgrade delivery now runs through separate hot and cold proxies with an observed cold HTTP fetch. It also says warm progress finishes before cold loads start, and it links …
Linked Issues check Passed Issue #353 is the only directly linked active issue. Its coding requirements are: (1) avoid timeouts from a one-off WebSocket notification that reconnection does not replay; (2) avoid cross-attributio…
Out of Scope Changes check Passed The changes are confined to redshift/src/scenarios/chainlink/program_upgrade_consistency.rs, which the issue names as the scope. The changed logic (separate hot and cold proxies and fault scopes, held…
Full details: Title check

Explanation

The title "fix(redshift): stabilize program upgrade coordination" matches the main change. The change gives the hot and cold ERs separate proxies and fault scopes, and it waits for an observed cold fetch before it releases the hot holds. The title names the redshift scope and the upgrade coordination concern, so a reviewer scanning history can identify the primary change.

Full details: Description check

Explanation

The description states that upgrade delivery now runs through separate hot and cold proxies with an observed cold HTTP fetch. It also says warm progress finishes before cold loads start, and it links the issue it closes (#353). It directly describes the changeset.

Full details: Linked Issues check

Explanation

Issue #353 is the only directly linked active issue. Its coding requirements are: (1) avoid timeouts from a one-off WebSocket notification that reconnection does not replay; (2) avoid cross-attribution of HTTP responses between hot and cold validators; (3) avoid cold fetch deadlines spent waiting on warm traffic; (4) coordinate held delivery through an observed cold fetch; (5) keep version, rollback, and convergence checks.

Evidence at the reviewed head (redshift/src/scenarios/chainlink/program_upgrade_consistency.rs):

  • (1) and (4): upgrade delivery is held on the hot ER through hot_faults.stall on accountNotification/programNotification WS selectors and on HTTP getAccountInfo/getMultipleAccounts responses for program_id and programdata. The test does not depend on the WS notification arriving.
  • (2): hot_proxy and cold_proxy are separate Proxy instances with separate Faults scopes. cold_fetch stalls the HTTP program-account response only on the cold proxy.
  • (3): progress.await_count(1, after_upgrade + 8) completes warm progress before the cold loads begin.
  • (4): the cold fetch is observed with cold_fetch.wait_hit(TIMEOUT). Holds are removed only after that hit.
  • (5): Invocations::verify keeps version-marker checks and rollback checks (id.is_multiple_of(4), FAILURE_CODE). Convergence is checked by await_accounts for fresh B materialization, progress.await_count(3, 12), and ensure!(state == self.expected). run still calls finish and resets both fault scopes before dropping both proxies.

Limitation: the textual diff was unavailable, so this assessment relies on the head file content and the PR summary. It does not compare against the base revision. The requirements are satisfied by the head content. No automated test run was inspected.

Full details: Out of Scope Changes check

Explanation

The changes are confined to redshift/src/scenarios/chainlink/program_upgrade_consistency.rs, which the issue names as the scope. The changed logic (separate hot and cold proxies and fault scopes, held upgrade delivery, observed cold fetch, warm-progress ordering, and fault and proxy cleanup) directly implements issue #353. The raw summary reports no changes to exported or public declarations. No unrelated files or behavior were identified in the available evidence. The textual diff could not be read, so the check covers the file content at head and the PR summary, not a line-by-line diff against base.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

Clippy (1.99.1)

Clippy execution timed out

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 6efa5ca and 66a27b4.

Files selected for processing: 1

  • redshift/src/scenarios/chainlink/program_upgrade_consistency.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@bzawisto
bzawisto marked this pull request as ready for review November 16, 2026 15:39
@bzawisto
bzawisto merged commit e17779f into main Nov 16, 2026
2 checks passed
@bzawisto
bzawisto deleted the fix/program-upgrade-coordination branch November 16, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Program upgrade checks can time out after missed notifications

1 participant