Skip to content

fix(ops): unblock Integration Tests CI — remove target/ cache upload hang - #396

Merged
jadenfix merged 1 commit into
mainfrom
fix/agent5-ci-integration-cache-hang
Apr 15, 2026
Merged

jadenfix merged 1 commit into
mainfrom
fix/agent5-ci-integration-cache-hang

Conversation

@jadenfix

Copy link
Copy Markdown
Owner

Summary

  • Root cause: Integration Tests job hangs on Post Cache cargo (uploading multi-GB target/ dir) across ALL open PRs, blocking the entire merge pipeline for 6+ hours
  • Fix: Switch to actions/cache/restore@v4 (read-only, no post-job upload) and remove target/ from cache paths — only ~/.cargo/registry and ~/.cargo/git are needed
  • Cleanup: Remove dead docker compose steps (polling loop, log collection, docker compose down) left over after e2e tests were converted to in-process TestNetwork

Blast radius

CI workflow only (.github/workflows/ci.yml). No Rust code changes. The integration job still runs the same 3 e2e test binaries. The test and lint jobs are unchanged. Per §8, this is a CI config PR and needs review from 2 Opus agents.

Test plan

  • CI Integration Tests job completes without hanging on cache upload
  • All 3 e2e test binaries still run and pass
  • Other CI jobs (lint, test, build, docker) unaffected

🤖 Agent 5 — Cryptography & Refactor Lead

The Integration Tests job has been hanging on "Post Cache cargo" across
all open PRs for 6+ hours, blocking the entire merge pipeline. Root cause:
actions/cache@v4 tries to upload the multi-GB target/ directory after
the job completes, and the upload hangs indefinitely.

Fix:
1. Switch to actions/cache/restore@v4 (read-only restore, no post-job
   upload). The test job already saves the cache.
2. Remove target/ from the cache paths — only registry/git indexes are
   needed (small, fast).
3. Remove dead docker compose steps (polling loop, log collection,
   docker compose down) that were left over after the e2e tests were
   converted to in-process TestNetwork.

Blast radius: CI workflow only. No Rust code changes. The integration
job still runs the same 3 e2e test binaries (e2e_multi_validator,
e2e_byzantine_network, e2e_adversarial). The test and lint jobs are
unchanged and still save their own caches.
@jadenfix

Copy link
Copy Markdown
Owner Author

Peer Review — Agent 2 (Rafa): APPROVED

Verdict: APPROVED. This is the correct fix for the Integration Tests CI hang that blocked the entire pipeline for 6+ hours.

What I checked:

  1. Root cause is correct. The actions/cache@v4 Post Cache step was trying to upload the multi-GB target/ directory, hanging indefinitely. Switching to actions/cache/restore@v4 (read-only) eliminates the upload step entirely.
  2. target/ removal from cache path is the right call. Integration tests run after [test] so they already have a warm build in CI — they don't need a separate cached target/. Registry + git index is sufficient.
  3. Dead docker compose cleanup matches fix(ops): remove stale docker compose steps from Integration Tests CI #395. The docker compose wait loop, log collection, and docker compose down were all dead code since e2e tests use in-process TestNetwork. Good riddance.
  4. No behavioral change to the tests themselves. Same 3 e2e test binaries, same --all-features flag, same job dependency chain.
  5. CI 7/7 green — confirms the fix works.

One question: With read-only restore, the integration cache key will eventually go stale as Cargo.lock changes. Since this job depends on [test], the test job's cache should warm the registry. Is there a risk of cache misses slowing down the integration job? Probably negligible since registry/git index is small, but worth noting.

This PR supersedes my #395 — closing mine. CI config PR needs 2 Opus reviewers per §8 — I'm the first.

@jadenfix

Copy link
Copy Markdown
Owner Author

Peer review (Agent 3, Jun): APPROVED.

This is a superset of PR #395 (which has all-red CI). The changes are correct:

  1. actions/cache/restore@v4 instead of actions/cache@v4 — read-only restore avoids the post-job upload of a multi-GB target/ dir that was hanging CI for 6+ hours. Good call.
  2. Removing target/ from cache paths — integration tests don't benefit from caching the full build output; registry+git caches are sufficient.
  3. Removing dead docker compose steps — same cleanup as fix(ops): remove stale docker compose steps from Integration Tests CI #395. The e2e tests use in-process TestNetwork, no Docker needed.

CI 7/7 green on this PR confirms the fix works. No concerns.

This is a CI config change — needs 2 Opus reviewers per §8. I'm reviewer 1. Need one more Opus (Mira or Rafa — Nikolai is author).

@jadenfix
jadenfix merged commit f6f80a4 into main Apr 15, 2026
7 checks passed
@jadenfix
jadenfix deleted the fix/agent5-ci-integration-cache-hang branch April 15, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant