fix(e2e): deliver image via k3d direct mode (root cause of the tools-node race)#186
Conversation
…ation Root cause found in k3d v5.9.0 source: executeInNode polls ContainerExecInspect and trusts Running=false/ExitCode=0, but Docker's exec lifecycle is async — the poll can misread the tools-node save-image exec as complete before the tar exists on the shared volume, and importWithToolsNode only logs (never returns) the resulting per-node ctr import failure, so k3d exits 0 claiming success. Direct mode streams docker save into each node's ctr stdin: no tarball, no shared-volume window, and node import failures are propagated. Verified on a live 4-node cluster with the project image. Caveat documented in-script: direct + ctr --all-platforms cannot handle multi-arch images saved by a containerd-store docker; PROJECT_IMAGE is always single-arch. The retry now also covers a loud nonzero exit from k3d, and the pin step remains the final verification that the image really landed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe ChangesImage import script update
Estimated code review effort: 2 (Simple) | ~10 minutes Related PRs: None found. Suggested labels: None Suggested reviewers: None 🐰 A hop, a skip, direct we go, 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
hack/e2e/load-image.sh (1)
165-204: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winHandle an empty node list here. If
cluster_node_namesproduces no matches, this returns success without pinning anything. Fail fast when no nodes are discovered.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e/load-image.sh` around lines 165 - 204, The import_image() retry loop currently succeeds even when cluster_node_names returns no nodes, because the pinning loop never runs and pinned stays true. Add an explicit check before iterating nodes: if cluster_node_names yields an empty list, treat it as a failure, log an error, and continue retrying or return nonzero instead of succeeding. Keep the fix localized to import_image() and the cluster_node_names/pin_imported_image flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@hack/e2e/load-image.sh`:
- Around line 165-204: The import_image() retry loop currently succeeds even
when cluster_node_names returns no nodes, because the pinning loop never runs
and pinned stays true. Add an explicit check before iterating nodes: if
cluster_node_names yields an empty list, treat it as a failure, log an error,
and continue retrying or return nonzero instead of succeeding. Keep the fix
localized to import_image() and the cluster_node_names/pin_imported_image flow.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 49328ea8-0909-43be-9c3b-a64ab4d2b584
📒 Files selected for processing (1)
hack/e2e/load-image.sh
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
1 participant
Follow-up to #185, with the root cause now pinned down in the k3d v5.9.0 source.
Why the import was failing (the actual mechanism)
Two k3d defects combine in the default tools-node mode:
pkg/runtimes/docker/node.go,executeInNode): k3d pollsContainerExecInspectevery 1s and treatsRunning=false, ExitCode=0as success. Docker's exec lifecycle is asynchronous, so a poll can misread thesave-imageexec as complete before the tarball exists on the shared image volume. In the failing run (28610319679) the "save" appeared to finish within the same second (passing runs take ~2s), thenserver-0'sctr image importhitno such file or directory.pkg/client/tools.go,importWithToolsNode): the per-node import failure is onlyErrorf-logged, never returned — k3d prints "Successfully imported" and exits 0. Only our pin/verify step told the truth.What this changes
k3d image import -m direct: streamsdocker savestraight into each node'sctr image importstdin (loadImageFromStream) — no tarball, no shared-volume timing window, and node import failures are propagated through an errgroup. The retry from #185 now also covers a loud nonzero k3d exit, and the pin step remains the final proof the image landed.Verified on a live 4-node cluster
-m directwith the real project image: 5.4s, image present + pinned in all 4 nodes.task test-e2ewith the stamp cleared (forcing the new path, including the image-refresh rebuild→reload cycles): 53/53 specs passed, 8 skipped, 9m15s.ctr --all-platformsfails (content digest not found) for multi-arch images saved by a containerd-store docker — the stream lacks foreign-platform blobs (file-based import of the same tar fails identically, so it's the save format, not stdin).PROJECT_IMAGEis always the single-arch controller image, so this doesn't apply to us.Validation
task lint: 0 issuestask test: pass (coverage 73.8%, within tolerance of 73.9% baseline)task test-e2e: pass (53/53)🤖 Generated with Claude Code
Summary by CodeRabbit