fix(ci): free runner disk after image loads; diagnose e2e rollout timeouts#187
Conversation
…eouts External PR #178's run (28623232380) failed with the controller rollout timing out after a fresh install in the image-refresh lane. The job summary shows the runner bottomed out at 1.4GB free disk - inside the kubelet eviction threshold of the k3d node, the exact image-GC / pod-not-ready failure mode load-image.sh pins against. Nothing about the failure was fork-specific: the untrusted path (artifact handoff, local build, import) worked end to end. - Delete the multi-GB docker-load tarballs right after loading in every consumer job; they were sitting on disk for the whole e2e run. - fail-fast: false on the e2e matrix: quickstart's failure cancelled E2E (full) mid-run, losing its signal and erroring its report upload. - deploy-controller.sh: widen the rollout wait to 300s (fresh installs also wait on cert-manager issuing the admission cert) and dump node conditions, pods, describe, and events on timeout so the next occurrence names its cause in the log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…holds The diagnostics added in the previous commit named the cause directly: the k3d node reported DiskPressure and evicted/rejected the controller pod, while the runner still had 3.3GB free. k3s defaults kubelet eviction to nodefs.available<5% - ~3.6GB of a 72GB CI disk - and the pressure condition then lingers for the default 5m transition period, longer than the rollout wait. - start-cluster.sh: absolute eviction thresholds (1Gi) and a 30s pressure transition period on all k3d nodes. Percentages don't express "actually almost full" when the node shares the host disk. (Invalidates _cluster-ready: next local e2e run rebuilds the cluster.) - ci.yml e2e job: free ~18GB of unused preinstalled toolchains (android, CodeQL, dotnet) before downloading artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Why
External PR #178's run (28623232380) failed in E2E (quickstart). Diagnosis:
STATUS: deployed).load-image.sh's pinning comments describe. The controller pod never became Ready androllout status --timeout=180sexpired.fail-fastcancelled E2E (full) mid-run, and the cancelled lane then errored its Ginkgo report upload ("No files were found").What
docker loadsteps now delete their multi-GB tarballs immediately after loading (all consumer jobs).fail-fast: falseon the e2e matrix so one lane's flake can't cancel the other.deploy-controller.sh: rollout wait 180s → 300s (fresh installs also wait on cert-manager issuing the admission cert), and on timeout it dumps node conditions (DiskPressure!), pods, describe, and events so a recurrence names its cause in the log.After merge, re-run PR #178's workflow — it should pass; its previous failure was environmental.
🤖 Generated with Claude Code