Skip to content

fix(ci): free runner disk after image loads; diagnose e2e rollout timeouts #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sunib merged 2 commits into from
Jul 3, 2026
Merged

fix(ci): free runner disk after image loads; diagnose e2e rollout timeouts #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f9226a9
fix(ci): free runner disk after image loads; diagnose e2e rollout tim…
sunib Jul 2, 2026
6df58ab
fix(e2e): stop k3d DiskPressure evictions with absolute kubelet thres…
sunib Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 34 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,10 @@ jobs:

- name: Load CI container image (PR)
if: github.event_name == 'pull_request'
run: docker load -i ci-container-image.tar
# rm after load: the multi-GB tarball otherwise stays on disk for the whole
# job — e2e runners have been observed within 1.5GB of the kubelet eviction
# threshold (run 28623232380).
run: docker load -i ci-container-image.tar && rm -f ci-container-image.tar

- name: Lint and package Helm chart
run: |
Expand Down Expand Up @@ -324,7 +327,10 @@ jobs:

- name: Load CI container image (PR)
if: github.event_name == 'pull_request'
run: docker load -i ci-container-image.tar
# rm after load: the multi-GB tarball otherwise stays on disk for the whole
# job — e2e runners have been observed within 1.5GB of the kubelet eviction
# threshold (run 28623232380).
run: docker load -i ci-container-image.tar && rm -f ci-container-image.tar

- name: Run lint tasks
run: |
Expand Down Expand Up @@ -391,7 +397,10 @@ jobs:

- name: Load CI container image (PR)
if: github.event_name == 'pull_request'
run: docker load -i ci-container-image.tar
# rm after load: the multi-GB tarball otherwise stays on disk for the whole
# job — e2e runners have been observed within 1.5GB of the kubelet eviction
# threshold (run 28623232380).
run: docker load -i ci-container-image.tar && rm -f ci-container-image.tar

- name: Run tests
run: |
Expand Down Expand Up @@ -445,7 +454,7 @@ jobs:

- name: Load project image (PR)
if: github.event_name == 'pull_request'
run: docker load -i project-image.tar
run: docker load -i project-image.tar && rm -f project-image.tar

- name: Pull project image (trusted)
if: github.event_name != 'pull_request'
Expand Down Expand Up @@ -489,6 +498,9 @@ jobs:
packages: read
id-token: write
strategy:
# Don't let one lane's flake cancel the other mid-run — a cancelled lane
# loses its signal *and* fails its report upload with "No files found".
fail-fast: false
matrix:
include:
- name: quickstart
Expand Down Expand Up @@ -541,6 +553,19 @@ jobs:
with:
persist-credentials: false

- name: Free runner disk
# The e2e stack (CI container, project image, k3d node volumes, local
# rebuilds) needs tens of GB; runners have been observed dipping below
# the k3d node's kubelet eviction threshold mid-run (28624646437).
# Drop the biggest preinstalled toolchains we never use.
run: |
df -h / | tail -1
sudo rm -rf /usr/local/lib/android /opt/hostedtoolcache/CodeQL /usr/share/dotnet &
RM_PID=$!
docker system prune -f >/dev/null 2>&1 || true
wait "${RM_PID}"
df -h / | tail -1

- name: Download release bundle artifact
if: matrix.needs_artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
Expand All @@ -557,7 +582,10 @@ jobs:

- name: Load CI container image (PR)
if: github.event_name == 'pull_request'
run: docker load -i ci-container-image.tar
# rm after load: the multi-GB tarball otherwise stays on disk for the whole
# job — e2e runners have been observed within 1.5GB of the kubelet eviction
# threshold (run 28623232380).
run: docker load -i ci-container-image.tar && rm -f ci-container-image.tar

- name: Login to Docker registry
if: github.event_name != 'pull_request'
Expand All @@ -576,7 +604,7 @@ jobs:
# Make PROJECT_IMAGE available in the runner's Docker daemon so the k3d
# image import (load delivery) finds it locally instead of pulling.
if: github.event_name == 'pull_request'
run: docker load -i project-image.tar
run: docker load -i project-image.tar && rm -f project-image.tar

- name: Start disk usage sampler
run: |
Expand Down
20 changes: 18 additions & 2 deletions hack/e2e/deploy-controller.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,5 +75,21 @@ else
kubectl --context "${CTX}" -n "${NAMESPACE}" \
rollout restart "deployment/${deploy}"
fi
kubectl --context "${CTX}" -n "${NAMESPACE}" \
rollout status "deployment/${deploy}" --timeout=180s
# 300s: a fresh-namespace install waits on cert-manager issuing the admission
# cert before the pod can mount it, and CI runners near their disk-eviction
# threshold start pods slowly; 180s was too tight in the tail (CI run
# 28623232380). On timeout, dump the state that explains *why* the pod never
# became ready — node conditions expose DiskPressure, describe/events expose
# mount and scheduling problems — so a recurrence is diagnosable from the log.
if ! kubectl --context "${CTX}" -n "${NAMESPACE}" \
rollout status "deployment/${deploy}" --timeout=300s; then
echo "=== rollout timed out; dumping diagnostics ===" >&2
kubectl --context "${CTX}" get nodes \
-o custom-columns='NAME:.metadata.name,CONDITIONS:.status.conditions[?(@.status=="True")].type' >&2 || true
kubectl --context "${CTX}" -n "${NAMESPACE}" get deploy,rs,pods -o wide >&2 || true
kubectl --context "${CTX}" -n "${NAMESPACE}" describe pods \
-l "${CONTROLLER_DEPLOY_SELECTOR}" >&2 || true
kubectl --context "${CTX}" -n "${NAMESPACE}" get events \
--sort-by=.metadata.creationTimestamp >&2 || true
exit 1
fi
11 changes: 11 additions & 0 deletions test/e2e/cluster/start-cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,17 @@ create_cluster() {
"--kube-apiserver-arg=audit-webhook-batch-max-size=10@server:0"
"--kube-apiserver-arg=max-requests-inflight=${KUBE_APISERVER_MAX_REQUESTS_INFLIGHT}@server:0"
"--kube-apiserver-arg=max-mutating-requests-inflight=${KUBE_APISERVER_MAX_MUTATING_REQUESTS_INFLIGHT}@server:0"
# k3s defaults kubelet eviction to nodefs.available<5% — on a CI runner's
# ~72GB disk that is ~3.6GB, so the node reports DiskPressure and evicts
# the controller while gigabytes are still free (CI run 28624646437). The
# k3d node shares the host filesystem, so percentage thresholds don't
# express "actually almost full"; use absolute ones, and drop the
# pressure condition quickly (default transition period is 5m) so a
# recovered node schedules pods again within the rollout wait.
"--kubelet-arg=eviction-hard=nodefs.available<1Gi,imagefs.available<1Gi@server:*"
"--kubelet-arg=eviction-hard=nodefs.available<1Gi,imagefs.available<1Gi@agent:*"
"--kubelet-arg=eviction-pressure-transition-period=30s@server:*"
"--kubelet-arg=eviction-pressure-transition-period=30s@agent:*"
)

echo "🚀 Creating k3d cluster '${CLUSTER_NAME}' with audit webhook support..."
Expand Down