Repository navigation
Conversation
Move the eight end-to-end tests (upgrade, backup-restore, three backup-schedule variants, vtorc-vtadmin, unmanaged-tablet, hpa) from the Buildkite public queue onto GitHub Actions using ubuntu-latest-8-cores runners. Each test runs as its own matrix job, in parallel, on a fresh VM per job — which lets us drop Buildkite's per-test concurrency gate since the collisions it was guarding against (shared vitess-operator-pr image tag, fixed localhost port-forward ports, shared kind docker network on sibling-container agents) no longer exist when each job gets its own runner and Docker daemon. utils.sh loses the BUILDKITE_JOB_ID coupling: the variable is renamed to CI_JOB_ID (set from github.run_id + run_attempt + matrix target), the sibling-container networking hack in setupKubectlAccessForCI is removed because kind now runs directly on the runner host, and the docker build progress flag keys off \$CI instead. The pre-exit hook that reset perms and cleaned up the shared Docker state is no longer needed — GHA runners are ephemeral. Branch protection on main and release-** will need the new e2e check names added and the Buildkite checks removed once the first run is green. Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
There was a problem hiding this comment.
Pull request overview
Migrates the repository’s end-to-end CI coverage from Buildkite to GitHub Actions by introducing a new GHA matrix workflow and removing Buildkite-specific pipelines/hooks, while adjusting the e2e test harness to be CI-agnostic.
Changes:
- Added a new GitHub Actions workflow that runs 8 e2e tests as parallel matrix jobs.
- Updated e2e test utilities to use a generic
CI_JOB_IDand removed Buildkite-only kubectl/kind networking setup. - Removed Buildkite pipeline/hook files and updated release documentation to remove a stale Buildkite reference.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
test/endtoend/utils.sh |
Replaces Buildkite-specific job ID handling with CI_JOB_ID, removes Buildkite-only kubectl access workaround, and tweaks CI log output behavior. |
docs/release-process.md |
Rewords a Buildkite-specific note to refer generically to end-to-end tests. |
.github/workflows/e2e-test.yaml |
Introduces the new GHA matrix workflow to run all e2e test targets in parallel. |
.buildkite/pipeline.yml |
Deletes the old Buildkite pipeline configuration. |
.buildkite/hooks/pre-exit |
Deletes the old Buildkite pre-exit cleanup/permission workaround hook. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| CI_JOB_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.test.target }} | ||
| steps: | ||
| - name: Check out code | ||
| uses: actions/checkout@v6 |
There was a problem hiding this comment.
actions/checkout is referenced by a mutable tag (@v6). For supply-chain safety and reproducibility, pin this action to a specific commit SHA (similar to how actions/setup-go is pinned) so workflow runs can't change behavior unexpectedly if the tag is moved.
Suggested change
| uses: actions/checkout@v6 | |
| uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # pinned for supply-chain safety |
Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Two issues surfaced on the first GHA run: 1. The `Backup Schedule vtctldclient Method Test` failed kind cluster creation with `sethostname: invalid argument` because the container hostname `kind-<CI_JOB_ID>-control-plane` exceeded the Linux 64-char HOST_NAME_MAX. The `github.run_id`+`run_attempt`+target scheme I used was overkill anyway — each GHA job runs in its own ephemeral VM, so there's no collision risk from reusing a short cluster name. Shorten CI_JOB_ID to just the matrix target. 2. The `Unmanaged Tablet Test` failed at `apt install chromium-browser` when the runner's snap layer hit an apparmor error on mesa-2404. That test doesn't even need chromium — only `vtorc-vtadmin-test` uses headless chromium. Narrow the install to only that matrix entry and route through the well-maintained `browser-actions/setup-chrome` action, aliasing the chrome binary to `chromium-browser` so the existing `getChromiumBinaryName` discovery in utils.sh still works. Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Every e2e test running under the new GHA runner group failed with mysqld pods in CrashLoopBackOff. The pattern (mysqld exiting ~17ms after spawn, before producing any InnoDB output) and the runner image tag (ubuntu-24.04 / Noble) point to the AppArmor userns restriction that Ubuntu 23.10 introduced: kernel.apparmor_restrict_unprivileged_userns=1 (default) This blocks processes inside nested containers from creating their own user namespaces, which mysqld depends on during startup. Buildkite's public queue runs on an older base, so it doesn't hit this. Workaround is the standard one for kind-in-Ubuntu-24.04 CI: sysctl the restriction off at the start of the job. Also bump inotify limits, which kind wants for its file watchers once a cluster has more than a couple of pods running. Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Previous apparmor_restrict_unprivileged_userns=0 alone didn't unblock mysqld inside the vttablet pods — 6 of 8 tests still CrashLoopBackOff with the same my.cnf symptom after kind comes up fine. Notably Unmanaged Tablet Test passed (the one test that does not run a vitess-operator-managed mysqld inside the cluster), which pins the remaining breakage on something specific to mysqld-in-nested-container. Ubuntu 24.04 ships multiple layers of restriction on unprivileged user namespaces plus a broader AppArmor profile for Docker. Belt-and- suspenders: also clear apparmor_restrict_unprivileged_unconfined, enable unprivileged_userns_clone, raise user.max_user_namespaces, and tear AppArmor down entirely. The runner VM is ephemeral, so neutering AppArmor for the job has zero blast radius. Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Stopping apparmor.service and running aa-teardown in the previous commit unloads all AppArmor profiles from the kernel, including the docker-default profile that BuildKit applies to build containers. The operator image build then died with `runc run failed ... unable to apply apparmor profile`, vitess-operator-pr:latest never got built, kind load fell through, and every pod sat at ErrImageNeverPull. Keep the userns sysctls (they were the actual target of the fix) and leave the AppArmor service and profiles alone so Docker keeps working. Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
Previous attempt to neuter AppArmor broke the docker build because
BuildKit requires the docker-default profile to launch build
containers. Previous attempt to keep AppArmor on while just setting
userns sysctls left mysqld still crashing in nested pods (same
my.cnf symptom, 5 of 8 e2e tests red).
Split the work across two workflow steps:
1. Build operator image — runs with AppArmor still up so BuildKit
is happy and produces vitess-operator-pr:latest.
2. Disable AppArmor before kind — stops apparmor.service and runs
aa-teardown now that no more docker builds need to happen.
setupBuildContainerImage in utils.sh gains a docker-image-inspect
short-circuit so the in-test build call is a no-op when CI already
pre-built the image. Locally (no pre-built image) it still builds as
before.
Signed-off-by: Nick Van Wiggeren <nick@planetscale.com>
maxenglander
approved these changes
Jul 15, 2026
mhamza15
added a commit
that referenced
this pull request
Jul 15, 2026
vttablet pods crashloop on the GitHub Actions runners: Ubuntu 24.04's AppArmor docker-default profile and unprivileged-userns restrictions kill mysqld in the nested vttablet -> kind -> docker stack, which Buildkite's older kernels never enforced. Validated in PR #781 on the same runners. Relaxes the userns sysctls and raises kind's inotify limits, then builds the operator image before tearing AppArmor down (BuildKit can't start build containers once the profile is unloaded). setupBuildContainerImage now skips the build when the image already exists so the make target reuses the pre-built image. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
Contributor
|
Built on top of this: #805 |
mhamza15
added a commit
that referenced
this pull request
Jul 16, 2026
* Migrate end-to-end tests to GitHub Actions The end-to-end suite still depends on Buildkite-specific workers and sibling-container networking. This replaces the eight Buildkite steps with a parallel GitHub Actions matrix on 16-core Depot runners, removes the obsolete agent setup, and keeps the test tool versions pinned across CI hosts. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Update pinned vtctldclient to Vitess 24.0.0-rc1 The pinned vtctldclient was still 21.0.0-rc1 (from October 2024) while the e2e clusters run `vitess/lite:v24.0.0-rc1`, so the client was driving servers three major versions ahead of it. Bumps the pin and its checksum to match the server images. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * use same version for kube as kubectl Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Surface pod-startup failures in e2e jobs before the job timeout Every e2e job hung waiting for the vitess-operator pod and was killed by GitHub with no diagnostics: `checkPodStatusWithTimeout` waits up to 20 minutes before dumping pod state and exiting, but `timeout-minutes` was also 20, so the job died a few minutes before the helper's error path could run. Raises the job timeouts above the helper's wait and adds a failure-only step that dumps pod status, recent events, pod descriptions and operator logs while the kind cluster is still alive. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Rename default kind cluster so it can't match pod-wait regexes Every e2e job hung waiting for the vitess-operator pod even though it was Running. The kind cluster was renamed to "vitess-operator" in this branch, which makes the node (and the static kube-system pods) embed that string: `kube-apiserver-vitess-operator-control-plane` and three siblings all match `checkPodStatusWithTimeout`'s `vitess-operator(.*)1/1(.*)Running` regex, so the helper counts 5 matches instead of the expected 1 and never succeeds. Defaults `KIND_CLUSTER_NAME` to "vtop-e2e", which is not a substring of any regex the tests wait on. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Drop explanatory comment on KIND_CLUSTER_NAME default Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Work around Ubuntu 24.04 AppArmor killing mysqld inside kind vttablet pods crashloop on the GitHub Actions runners: Ubuntu 24.04's AppArmor docker-default profile and unprivileged-userns restrictions kill mysqld in the nested vttablet -> kind -> docker stack, which Buildkite's older kernels never enforced. Validated in PR #781 on the same runners. Relaxes the userns sysctls and raises kind's inotify limits, then builds the operator image before tearing AppArmor down (BuildKit can't start build containers once the profile is unloaded). setupBuildContainerImage now skips the build when the image already exists so the make target reuses the pre-built image. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Dump per-pod logs in the e2e failure diagnostics The vtorc pod crashloops with exit code 1 but the failure dump only captured pod listings, events and descriptions, none of which say why. Dumps current and previous container logs for every pod in the example namespace, and fixes the operator log dump to use the default namespace the operator actually deploys into. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * vtorc: add required `--cell` flag Current VTOrc builds reject startup when `--cell` is absent, leaving the pod in CrashLoopBackOff. This passes the tablet pool cell already stored in `vtorc.Spec` to the VTOrc command. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Update end-to-end-test.yaml Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Fix end-to-end test dependency installation The dependency script installs kind and vtctldclient in top-level blocks, while kubectl uses a named helper. The VTAdmin job also installs Ubuntu's snap-based Chromium launcher, which stops working after the workflow tears down AppArmor. This adds matching installer functions for kind and vtctldclient. It also installs a normal Chrome package for the VTAdmin job and exposes it as chromium before AppArmor is disabled. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Apply suggestions from code review Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> --------- Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
mattlord
pushed a commit
that referenced
this pull request
Sep 2, 2026
* Migrate end-to-end tests to GitHub Actions The end-to-end suite still depends on Buildkite-specific workers and sibling-container networking. This replaces the eight Buildkite steps with a parallel GitHub Actions matrix on 16-core Depot runners, removes the obsolete agent setup, and keeps the test tool versions pinned across CI hosts. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Update pinned vtctldclient to Vitess 24.0.0-rc1 The pinned vtctldclient was still 21.0.0-rc1 (from October 2024) while the e2e clusters run `vitess/lite:v24.0.0-rc1`, so the client was driving servers three major versions ahead of it. Bumps the pin and its checksum to match the server images. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * use same version for kube as kubectl Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Surface pod-startup failures in e2e jobs before the job timeout Every e2e job hung waiting for the vitess-operator pod and was killed by GitHub with no diagnostics: `checkPodStatusWithTimeout` waits up to 20 minutes before dumping pod state and exiting, but `timeout-minutes` was also 20, so the job died a few minutes before the helper's error path could run. Raises the job timeouts above the helper's wait and adds a failure-only step that dumps pod status, recent events, pod descriptions and operator logs while the kind cluster is still alive. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Rename default kind cluster so it can't match pod-wait regexes Every e2e job hung waiting for the vitess-operator pod even though it was Running. The kind cluster was renamed to "vitess-operator" in this branch, which makes the node (and the static kube-system pods) embed that string: `kube-apiserver-vitess-operator-control-plane` and three siblings all match `checkPodStatusWithTimeout`'s `vitess-operator(.*)1/1(.*)Running` regex, so the helper counts 5 matches instead of the expected 1 and never succeeds. Defaults `KIND_CLUSTER_NAME` to "vtop-e2e", which is not a substring of any regex the tests wait on. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Drop explanatory comment on KIND_CLUSTER_NAME default Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Work around Ubuntu 24.04 AppArmor killing mysqld inside kind vttablet pods crashloop on the GitHub Actions runners: Ubuntu 24.04's AppArmor docker-default profile and unprivileged-userns restrictions kill mysqld in the nested vttablet -> kind -> docker stack, which Buildkite's older kernels never enforced. Validated in PR #781 on the same runners. Relaxes the userns sysctls and raises kind's inotify limits, then builds the operator image before tearing AppArmor down (BuildKit can't start build containers once the profile is unloaded). setupBuildContainerImage now skips the build when the image already exists so the make target reuses the pre-built image. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Dump per-pod logs in the e2e failure diagnostics The vtorc pod crashloops with exit code 1 but the failure dump only captured pod listings, events and descriptions, none of which say why. Dumps current and previous container logs for every pod in the example namespace, and fixes the operator log dump to use the default namespace the operator actually deploys into. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * vtorc: add required `--cell` flag Current VTOrc builds reject startup when `--cell` is absent, leaving the pod in CrashLoopBackOff. This passes the tablet pool cell already stored in `vtorc.Spec` to the VTOrc command. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Update end-to-end-test.yaml Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Fix end-to-end test dependency installation The dependency script installs kind and vtctldclient in top-level blocks, while kubectl uses a named helper. The VTAdmin job also installs Ubuntu's snap-based Chromium launcher, which stops working after the workflow tears down AppArmor. This adds matching installer functions for kind and vtctldclient. It also installs a normal Chrome package for the VTAdmin job and exposes it as chromium before AppArmor is disabled. Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> * Apply suggestions from code review Signed-off-by: Mohamed Hamza <mhamza@fastmail.com> --------- Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the eight end-to-end tests (upgrade, backup-restore, three backup-schedule variants, vtorc-vtadmin, unmanaged-tablet, hpa) off the Buildkite
publicqueue and onto GitHub Actions. Each test runs as a matrix job onubuntu-latest-8-cores(8 vCPU / 32 GB), all in parallel, with no per-test concurrency gate.Why
Buildkite's agent model put every job inside a
docker:latestcontainer talking to a shared host Docker daemon, which forced a lot of incidental complexity on us: a 15-line Alpine bootstrap in every step to install Go from a tarball, a sibling-container networking hack to makekubectlreachkind, apre-exithook that ransudo fix-buildkite-agent-builds-permissionsto undo chown damage from the operator container mutating the shared checkout, and aconcurrency: 1gate per test to keep two jobs on the same agent from fighting over the hardcodedvitess-operator-pr:latestimage tag and the fixed15306/14001/15999port-forward ports.On GitHub-hosted runners every job gets a fresh VM with its own Docker daemon and its own localhost, so all four problems evaporate for free. The Alpine bootstrap becomes
actions/setup-go, the sibling-container hack is deleted, the permission fix-up is unnecessary on ephemeral runners, and the concurrency gate is gone because the resources the tests share on Buildkite aren't shared on GHA.Changes
.github/workflows/e2e-test.yaml— matrix over the 8 test targets,fail-fast: false, 40-minute timeout per job, installsmysql-client+chromium-browser(the latter is only needed byvtorc-vtadminbut harmless to preinstall everywhere).test/endtoend/utils.shBUILDKITE_JOB_ID→CI_JOB_ID(still defaults to0somake upgrade-teston a laptop is unchanged). The workflow sets it to${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.test.target }}.setupKubectlAccessForCIentirely — the Buildkite-specificdocker network connect kind+kind get kubeconfig --internaldance is not needed when kind runs directly on the runner host.docker build --progress plainbranch now keys off$CI(GHA sets this automatically)..buildkite/pipeline.ymland.buildkite/hooks/pre-exit.docs/release-process.md— updated one stale "buildkite" reference.Follow-ups (outside this PR)
ubuntu-latest-8-coresresolves — if the org hasn't provisioned larger runners, the first run will fail fast and we can swap the label toubuntu-latest(4 vCPU / 16 GB) or whatever larger label is available.mainandrelease-**to require the newe2ematrix check names and drop the Buildkite required checks.