Add DRA version bump automation by Apmats · Pull Request #3986 · elastic/connectors

Apmats · 2026-04-03T08:51:43Z

Includes and builds on #3951. Replaces the block step with automated version bump logic.

The centralized release-eng pipeline triggers connectors-version-bump with NEW_VERSION, BRANCH, and WORKFLOW. For patch, the script opens an auto-merging PR bumping both VERSION files on the release branch. For minor, it updates pipeline.yml's DRA branch list, creates the new release branch, and bumps main to the next minor -- each step idempotent. The merged commits trigger existing CI which publishes DRA artifacts, and the json-watcher steps confirm completion.

Requires elastic/terrazzo#1190 for pull_requests: write permission. Set DRY_RUN=true to preview without side effects.

Note to reviewer: we should port this to 9.2, 9.1, and 8.19 unless we explicitly decide to only do this for 9.3 onwards. Those branches have a single VERSION file (pre-monorepo layout) so this can't be backported as-is -- I'll work on bringing a separate PR to 9.2 to be backported.

Test plan

Local: shell syntax, pipeline YAML, Python branch-insertion cases, env var validation, DRY_RUN for both workflows
Buildkite: DRY_RUN trigger, patch PR (close without merging), minor flow with fake branch, end-to-end with @ninalee12

Replace the block step placeholder with an automated version bump script that handles both patch and minor workflows. Fix Slack notification channel and catalog-info.yaml trailing newline. Split DRA artifact polling into separate staging (conditional) and snapshot steps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace fragile sed-based DRA branch list update with a Python helper that uses the ci:packaging condition as an anchor point, is idempotent, and fails clearly if the expected pipeline.yml structure changes. Add DRY_RUN=true support to preview all operations without committing or pushing. Add push_with_retry to handle concurrent commits landing on main during the minor workflow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace direct pushes with gh pr create + gh pr merge --auto --squash so version bump changes go through PRs, matching the expected pattern across other teams in the unified release process. Add ci:version-bump label fast path to pipeline.yml so version bump PRs skip the full CI suite (notice, lint, tests, packaging) and only run a lightweight gate step. Increase version-bump pipeline timeout to 90min to accommodate waiting for PR CI and merge. Requires pull_requests: write in the GithubPermissionSet (elastic/terrazzo#1190). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add idempotency checks so re-runs skip already-completed steps. Clean up review feedback: build link in PR body, proper file handling in Python, remove dead checkouts, gate token check on DRY_RUN. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Buildkite expression language doesn't support `not` as a prefix operator. Use `!(... includes ...)` instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

markjhoy · 2026-04-23T12:44:28Z

Overall - looking pretty good.

Since this script is manually launched from Buildkite, I'm also wondering if it should also bump the DRA version numbers for minor branches here in the main Buildkite file?

Apmats · 2026-04-23T15:41:36Z

@markjhoy the buildkite/publish/version-bump.sh script updates that exact line in pipeline.yml to add the new branch to the DRA condition before creating the release branch (see the Python branch-insertion logic in there). That way I hope DRA publishing works on the new branch from day one instead of needing a manual follow-up.

markjhoy · 2026-04-23T16:38:25Z

@markjhoy the buildkite/publish/version-bump.sh script updates that exact line in pipeline.yml to add the new branch to the DRA condition before creating the release branch (see the Python branch-insertion logic in there). That way I hope DRA publishing works on the new branch from day one instead of needing a manual follow-up.

Ah - I missed that! Thanks for pointing it out.

Defaults to true (preserves production behavior). When false, the script creates the PR but skips enabling auto-merge and the merge-wait poll, leaving the PR open for inspection. Useful for testing the real PR creation path in Buildkite without landing changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Apmats · 2026-04-24T09:58:01Z

Hi, this is Claude (driving Apostolos's terminal) -- we added an AUTO_MERGE env var to .buildkite/publish/version-bump.sh. Defaults to true so production behavior is unchanged. Setting AUTO_MERGE=false creates the PR but skips enabling auto-merge and the merge-wait poll, so PRs stay open for inspection and manual close. Useful for verifying the real PR creation path in Buildkite without landing changes.

Apmats · 2026-04-24T10:17:25Z

@markjhoy if you're okay with the above, you can give me a ✅ and I'll move onto testing.

markjhoy · 2026-04-24T12:21:22Z

  DRA_PYTHON_VERSION: "3.11"

 steps:
+  # Version bump PRs only touch VERSION files -- skip the full CI suite.


If we're checking the labels for each step below - we don't really need this step to just print out that we're skipping CI... I'd remove this block

The more I look at this - since the catalog-info.yaml changes already call ".buildkite/version-bump-pipeline.yml" instead of using the main pipeline, I don't think it's necessary to change this main pipeline file at all... @seanstory - you probably have a bit more context here with the CI, thoughts?

Not sure I've got more context. But I do question if we actually want to skip CI on version bump. I think we probably don't. Our CI is not that long, and if there's something that breaks, I'd rather it break on the branch than on main.

The only concern and why I did this is that we don't control the timeout set on the Buildkite trigger step that calls us to do this bump. I agree it's better to break on the PR - but adding ~X min per PR could eventually push us past whatever timeout release-eng set on their side, and we'd have to figure out what's happening to adjust.
This matters more for minor bumps, where the script opens two PRs sequentially (one to add the new branch to pipeline.yml's DRA list on main, then a second to bump main to the next minor after cutting the release branch).

I can change this if you guys are sure this is preferable.

Did they provide explicit guidance that CI should be skipped and that they have a timeout we need to worry about? If they did, I'll bow to your judgement. But if they haven't explicitly asked us to disable CI, I'd rather not.

Actually dug out the following guidance - in PSI Version Bump Automation

*Team pipelines should aim to achieve within a maximum of 2–3 hours (including DRA builds), when full automation is in place, to ensure an efficient full version bump process *Teams should ideally configure their version bump PRs to skip ITs and E2E tests to reduce the duration of the process; version bump PRs only increment the version number (displayed with --version flag), presenting low risk of introducing critical bugs, and if bugs are introduced, DRA artifacts can be published and fixed in a follow-up DRA build to unblock dependent teams

Running the tests X times as we do means we might end up pushing close to the suggested 2-3 hours. I know guidance says "ITs and E2E" but I think that's shorthand for "non-instant tests".

@markjhoy I don't think skipping tests gives up any real safety though - DRA publishing is gated on tests passing, it just means we will have to revert from a protected branch which is a hassle.

Updated thoughts? 😁

Our builds on main reliably take about 30 min, that that includes a packaging phase that won't run on PRs. We should not be anywhere near the 2-3 hour mark.

I expect that guidance is more applicable for monorepos like elasticsearch/cloud/kibana that have obscene numbers of tests.

Should this be a fully separate pipeline instead that calls non-conditionally all tests/ftests?

Two separate things.
One pipeline creates the version bump as a PR.

This generated PR runs our normal pipeline. And I maintain should run it normally.

Alright- undid this.

I got Claude to look at 50 past PR and build runs on buildkite, run a simulation and give me p values for the total run time to run tests 2 or 3 times depending on the flow.

Scenario Mean P50 P90 P95 P99

Patch (PR + triggered build) 48.8 47.3 65.6 73.8 89.0

Minor (PR1 + PR2 + triggered build) 71.6 69.8 94.9 104.5 120.9

So for now we're mostly safely under 2 hours.

If our times grow we can look at skipping tests at some point in this process while keeping them in the PR for revert-aversion 😁

markjhoy · 2026-04-24T12:33:37Z

+with open(pipeline_yml) as f:
+    pipeline = f.read()
+
+dra_marker = "ci:packaging"


It might be safer to search for the comment # Add new maintenance branches here as well as a comment in the main .yml file telling people to keep the comment as-is...) as who knows what might happen in the future with this script (we might add a step in that meets these conditions 🤷 I tend to err on the side of caution)

Had claude add a unit test for this then realized that it doesn't actually work because it needs to be discovered. Followed what you suggested. 👍

markjhoy · 2026-04-24T12:35:09Z

+  python3 -c '
+import sys
+
+pipeline_yml = sys.argv[1]


(sorry about the last comment I made here --- just realized this was within a python code block 🤦 ) - ignore the (now deleted) last comment I had

markjhoy · 2026-04-24T12:39:48Z

+  local version="$1"
+  local current
+  current=$(cat "${VERSION_FILES[0]}" 2>/dev/null || echo "")
+  [[ "${current}" == "${version}" ]]


Will it be enough to check to equality here? Or should this check be to ensure that the current version is >= to the version? It's a bit more work to separate / check the version parts as numeric, but might be safer.

Good call, Claude added a version_is_downgrade check here.

markjhoy · 2026-04-24T12:43:17Z

+
+insertion = f"build.branch == {escaped}{new_branch}{escaped} || "
+lines[target_idx] = line.replace(pr_label_marker, insertion + pr_label_marker)
+


Would we want to consider removing older branches on this line at some point? IIRC - we only need to keep the last two minors of a major

I don't know if this is a stable thing historically, and if we want to encode it in the code.
I trust you know better, so if you tell me it makes sense I can do it, but I would myself try to contain the changes.

markjhoy · 2026-04-24T12:45:09Z

@markjhoy if you're okay with the above, you can give me a ✅ and I'll move onto testing.

I think it's good... I'd like to get a second set of eyes on it (maybe @artem-shelkovnikov ) though, especially around the questions I have about needing to edit the primary pipeline.yml file...

Require both the "ci:packaging" token and the "# Add new maintenance branches here" trailing comment when locating the DRA condition line. Prevents accidental rewrites if a future `if:` condition elsewhere in pipeline.yml happens to reference "ci:packaging". Also add a note above the target line reminding editors that the trailing comment is load-bearing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

artem-shelkovnikov · 2026-04-24T17:48:58Z

+  echo "Adding ${new_branch} to DRA branch list in pipeline.yml"
+
+  python3 -c '
+import sys


We can just make it a separate script file?

Also, we can theoretically make this whole script python if needed.

++ to this - might make it easier to maintain (at least the inline Python script here - I'm OK with the primary pipeline being in Bash)

Done. Tests not added because I'm unsure when they would run and how. If you have ideas tell me.

Per review feedback (Sean, Mark), don't skip CI on version bump PRs. Without the bypass, PR CI runs the full test suite and catches breakage before the change lands on a protected release branch, instead of relying on post-merge CI that doesn't even auto-trigger on release-branch pushes (confirmed empirically: release branches are only built via daily schedule or manual UI trigger, never via webhook on push). Remove: - `ci:version-bump` gate step in pipeline.yml - `!(build.pull_request.labels includes "ci:version-bump")` conditions gating notice/lint/unit_tests/smoke/ftests/test_packages - `--label ci:version-bump` on gh pr create in version-bump.sh Bump timeouts to accommodate full PR CI (~30 min each): - Version-bump step timeout: 90 -> 180 minutes - PR_MERGE_TIMEOUT: 3600 -> 5400 seconds (60 -> 90 minutes) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Apmats · 2026-04-24T22:23:12Z

@markjhoy @seanstory @artem-shelkovnikov addressed most things I think - what we're not covering is this part of the RM duty on minor versions:

Multiple pipeline schedules in catalog-info.yaml
.backportrc.json
renovate.json

Worth trying to take care of these as well?

seanstory · 2026-04-27T14:52:14Z

Let's get this in for now, and we can tackle those later if necessary. I don't personally think that the RM duties are too heavy right now, the motivation for this is more about the release team not being blocked by our convenience, and none of those that you list matter to the release team, so they can still wait on our convenience.

Apmats · 2026-04-28T12:46:30Z

@markjhoy @artem-shelkovnikov let me know if anything important remains unadddressed

markjhoy

👍 I think this is good for now. As @seanstory mentioned, we can address the other items later after we get this initial piece in.

`gh pr view <branch>` matches any state, so on re-runs with a recycled branch name it returns a stale merged PR and the script then fails trying to auto-merge it. Made-with: Cursor

Missing `${BRANCH}` / `${NEW_VERSION}` interpolate to empty strings at upload time and surface only as a 4-hour json-watcher timeout. Guard up front so the failure is immediate and obvious. Made-with: Cursor

Apmats · 2026-04-28T15:42:30Z

Merging and moving to dry run tests - will coordinate for end to end later.

github-actions · 2026-04-28T15:43:03Z

💔 Failed to create backport PR(s)

The backport operation could not be completed due to the following error:
There are no branches to backport to. Aborting.

The backport PRs will be merged automatically after passing CI.

To backport manually run:
backport --pr 3986 --autoMerge --autoMergeMethod squash

## Summary Adds support for pre-9.3 release branches (8.19, 9.0, 9.1, 9.2) in the version-bump script introduced in #3986. The original PR's description noted that pre-monorepo branches (single `connectors/VERSION` file at a different path) couldn't be supported as-is and would need a separate backport. That turns out not to be necessary: the script always runs from `main`'s checkout (the `connectors-version-bump` Buildkite pipeline uploads `version-bump-pipeline.yml` from `main`, regardless of the target release branch), so a single `case` on `${BRANCH}` in the on-`main` script is sufficient. No backport, no per-branch script copies. I had originally assumed the script ran from each release branch's checkout, which is why #3986 left this as a follow-up. After verifying how CI actually invokes the script, this turned out to be a one-file change. ## Changes In `.buildkite/publish/version-bump.sh`, replace the hardcoded `VERSION_FILES` array with a `case` on `${BRANCH}`: - `8.*`, `9.0`, `9.1`, `9.2` → `connectors/VERSION` (pre-monorepo single-file layout) - everything else → the two post-monorepo paths (current behavior, unchanged for 9.3+/main) Made with [Cursor](https://cursor.com) --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

## Summary Adds `release-eng: BUILD_AND_READ` to the `connectors` pipeline's teams block so release-eng team members can trigger builds. ## Why The `connectors-version-bump` pipeline (introduced in #3986) executes a `trigger:` step that starts a DRA build on the target release branch on the main `connectors` pipeline. Buildkite propagates the original triggering user's identity through trigger chains, so the downstream build on `connectors` is created as that user -- and the user needs Build access on `connectors` for the trigger to succeed. We already granted release-eng `BUILD_AND_READ` on `connectors-version-bump` (so they can trigger our pipeline at all), but missed adding it on the main `connectors` pipeline. This caused failures in [build #3](https://buildkite.com/elastic/connectors-version-bump/builds/3) (9.3.5) and [build #4](https://buildkite.com/elastic/connectors-version-bump/builds/4) (8.19.16) when triggered by Nina Lee, while [build #2](https://buildkite.com/elastic/connectors-version-bump/builds/2) (9.4.1) triggered by Julien Mailleret succeeded -- presumably via individual grants. This grants team-level access so any release-eng member can complete the version bump end-to-end. ## Test plan - [ ] After merge and Backstage sync (~few minutes), re-trigger build #3 or #4 from the centralized pipeline and confirm the `Trigger DRA build on ${BRANCH}` step succeeds --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

ninalee12 and others added 6 commits February 23, 2026 14:40

add version bump pipeline to catalog

b0901f4

add version bump pipeline

29e651b

Update NOTICE.txt

c2706f3

fix name, update pipeline setting

5369903

reduce agent footprint

6054327

update slack channel

a03bdb9

github-actions Bot added auto-backport v9.4.0 labels Apr 3, 2026

Apmats force-pushed the am/version-bump-automation branch from b10ae5a to 9766b58 Compare April 3, 2026 09:00

elasticmachine and others added 2 commits April 3, 2026 09:02

Update NOTICE.txt

6a4f322

Apmats mentioned this pull request Apr 9, 2026

Add DRA version bump pipeline #3951

Closed

Apmats marked this pull request as ready for review April 16, 2026 08:33

Apmats requested a review from a team as a code owner April 16, 2026 08:33

Apmats and others added 5 commits April 16, 2026 11:59

Merge branch 'main' into am/version-bump-automation

026ed48

Fix invalid not includes syntax in pipeline.yml

901d4e3

Buildkite expression language doesn't support `not` as a prefix operator. Use `!(... includes ...)` instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Update NOTICE.txt

d6472ee

markjhoy reviewed Apr 23, 2026

View reviewed changes

Comment thread .buildkite/version-bump-pipeline.yml

Apmats and others added 2 commits April 24, 2026 12:50

Update NOTICE.txt

b7d1708

markjhoy reviewed Apr 24, 2026

View reviewed changes

Apmats force-pushed the am/version-bump-automation branch from eaaee4b to a600739 Compare April 24, 2026 17:47

Update NOTICE.txt

244de23

artem-shelkovnikov reviewed Apr 24, 2026

View reviewed changes

Apmats and others added 3 commits April 24, 2026 22:12

Address review comments

f05e4bd

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Update NOTICE.txt

ac0c097

Apmats requested review from artem-shelkovnikov, markjhoy and seanstory April 28, 2026 12:46

markjhoy approved these changes Apr 28, 2026

View reviewed changes

Apmats and others added 3 commits April 28, 2026 16:50

Restrict version-bump PR lookup to open PRs

e18363b

`gh pr view <branch>` matches any state, so on re-runs with a recycled branch name it returns a stale merged PR and the script then fails trying to auto-merge it. Made-with: Cursor

Validate trigger params in version-bump pipeline

cc0e8f9

Missing `${BRANCH}` / `${NEW_VERSION}` interpolate to empty strings at upload time and surface only as a 4-hour json-watcher timeout. Guard up front so the failure is immediate and obvious. Made-with: Cursor

Update NOTICE.txt

9a6a563

Apmats merged commit 1375593 into main Apr 28, 2026
2 checks passed

Apmats deleted the am/version-bump-automation branch April 28, 2026 15:42

Apmats mentioned this pull request Apr 30, 2026

Support pre-monorepo branches in version-bump script #4001

Merged

Apmats mentioned this pull request May 6, 2026

Grant release-eng build access to connectors pipeline #4004

Merged

1 task

Scenario	Mean	P50	P90	P95	P99
Patch (PR + triggered build)	48.8	47.3	65.6	73.8	89.0
Minor (PR1 + PR2 + triggered build)	71.6	69.8	94.9	104.5	120.9


		insertion = f"build.branch == {escaped}{new_branch}{escaped} \|\| "
		lines[target_idx] = line.replace(pr_label_marker, insertion + pr_label_marker)

Conversation

Apmats commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Uh oh!

Uh oh!

markjhoy commented Apr 23, 2026

Uh oh!

Apmats commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markjhoy commented Apr 23, 2026

Uh oh!

Apmats commented Apr 24, 2026

Uh oh!

Apmats commented Apr 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markjhoy commented Apr 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markjhoy Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apmats commented Apr 24, 2026

Uh oh!

seanstory commented Apr 27, 2026

Uh oh!

Apmats commented Apr 28, 2026

Uh oh!

markjhoy left a comment

Choose a reason for hiding this comment

Uh oh!

Apmats commented Apr 28, 2026

Uh oh!

Uh oh!

github-actions Bot commented Apr 28, 2026

💔 Failed to create backport PR(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Apmats commented Apr 3, 2026 •

edited

Loading

Apmats commented Apr 23, 2026 •

edited

Loading

markjhoy Apr 24, 2026 •

edited

Loading