Skip to content

chore(gha): parallelize provider binding generation for documentation examples - #244

Merged
jsteinich merged 2 commits into
open-constructs:mainfrom
X-Guardian:chore/parallelize-doc-example-binding-generation
Jun 8, 2026
Merged

jsteinich merged 2 commits into
open-constructs:mainfrom
X-Guardian:chore/parallelize-doc-example-binding-generation

Conversation

@X-Guardian

@X-Guardian X-Guardian commented Jun 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

The C#, Java, Java-gradle, and Python *-documentation examples are the critical-path tail of the examples CI matrix — they finish at ~25–29 min while every other example is long done. Almost all of that time is cdktn get generating provider bindings, and it runs fully serially. This PR lets it run in parallel, mirroring how go-documentation is already configured.

Why

Profiling the csharp-documentation job showed cdktn get took 504s of a 691s job — generating + JSII-compiling 6 provider bindings (aws and kubernetes dominate) one at a time. The serialisation is deliberate: these four examples pass --parallelism 1, because cdktn get defaults to --parallelism -1 (all providers concurrently), and the heavy providers need ~13–17 GB peak RAM together — more than a standard ubuntu-latest runner (16 GB) can hold without OOM. go-documentation already side-steps this: it runs on depot-ubuntu-24.04-16 with the default parallelism and finishes the same generation step in ~60s (17.6 GB peak).

So the throttle and the runner choice are two halves of one decision. Lifting the throttle requires the bigger runner, and the bigger runner is wasted while the throttle is in place. This PR changes both together for the four JSII-translated documentation examples.

Approach

  1. Route the JSII-language documentation examples onto Depot runners in examples.yml. The runs-on selector was go-documentation-only (on depot-ubuntu-24.04-16); it now also routes csharp, java, java-documentation-gradle, and python documentation examples to depot-ubuntu-24.04-8. go-documentation stays on -16 (unchanged). typescript-documentation is intentionally excluded — TypeScript bindings aren't JSII-translated, so --parallelism doesn't apply and the job is already cheap (~226s).
  2. Drop --parallelism 1 from the build script of the four examples (csharp/documentation, java/documentation, java/documentation-gradle, python/documentation), letting cdktn get fall back to its -1 default and generate all providers concurrently — matching go-documentation.

Measured impact (on the shipped depot-ubuntu-24.04-8 runners): csharp-documentation's cdktn get runs in 1m13s, down from 8m24s serial — a ~7× speedup — taking the job from ~11m31s to 4m21s. The other three are comparable: java-documentation get 1m24s (job 4m33s), java-documentation-gradle get 1m13s (job 4m29s), python-documentation get 1m08s (job 3m42s) — each down from ~11–13 min to ~3.5–4.5 min. The serial --parallelism 1 builds previously peaked at ~4 GB; running all providers concurrently peaks at 23–26 GB, which would OOM a 16 GB ubuntu-latest runner — confirming the throttle and the runner choice had to move together.

Right-sizing the runner — -8 vs -16: the change was first measured on depot-ubuntu-24.04-16 (16 vCPU / 64 GB), then re-run on depot-ubuntu-24.04-8 (8 vCPU / 32 GB) to avoid over-provisioning. Both work: get is only ~10–13s slower on -8 (fewer cores → slightly less compile concurrency), and peak RAM (23–26 GB) sits within the -8 tier's 32 GB with ~6–9 GB headroom. Per-run variance (~30–60s) exceeds the -8-vs--16 gap — at this provider count cores aren't the binding constraint — so the four examples ship on -8.

Checklist

  • I have updated the PR title to match CDKTN's style guide
  • I have run the linter on my code locally
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation if applicable
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works if applicable
  • New and existing unit tests pass locally with my changes

@X-Guardian
X-Guardian marked this pull request as ready for review June 5, 2026 11:20
@X-Guardian
X-Guardian requested a review from a team as a code owner June 5, 2026 11:20
@jsteinich
jsteinich merged commit 0a9c955 into open-constructs:main Jun 8, 2026
259 checks passed
so0k pushed a commit that referenced this pull request Jul 3, 2026
🤖 Release PR — merge to cut a new release. Kept open and rebased
as commits land on `main`.
---


<details><summary>0.23.4</summary>

##
[0.23.4](v0.23.3...v0.23.4)
(2026-07-02)


### Features

* **cli:** validate Terraform CLI feature versions
([#237](#237))
([32d434c](32d434c))
* declared Terraform/OpenTofu target versions (validation foundation)
([#269](#269))
([0aa2343](0aa2343))
* **lib:** S3-backend - Add `useLockfile` field and update
`dynamodbTable` deprecated JSDoc
([#234](#234))
([32a408f](32a408f))
* **lib:** validate Fn usage against Terraform/OpenTofu versions
([#268](#268))
([cdf8115](cdf8115))


### Bug Fixes

* **cli:** don't mistake terraform error output for a missing-variable
prompt
([#267](#267))
([0ea946f](0ea946f))
* **cli:** release terraform state lock when interrupting diff/deploy
([#284](#284))
([de617db](de617db))
* **cli:** resolve prebuilt Java provider versions via repo1.maven.org
([#285](#285))
([d5af07d](d5af07d))
* **gha:** Adding CHANGELOG.md to prettier ignore list
([#258](#258))
([5160469](5160469))
* **gha:** pull jsii-terraform CI image from GHCR instead of Docker Hub
([#261](#261))
([f19d34b](f19d34b))
* **provider-generator:** copy only publishable files into the jsii
compile bundle
([#292](#292))
([562d4c7](562d4c7))
* **provider-generator:** match shorthand provider sources against
OpenTofu schema keys
([#293](#293))
([fda0103](fda0103))
* **release:** add Changelog title for correct release-please insertion
([#255](#255))
([004818a](004818a))
* **release:** drop devEngines.packageManager that broke npm publish
([ef508a6](ef508a6))
* **tools:** repair generate-function-bindings tool
([#273](#273))
([821e1cf](821e1cf))


### Miscellaneous Chores

* Add Gradle to Mise config
([#271](#271))
([1578270](1578270))
* add pnpm to mise
([#266](#266))
([48ab91f](48ab91f))
* add uv to mise
([#260](#260))
([163cf3d](163cf3d))
* **deps:** bump nrwl/nx-set-shas from
38457b511e60ee4e1da09255b658e480748a1af4 to
afb73a62d26e41464e9254689e1fd6122ee683c1
([#236](#236))
([b5ac018](b5ac018))
* **deps:** bump the github-actions-backward-compatible group across 1
directory with 7 updates
([#235](#235))
([de67d9f](de67d9f))
* **deps:** replace uuid dependency with node:crypto.randomUUID
([#291](#291))
([6d3657a](6d3657a))
* ESLint ignore test edge-provider-bindings
([#270](#270))
([ba407bb](ba407bb))
* Fix Verdaccio logging config
([#272](#272))
([aded64b](aded64b))
* **gha:** parallelize provider binding generation for documentation
examples
([#244](#244))
([0a9c955](0a9c955))
* Migrate package manager from yarn to pnpm
([#170](#170))
([d23da4e](d23da4e))
* migrate planning artifacts to cdktn-planning [skip ci]
([968c358](968c358))
* **release:** add release-please automation for versioning
([#243](#243))
([c759eec](c759eec))
* Remove dependency-pinner tool
([#250](#250))
([42c27d5](42c27d5))
* Remove unused dependencies
([#281](#281))
([c7de65e](c7de65e))
* **tests:** run verdaccio in-process
([#248](#248))
([c17677d](c17677d))
* Update JSII version constraint
([#245](#245))
([d89558f](d89558f))
* Update scheduled workflows to only run on open-constructs/cdk-terrain
([#256](#256))
([ba408cf](ba408cf))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: open-constructs-cdktn[bot] <291052431+open-constructs-cdktn[bot]@users.noreply.github.com>
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

I'm going to lock this pull request because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Jul 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants