Repository navigation
Radius.Compute/containerImages type: add rootless BuildKit sidecar to dynamic-rp chart - #11882
Merged
Merged
Conversation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the deploy/Chart dynamic-rp deployment to optionally run a rootless BuildKit sidecar (default on) and adds supporting documentation/tests, enabling in-cluster image build/push scenarios without relying on a host Docker socket.
Changes:
- Add
dynamicrp.buildkit.*values and wire abuildkitdsidecar +buildctl-initinit container into the dynamic-rp Deployment. - Fix Terraform pre-mount pathing in the chart and add a drift-guard helm-unittest to keep chart/runtime paths aligned.
- Add operator-facing NOTES warnings plus new design/contributor docs for the subsystem.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/recipes/terraform/install.go | Logs an INFO message when no pre-mounted Terraform binary is present (to aid diagnosis). |
| eng/design-notes/recipes/2026-04-container-images-resource-type.md | Adds a design note for the containerImages resource type and how BuildKit is used. |
| docs/contributing/contributing-code/contributing-code-writing/buildkit-recipes.md | Documents the chart’s BuildKit sidecar and the recipe authoring pattern it enables. |
| deploy/Chart/values.yaml | Introduces the dynamicrp.buildkit values surface (enabled/psaMode/image/credentials/resources). |
| deploy/Chart/tests/helpers_test.yaml | Adds helm-unittest coverage for BuildKit enable/disable and Terraform-path drift guard. |
| deploy/Chart/templates/NOTES.txt | Adds install-time warnings for incompatible PSA mode / missing registry credentials. |
| deploy/Chart/templates/dynamic-rp/rbac.yaml | Grants dynamic-rp RBAC permissions for batch Jobs. |
| deploy/Chart/templates/dynamic-rp/deployment.yaml | Implements the terraform pre-mount fix and adds BuildKit containers/env/volumes. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11882 +/- ##
=======================================
Coverage 52.35% 52.36%
=======================================
Files 737 737
Lines 47240 47240
=======================================
+ Hits 24733 24736 +3
+ Misses 20141 20140 -1
+ Partials 2366 2364 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
willdavsmith
marked this pull request as draft
May 14, 2026 17:59
willdavsmith
force-pushed
the
feat/containerimages-buildkit-sidecar
branch
from
May 18, 2026 18:36
5df989d to
9d6ddfa
Compare
willdavsmith
changed the title
willdavsmith
force-pushed
the
feat/containerimages-buildkit-sidecar
branch
from
May 28, 2026 20:18
48b5a64 to
1f983a8
Compare
willdavsmith
added
the
pr:standard
willdavsmith
force-pushed
the
feat/containerimages-buildkit-sidecar
branch
from
May 29, 2026 19:47
e200448 to
5903352
Compare
willdavsmith
added a commit
that referenced
this pull request
Jun 2, 2026
Resolves several drift points the design had picked up over the review rounds on the chart and recipe PRs: Recipe / schema: - properties.tag is now optional for git sources too. The recipe hashes the resolved BuildKit URL (incl. ref + subdir) for git sources and the file tree for local sources; both feed into a content-addressable sha256 tag default. Drop the 'validate_git_tag' precondition and the 'tag required for git' language everywhere it appeared. - environment and application are marked required in the schema to match every other resource type in this PR's wave; update the properties table accordingly. - Add build.args to the schema properties table; the recipe validates keys (env-var-name shape) and values (no shell metacharacters) and feeds them into the tag hash. - The kubernetes_secret data source returns plain-text values (the provider auto-decodes), so drop all base64-decode references in the recipe sketch, the contract section, and the security section. - Refresh the recipe sketch to match what main.tf actually does (no base64decode, includes build_args and git-URL hashing). Chart: - buildctl is mounted at /usr/local/bin/buildctl via 'subPath: buildctl', landing it on the image's standard PATH without a PATH env-var override and without shadowing /usr/local/bin. Drop every reference to extending PATH; only BUILDKIT_HOST is set on dynamic-rp. - fsGroup: 65532 + supplementalGroups: [65532] are set at the pod level whenever the sidecar is enabled, not just under psaMode=restricted. The chown lets dynamic-rp read the shared emptyDir; supplementalGroups lets buildkitd (UID 1000) write to its chown'd state volume. - The NOTES.txt preflight, contributor doc, and sample recipe-pack Bicep are moved out of the initial-PR scope into the new Phasing section as Wave 2 follow-ups. None of them blocks Wave 1. Phasing: - Add a Phasing section that splits the design's scope across three waves: the initial chart + recipe (Wave 1), independent follow-ups (Wave 2: preflight, recipe-pack samples, contributor doc), and the coordinated local-context upload trio (Wave 3: rad CLI + dynamic-rp endpoint + recipe-side change). Architecture diagram: - Update the buildctl-init box to show the subPath mount at /usr/local/bin/buildctl instead of /opt/buildctl/bin. Signed-off-by: willdavsmith <willdavsmith@gmail.com>
willdavsmith
marked this pull request as ready for review
June 2, 2026 18:19
lakshmimsft
reviewed
Jun 2, 2026
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Cite moby/buildkit#4022 where the maintainer (AkihiroSuda, also the rootlesskit author) confirms 'the default apparmor profile prohibits mounting, so it still cannot be enabled' -- i.e. this is not a chart bug or a configuration-we-missed but a documented upstream constraint. Also reference moby/buildkit#3217 (GKE Autopilot same failure mode) and note the broader ecosystem state: Kaniko is archived as of June 2025, img is unmaintained since 2024, and Buildah hits the same seccomp constraint. There is no maintained tool today that builds OCI images inside a PSA-baseline Pod. Add concrete possible future paths (Localhost seccomp profile via DaemonSet, out-of-cluster builds, K8s 1.36 user namespace defaults, direct sandboxed builders like crane) so reviewers have something to react to rather than just an open-ended 'not supported'. Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Per design review: flip dynamicrp.buildkit.enabled default from true to false. Rationale: keeping the default install PSA-baseline compatible outweighs the discoverability cost. Users who want Radius.Compute/containerImages opt in explicitly via `--set dynamicrp.buildkit.enabled=true`. Behavior: - Default install: no BuildKit sidecar, no Radius.Compute/containerImages capability, PSA baseline compatible on radius-system. - Opt-in install: sidecar provisioned, requires PSA privileged on radius-system as before. Updates: - values.yaml: enabled: true -> false; comment recast as opt-in - PSA-NOTES.md: rewritten around the opt-in model; drops the regression framing and the alternative-builders ecosystem survey (no longer load-bearing for the operator decision) Signed-off-by: willdavsmith <willdavsmith@gmail.com>
The standalone PSA-baseline workflows in the demo repo are being deleted (no longer needed now that the chart sidecar is opt-in). Collapse the per-workflow bullets in the Validation section into a single paragraph that still cites the empirical EKS run, since the workflow run history persists on GitHub even after the workflow files are removed. Signed-off-by: willdavsmith <willdavsmith@gmail.com>
…elm tests for opt-in default - Add Akihiro, Suda, distros to .cspellignore for PSA-NOTES.md prose - Remove duplicate dynamicrp.resources block in values.yaml (leftover from earlier refactor; second block silently overwrote the first) - Gate securityContext.appArmorProfile on K8s >=1.30; emit the legacy container.apparmor.security.beta annotation on older clusters so the buildkitd sidecar gets Unconfined AppArmor on both - Invert two helm-unittest assertions to reflect the now-opt-in default for the BuildKit sidecar Signed-off-by: willdavsmith <willdavsmith@gmail.com>
willdavsmith
force-pushed
the
feat/containerimages-buildkit-sidecar
branch
from
June 15, 2026 16:13
58900e0 to
997c3b7
Compare
willdavsmith
marked this pull request as ready for review
June 15, 2026 17:35
PSA tradeoffs are now documented inline in values.yaml comments and the PR description, so the standalone PSA-NOTES.md doc is no longer needed. Removing it also drops three proper nouns (Akihiro, Suda, distros) from .cspellignore that were only used in that doc. Inline comments in deployment.yaml and values.yaml still explain the PSA implications at the point of use. Signed-off-by: willdavsmith <willdavsmith@gmail.com>
willtsai
pushed a commit
to willtsai/radius-project-resource-types-contrib
that referenced
this pull request
Jun 16, 2026
…form recipe (radius-project#151) ## Summary Adds a new `Radius.Compute/containerImages` resource type that builds a container image from source and pushes it to a remote registry, enabling source-to-running-app developer workflows in Radius without a host Docker socket, a privileged Pod, or per-node host preparation. The Kubernetes Terraform recipe drives a rootless BuildKit sidecar (provided by the dynamic-rp Helm chart in **radius-project/radius#11882**) via the `buildctl` CLI. ## What's in this PR - `Compute/containerImages/containerImages.yaml` — resource type schema (API version `2025-08-01-preview`) - `Compute/containerImages/recipes/kubernetes/terraform/{main,var}.tf` — Kubernetes Terraform recipe - `Compute/containerImages/README.md` — overview, prerequisites (Recipe Pack + `Radius.Security/secrets` example), recipe input/output properties ## Coordination Companion PR in `radius-project/radius` adds the BuildKit sidecar to the `dynamic-rp` chart. Both should land together; this one is reviewable independently. ## Notable details - **Shell-injection safety.** The recipe uses `local-exec` to invoke `buildctl`. Every interpolated value (registry, resource name, tag, dockerfile path, build context, platforms, build args) is gated by `precondition` blocks on a `validate_inputs` `terraform_data` resource that the build resource depends on. Inputs failing the regex are rejected at plan time. - **Content-addressable tags.** When `properties.tag` is unset, the recipe derives a `sha256-<hash>` tag from the build inputs (file tree for local sources, resolved URL for git sources, plus dockerfile path, platforms, and build args). For git sources, pin to a SHA or immutable tag — moving refs like `?ref=main` will produce a stable hash even as upstream advances. - **Multi-arch.** `build.platforms` is supported and uses BuildKit cross-compilation. The Dockerfile must use `FROM --platform=$BUILDPLATFORM` and `TARGETARCH`. There is intentionally no QEMU/binfmt fallback. - **Destroy semantics.** `terraform destroy` removes the resource from state but does **not** delete the pushed image from the registry. Image cleanup is an operator-side retention concern (registries differ widely on deletion semantics). ## Follow-ups (separate PRs) - CI harness enhancement: per-recipe `--parameters` injection and an in-cluster test registry, so this and similar recipes can be exercised in `test-recipe.sh` (radius-project#154). - Sample test fixture (`test/app.bicep`) once the CI harness can drive it end-to-end. - Local-path uploads: support `source: './frontend'` from a developer workstation. Requires the dynamic-rp tarball endpoint and rad CLI changes tracked in radius-project/radius#11882's Wave 3. --------- Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Contributor
|
/LGTM once the issue flagged above on append-root-ca is addressed. |
The append-root-ca initContainer was nested inside the 'if global.terraform.enabled' block, but the ssl-certs emptyDir volume and the dynamic-rp container's mount at /etc/ssl/certs are conditioned only on global.appendRootCA.cert. Operators who set appendRootCA.cert with terraform disabled (or with only BuildKit enabled) ended up with an empty emptyDir mounted over /etc/ssl/certs, breaking the runtime trust store. Hoist the append-root-ca block out of the terraform gate and expand the outer 'initContainers:' OR condition to include appendRootCA.cert so the initContainers list is emitted whenever any of terraform, buildkit, or appendRootCA needs it. Existing ordering (append-root-ca first, then terraform-init, then BuildKit init containers) is preserved when multiple paths are enabled. Add two helm-unittest cases pinning the previously-broken combinations: terraform off + appendRootCA on (only append-root-ca emitted), and terraform off + buildkit on + appendRootCA on (append-root-ca runs before the buildkit init containers). Addresses Copilot review feedback on PR #11882. Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Radius functional test overviewClick here to see the test run details
Test Status⌛ Building Radius and pushing container images for functional tests... |
lakshmimsft
approved these changes
Jun 17, 2026
shivg7795
pushed a commit
to shivg7795/resource-types-contrib-shivg
that referenced
this pull request
Jul 9, 2026
…form recipe (radius-project#151) ## Summary Adds a new `Radius.Compute/containerImages` resource type that builds a container image from source and pushes it to a remote registry, enabling source-to-running-app developer workflows in Radius without a host Docker socket, a privileged Pod, or per-node host preparation. The Kubernetes Terraform recipe drives a rootless BuildKit sidecar (provided by the dynamic-rp Helm chart in **radius-project/radius#11882**) via the `buildctl` CLI. ## What's in this PR - `Compute/containerImages/containerImages.yaml` — resource type schema (API version `2025-08-01-preview`) - `Compute/containerImages/recipes/kubernetes/terraform/{main,var}.tf` — Kubernetes Terraform recipe - `Compute/containerImages/README.md` — overview, prerequisites (Recipe Pack + `Radius.Security/secrets` example), recipe input/output properties ## Coordination Companion PR in `radius-project/radius` adds the BuildKit sidecar to the `dynamic-rp` chart. Both should land together; this one is reviewable independently. ## Notable details - **Shell-injection safety.** The recipe uses `local-exec` to invoke `buildctl`. Every interpolated value (registry, resource name, tag, dockerfile path, build context, platforms, build args) is gated by `precondition` blocks on a `validate_inputs` `terraform_data` resource that the build resource depends on. Inputs failing the regex are rejected at plan time. - **Content-addressable tags.** When `properties.tag` is unset, the recipe derives a `sha256-<hash>` tag from the build inputs (file tree for local sources, resolved URL for git sources, plus dockerfile path, platforms, and build args). For git sources, pin to a SHA or immutable tag — moving refs like `?ref=main` will produce a stable hash even as upstream advances. - **Multi-arch.** `build.platforms` is supported and uses BuildKit cross-compilation. The Dockerfile must use `FROM --platform=$BUILDPLATFORM` and `TARGETARCH`. There is intentionally no QEMU/binfmt fallback. - **Destroy semantics.** `terraform destroy` removes the resource from state but does **not** delete the pushed image from the registry. Image cleanup is an operator-side retention concern (registries differ widely on deletion semantics). ## Follow-ups (separate PRs) - CI harness enhancement: per-recipe `--parameters` injection and an in-cluster test registry, so this and similar recipes can be exercised in `test-recipe.sh` (radius-project#154). - Sample test fixture (`test/app.bicep`) once the CI harness can drive it end-to-end. - Local-path uploads: support `source: './frontend'` from a developer workstation. Requires the dynamic-rp tarball endpoint and rad CLI changes tracked in radius-project/radius#11882's Wave 3. --------- Signed-off-by: willdavsmith <willdavsmith@gmail.com> Signed-off-by: shivg7795 <shivg@microsoft.com>
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an opt-in rootless BuildKit sidecar to the
dynamic-rpPod. When enabled, this lets in-cluster Terraform recipes build and push OCI images without a host Docker socket, a privileged Pod, per-node host preparation, or a separate build cluster.The motivating consumer is the new
Radius.Compute/containerImagesresource type — see companion PR radius-project/resource-types-contrib#151.Why opt-in (default
enabled: false)Building OCI images inside a Kubernetes Pod fundamentally requires Linux capabilities (mount, unshare, pivot_root, /proc manipulation) that PSA
baselinedisallows. After extensive validation we concluded:baselineforbids.rad install kubernetesshould remain PSA-baseline-compatible. Therefore the sidecar is off by default, and operators who wantRadius.Compute/containerImagesopt in:When the sidecar is enabled,
radius-systemmust be PSA-labeledprivileged(or unlabeled, which defaults to privileged on AKS, EKS, GKE, kind, k3d).What's in this PR
Files changed (4, +334/-11):
deploy/Chart/templates/dynamic-rp/deployment.yaml— Adds (whendynamicrp.buildkit.enabled: true) abuildkitdsidecar (rootless, listens on Pod loopback TCP127.0.0.1:1234), abuildctl-initinitContainer that stages thebuildctlCLI on a shared volume mounted into dynamic-rp'sPATH, and abuildkit-volumes-initinitContainer that scopes ownership of the BuildKit emptyDirs without applying a Pod-levelfsGroup(avoids leaking the encryption-key Secret to the sidecar).securityContext.appArmorProfileis gated on K8s ≥ 1.30 with the legacycontainer.apparmor.security.betaannotation as fallback for older clusters.deploy/Chart/values.yaml— Addsdynamicrp.buildkit.{enabled, image, resources}.enableddefaults tofalse. Removes a duplicatedynamicrp.resourcesblock that was a leftover from an earlier refactor.deploy/Chart/tests/helpers_test.yaml—helm-unittestcases covering the opt-in default (sidecar absent), the opt-in path (sidecar + buildctl-init present), and interaction with the existing terraform-init container.No Go code changes. No RBAC changes. No public-API changes. No host-level changes.
Default install behavior (no flags)
buildkitdsidecarbuildctl-initinitContainerbuildkit-volumes-initinitContainerdynamicrp.buildkit.*values surfaceThe default
helm templateoutput is byte-for-byte equivalent to today's chart for the dynamic-rp Pod when the rest ofdynamicrp.*is unchanged. Verified by helm template diff.Opt-in install behavior (
--set dynamicrp.buildkit.enabled=true)radius-systemnamespace must be PSA-labeledprivileged(or unlabeled, which defaults to privileged on AKS, EKS, GKE, kind, k3d).hostNetwork/hostPID. Sidecar communicates with the dynamic-rp container over Pod-localhost TCP only.Testing
helm unittest .— 82/82 tests pass (includes the new BuildKit cases).helm lint .— 0 failures.helm template— verified to render correctly with--kube-version 1.29.0(annotation fallback),--kube-version 1.30.0(modern field), and with buildkit on / off.Coordination
Radius.Compute/containerImagesdesign #11734.Wave structure
This PR is Wave 1: the chart runtime piece the recipe needs to function end-to-end with git-context builds. Independently reviewable and mergeable.
Wave 2 (follow-up PRs in this repo, no dependency on Wave 1):
local-exec-via-buildctlrecipe pattern.Wave 3 (local-context upload, depends on Wave 1):
emptyDirfor the recipe to consume.radCLI: whenbuild.sourceis a local path, tar with.dockerignorehonored and POST to dynamic-rp before recipe execution.radius-project/resource-types-contribto accept the staged context path.Until Wave 3 lands,
build.sourceis restricted togit::https://...URLs and absolute filesystem paths already available to the recipe runtime.