Skip to content

Radius.Compute/containerImages design - #11734

Merged
willdavsmith merged 16 commits into
mainfrom
willdavsmith/dynamic-rp-docker-support
Jun 15, 2026
Merged

Radius.Compute/containerImages design#11734
willdavsmith merged 16 commits into
mainfrom
willdavsmith/dynamic-rp-docker-support

Conversation

@willdavsmith

@willdavsmith willdavsmith commented Apr 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

  • Add design note for Radius.Compute/containerImages

Type of change

  • This pull request is a design document.

Contributor checklist

Please verify that the PR meets the following requirements, where applicable:

  • An overview of proposed schema changes is included in a linked GitHub issue.
    • Yes
    • Not applicable
  • A design document PR is created in the design-notes repository, if new APIs are being introduced.
    • Yes
    • Not applicable
  • The design document has been reviewed and approved by Radius maintainers/approvers.
    • Yes
    • Not applicable
  • A PR for the samples repository is created, if existing samples are affected by the changes in this PR.
    • Yes
    • Not applicable
  • A PR for the documentation repository is created, if the changes in this PR affect the documentation or any user facing updates are made.
    • Yes
    • Not applicable
  • A PR for the recipes repository is created, if existing recipes are affected by the changes in this PR.
    • Yes
    • Not applicable

@willdavsmith willdavsmith added the pr:standard Ongoing maintenance, minor improvements, documentation updates, and routine development work label Apr 22, 2026
@github-actions

github-actions Bot commented Apr 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Unit Tests

    2 files  ±0    439 suites  ±0   7m 33s ⏱️ +6s
5 366 tests ±0  5 364 ✅ ±0  2 💤 ±0  0 ❌ ±0 
6 546 runs  ±0  6 544 ✅ ±0  2 💤 ±0  0 ❌ ±0 

Results for commit 15f29e0. ± Comparison against base commit 7b81a1c.

♻️ This comment has been updated with latest results.

@codecov

codecov Bot commented Apr 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.26%. Comparing base (7b81a1c) to head (15f29e0).

Additional details and impacted files 
@@           Coverage Diff           @@
##             main   #11734   +/-   ##
=======================================
  Coverage   52.26%   52.26%           
=======================================
  Files         736      736           
  Lines       47043    47043           
=======================================
+ Hits        24587    24589    +2     
+ Misses      20102    20101    -1     
+ Partials     2354     2353    -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

zachcasper
zachcasper suggested changes May 13, 2026
View reviewed changes
Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
@zachcasper zachcasper mentioned this pull request May 13, 2026
Closed
12 tasks
kachawla
kachawla reviewed May 13, 2026
View reviewed changes

@kachawla kachawla left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed that containerImage manifest was reverted from resource-types-contrib, can you please revert it from the Radius repo as well? https://github.com/radius-project/radius/blob/main/deploy/manifest/built-in-providers/dev/radius_compute.yaml#L730C3-L730C15

brooke-hamilton
brooke-hamilton reviewed May 13, 2026
View reviewed changes

@brooke-hamilton brooke-hamilton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
Comment thread eng/design-notes/recipes/2026-04-container-images-resource-type.md Resolved
@willdavsmith

willdavsmith commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

I just noticed that containerImage manifest was reverted from resource-types-contrib, can you please revert it from the Radius repo as well? https://github.com/radius-project/radius/blob/main/deploy/manifest/built-in-providers/dev/radius_compute.yaml#L730C3-L730C15

I will do this

@github-actions

github-actions Bot commented May 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@willdavsmith
willdavsmith marked this pull request as ready for review May 26, 2026 21:10
Copilot AI review requested due to automatic review settings May 26, 2026 21:10
@willdavsmith
willdavsmith requested review from a team as code owners May 26, 2026 21:10
Copilot AI reviewed May 26, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a single design note proposing a new Radius.Compute/containerImages resource type, which uses BuildKit (running as a rootless sidecar in dynamic-rp) and a Terraform recipe to build and push container images as part of a Radius application deployment.

Changes:

  • New design note describing schema, recipe contract, and Helm chart changes
  • Discusses PSA baseline/restricted modes, multi-arch via cross-compile, and credential wiring through Radius.Security/secrets
  • Outlines local-context upload workflow via a new dynamic-rp endpoint

willdavsmith added 12 commits May 29, 2026 10:38
@willdavsmith
Add design note for containerImages
ffa3edd 
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
v1 add docker to dynamic-rp
d3044d1 
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
PR review round
b0834b6 
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
Remove dynamic-rp Helm chart changes from PR
4e8408b 
The chart changes (buildkitd sidecar, NOTES.txt preflight, values)
will be split into a separate PR. This PR is now design-doc-only.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
Address PR review comments in design doc
0f97a0a 
- Make environment and application optional in schema
- Recipe registration via Radius.Core/recipePacks (recipe-pack flow)
- Move registry credentials into Radius.Core/secrets +
  Radius.Core/terraformConfigurations; drop credential recipe params
- Drop per-resource registry override from schema and recipe sketch
- Reframe 'why this type is different' around the no-special-casing
  design principle
- Add 'Implementation choice: recipe vs. built-in provider' section
  and corresponding open question
- Add 'Bring-your-own published image' escape hatch
- Sidecar default-on with dynamicrp.buildkit.enabled opt-out;
  resolve idle-cost open question
- Update development plan, advantages/disadvantages, security, and
  implementation details tables to match

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
Use registrySecretName parameter for registry auth
52f58d3 
The previous draft referenced a Radius.Core/terraformConfigurations
resource that does not exist. The closest in-flight type
(Radius.Core/terraformSettings) only models terraformrc /
backend / env / logging and does not provide a place for
per-provider configuration like the Docker provider's
registry_auth block.

Switch to the secretName pattern already used by
Radius.Data/mySqlDatabases and other authenticated resource
types: the recipe takes a registrySecretName parameter naming a
Radius.Security/secrets resource and resolves it at execution
time to populate the Docker provider's registry_auth block.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
manual review round
6fbba26 
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
docs: align containerImages design doc with demo
82e24c8 
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
docs: model registry credentials as a Radius.Security/secrets resourc…
38a681e 
…e reference

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
Update containerImages design doc to recipePack-parameter UX
0a6086c 
Replaces the registryCredentials resource-ID UX (which would have
required engine-side dereferencing of Radius.Security/secrets in
recipe parameters) with the registrySecretName string-parameter
pattern matching Radius.Data/postgreSqlDatabases.secretName.

The PE passes the secret's name as a literal string through the
recipePack; the Radius.Security/secrets recipe materializes a
same-named Kubernetes Secret in the env namespace; the
containerImages recipe reads it via 'data "kubernetes_secret"'.
No engine work required.

Sections updated: sample input, recipe registration (with explicit
BCP080-cycle workaround via local var), schema table, tag strategy,
terraform recipe bullets, HCL sketch, helm chart endpoint wiring,
implementation-choice + implementation-details + development-plan
tables (dropping the engine 'recipe-parameter secret resolution'
row), security, alternatives K.

Also drops per-resource pull-Secret materialization and default-SA
patching from the recipe; cluster image pull is now described as
a platform-engineer concern handled out-of-band.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
containerImages design doc: replace var-workaround with two-file split
3595de8 
Splits the PE Bicep into platform.bicep (env + recipePack) and
secrets.bicep (Radius.Security/secrets; env via 'existing'),
removing the BCP080 cycle without the local-var workaround that
pretended a symbolic dep wasn't there.

Updates Recipe registration section (full rewrite with both
files + explanation), Disadvantages -> Credential bootstrap,
Implementation details / Development plan tables.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
schema rename per reviewer feedback
2ba07b4 
- imageTag -> tag
- imageName dropped (use resource name)
- build.context -> build.source
- image -> imageReference
- buildContext (Bicep param) -> buildSource

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
willdavsmith force-pushed the willdavsmith/dynamic-rp-docker-support branch from 7debdf9 to 4b4b57c Compare May 29, 2026 17:39
@willdavsmith
willdavsmith requested a review from zachcasper May 29, 2026 17:43
willdavsmith added a commit to radius-project/resource-types-contrib that referenced this pull request May 29, 2026
@willdavsmith
test: remove app.bicep until CI cluster has buildkit sidecar + registry
f44ce45 
The Kubernetes Terraform recipe requires the in-Pod BuildKit sidecar
provisioned by radius-project/radius#11734 (dynamic-rp chart) and a
reachable container registry. Neither is available in the
validate-resource-types CI cluster yet, so end-to-end testing is
deferred. Same pattern as Compute/persistentVolumes and Compute/routes.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
cspell: add buildkitd, buildctl, binfmt, kubelet, Buildah, Kaniko
9f13093 
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
willdavsmith force-pushed the willdavsmith/dynamic-rp-docker-support branch from 4b4b57c to 9f13093 Compare May 29, 2026 22:10
@willdavsmith willdavsmith mentioned this pull request May 29, 2026
Merged
@willdavsmith
Drop 'kind: generic' from secrets.bicep example
53536de 
'kind' is optional on Radius.Security/secrets and defaults to
'generic'. Per reviewer feedback, drop it from the example to
match how PEs actually write these in practice.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
Reconcile design with implementation in #11882 and #151
f5ed3e7 
Resolves several drift points the design had picked up over the
review rounds on the chart and recipe PRs:

Recipe / schema:
- properties.tag is now optional for git sources too. The recipe
  hashes the resolved BuildKit URL (incl. ref + subdir) for git
  sources and the file tree for local sources; both feed into a
  content-addressable sha256 tag default. Drop the
  'validate_git_tag' precondition and the 'tag required for git'
  language everywhere it appeared.
- environment and application are marked required in the schema
  to match every other resource type in this PR's wave; update
  the properties table accordingly.
- Add build.args to the schema properties table; the recipe
  validates keys (env-var-name shape) and values (no shell
  metacharacters) and feeds them into the tag hash.
- The kubernetes_secret data source returns plain-text values
  (the provider auto-decodes), so drop all base64-decode
  references in the recipe sketch, the contract section, and
  the security section.
- Refresh the recipe sketch to match what main.tf actually does
  (no base64decode, includes build_args and git-URL hashing).

Chart:
- buildctl is mounted at /usr/local/bin/buildctl via 'subPath:
  buildctl', landing it on the image's standard PATH without a
  PATH env-var override and without shadowing /usr/local/bin.
  Drop every reference to extending PATH; only BUILDKIT_HOST is
  set on dynamic-rp.
- fsGroup: 65532 + supplementalGroups: [65532] are set at the
  pod level whenever the sidecar is enabled, not just under
  psaMode=restricted. The chown lets dynamic-rp read the
  shared emptyDir; supplementalGroups lets buildkitd (UID 1000)
  write to its chown'd state volume.
- The NOTES.txt preflight, contributor doc, and sample
  recipe-pack Bicep are moved out of the initial-PR scope into
  the new Phasing section as Wave 2 follow-ups. None of them
  blocks Wave 1.

Phasing:
- Add a Phasing section that splits the design's scope across
  three waves: the initial chart + recipe (Wave 1), independent
  follow-ups (Wave 2: preflight, recipe-pack samples, contributor
  doc), and the coordinated local-context upload trio (Wave 3:
  rad CLI + dynamic-rp endpoint + recipe-side change).

Architecture diagram:
- Update the buildctl-init box to show the subPath mount at
  /usr/local/bin/buildctl instead of /opt/buildctl/bin.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith
willdavsmith enabled auto-merge (squash) June 15, 2026 15:41
@radius-functional-tests

radius-functional-tests Bot commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Radius functional test overview

🔍 Go to test action run

Click here to see the test run details
Name Value
Repository radius-project/radius
Commit ref 15f29e0
Unique ID func54d3783960
Image tag pr-func54d3783960
  • gotestsum 1.13.0
  • KinD: v0.29.0
  • Dapr: 1.14.4
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func54d3783960
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func54d3783960
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func54d3783960
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func54d3783960
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func54d3783960
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
❌ corerp-cloud functional test failed. Please check the logs for more details
⌛ Starting corerp-cloud functional tests...
✅ corerp-cloud functional tests succeeded

@willdavsmith
willdavsmith merged commit ef934de into main Jun 15, 2026
66 of 68 checks passed
@willdavsmith
willdavsmith deleted the willdavsmith/dynamic-rp-docker-support branch June 15, 2026 18:20
willdavsmith added a commit that referenced this pull request Jun 17, 2026
@willdavsmith
Radius.Compute/containerImages type: add rootless BuildKit sidecar to…
7ff1efa 
… dynamic-rp chart (#11882)

# Description

Adds an **opt-in** rootless BuildKit sidecar to the `dynamic-rp` Pod.
When enabled, this lets in-cluster Terraform recipes build and push OCI
images without a host Docker socket, a privileged Pod, per-node host
preparation, or a separate build cluster.

The motivating consumer is the new `Radius.Compute/containerImages`
resource type — see companion PR
**radius-project/resource-types-contrib#151**.

## Why opt-in (default `enabled: false`)

Building OCI images inside a Kubernetes Pod fundamentally requires Linux
capabilities (mount, unshare, pivot_root, /proc manipulation) that PSA
`baseline` disallows. After extensive validation we concluded:

- A baseline-compatible BuildKit sidecar is not achievable today;
rootlesskit needs Unconfined seccomp + AppArmor profiles that PSA
`baseline` forbids.
- Default `rad install kubernetes` should remain
PSA-baseline-compatible. Therefore the sidecar is off by default, and
operators who want `Radius.Compute/containerImages` opt in:
  ```
  rad install kubernetes --set dynamicrp.buildkit.enabled=true
  ```
  Or via Helm:
  ```
helm install radius radius/radius --set dynamicrp.buildkit.enabled=true
  ```

When the sidecar is enabled, `radius-system` must be PSA-labeled
`privileged` (or unlabeled, which defaults to privileged on AKS, EKS,
GKE, kind, k3d).

## What's in this PR

Files changed (4, +334/-11):

- **`deploy/Chart/templates/dynamic-rp/deployment.yaml`** — Adds (when
`dynamicrp.buildkit.enabled: true`) a `buildkitd` sidecar (rootless,
listens on Pod loopback TCP `127.0.0.1:1234`), a `buildctl-init`
initContainer that stages the `buildctl` CLI on a shared volume mounted
into dynamic-rp's `PATH`, and a `buildkit-volumes-init` initContainer
that scopes ownership of the BuildKit emptyDirs without applying a
Pod-level `fsGroup` (avoids leaking the encryption-key Secret to the
sidecar). `securityContext.appArmorProfile` is gated on K8s ≥ 1.30 with
the legacy `container.apparmor.security.beta` annotation as fallback for
older clusters.
- **`deploy/Chart/values.yaml`** — Adds `dynamicrp.buildkit.{enabled,
image, resources}`. `enabled` defaults to `false`. Removes a duplicate
`dynamicrp.resources` block that was a leftover from an earlier
refactor.
- **`deploy/Chart/tests/helpers_test.yaml`** — `helm-unittest` cases
covering the opt-in default (sidecar absent), the opt-in path (sidecar +
buildctl-init present), and interaction with the existing terraform-init
container.

No Go code changes. No RBAC changes. No public-API changes. No
host-level changes.

## Default install behavior (no flags)

| Component | Present in default install? |
|---|---|
| `buildkitd` sidecar | No |
| `buildctl-init` initContainer | No |
| `buildkit-volumes-init` initContainer | No |
| AppArmor annotation/field | No |
| New `dynamicrp.buildkit.*` values surface | Yes (chart values only —
no runtime impact) |

The default `helm template` output is byte-for-byte equivalent to
today's chart for the dynamic-rp Pod when the rest of `dynamicrp.*` is
unchanged. Verified by helm template diff.

## Opt-in install behavior (`--set dynamicrp.buildkit.enabled=true`)

- BuildKit sidecar runs rootless under uid/gid 1000, in its own user
namespace.
- `radius-system` namespace must be PSA-labeled `privileged` (or
unlabeled, which defaults to privileged on AKS, EKS, GKE, kind, k3d).
- No host Docker socket, no privileged container, no
`hostNetwork`/`hostPID`. Sidecar communicates with the dynamic-rp
container over Pod-localhost TCP only.

## Testing

- `helm unittest .` — 82/82 tests pass (includes the new BuildKit
cases).
- `helm lint .` — 0 failures.
- `helm template` — verified to render correctly with `--kube-version
1.29.0` (annotation fallback), `--kube-version 1.30.0` (modern field),
and with buildkit on / off.
- End-to-end multi-arch build + push validated against kind, k3d, AKS,
and EKS in
[willdavsmith/radius-containerimagetype-demo](https://github.com/willdavsmith/radius-containerimagetype-demo)
— see the e2e workflows there for the full matrix.

## Coordination

- **Companion PR**:
[radius-project/resource-types-contrib#151](radius-project/resource-types-contrib#151)
(resource type + recipe). They should land together; this one is
reviewable independently.
- **Related fix**:
[radius-project/resource-types-contrib#178](radius-project/resource-types-contrib#178)
addresses the secrets recipe double-base64-encoding bug surfaced during
validation.
- **Design**: #11734.

## Wave structure

This PR is **Wave 1**: the chart runtime piece the recipe needs to
function end-to-end with git-context builds. Independently reviewable
and mergeable.

**Wave 2 (follow-up PRs in this repo, no dependency on Wave 1):**
- Contributor doc explaining the sidecar + `local-exec`-via-`buildctl`
recipe pattern.

**Wave 3 (local-context upload, depends on Wave 1):**
- New dynamic-rp endpoint accepting tarball uploads, staging the build
context in an `emptyDir` for the recipe to consume.
- `rad` CLI: when `build.source` is a local path, tar with
`.dockerignore` honored and POST to dynamic-rp before recipe execution.
- Recipe-side change in `radius-project/resource-types-contrib` to
accept the staged context path.

Until Wave 3 lands, `build.source` is restricted to `git::https://...`
URLs and absolute filesystem paths already available to the recipe
runtime.

---------

Signed-off-by: willdavsmith <willdavsmith@gmail.com>
@willdavsmith willdavsmith mentioned this pull request Jul 13, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nellshamrell nellshamrell nellshamrell left review comments
@DariuszPorowski DariuszPorowski DariuszPorowski left review comments
@willtsai willtsai willtsai left review comments
@brooke-hamilton brooke-hamilton brooke-hamilton left review comments
@Reshrahim Reshrahim Reshrahim left review comments
@kachawla kachawla kachawla left review comments
Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@nithyatsu nithyatsu nithyatsu approved these changes
+2 more reviewers
@sylvainsf sylvainsf sylvainsf left review comments
@zachcasper zachcasper zachcasper approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

pr:standard Ongoing maintenance, minor improvements, documentation updates, and routine development work

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.