diff --git a/eng/design-notes/recipes/2026-05-direct-recipe-modules.md b/eng/design-notes/recipes/2026-05-direct-recipe-modules.md new file mode 100644 index 00000000000..41aa47a3643 --- /dev/null +++ b/eng/design-notes/recipes/2026-05-direct-recipe-modules.md @@ -0,0 +1,371 @@ +# Topic: Direct Recipe Modules + +* **Author**: Reshma Abdul Rahim (@Reshrahim) + +## Topic Summary + +Radius Recipes enable platform engineers to define reusable infrastructure templates that application developers can consume without understanding the underlying cloud resources. Today, to use an existing community module — e.g., an [AWS RDS Terraform module](https://registry.terraform.io/modules/terraform-aws-modules/rds/aws/latest) from the Terraform Registry, or an [Azure Verified Bicep Module](https://azure.github.io/Azure-Verified-Modules/) from the Microsoft Container Registry — as a Recipe, a platform engineer must write a thin recipe wrapper that adds Radius-specific conventions — namely a `context` input variable and a structured `result` output, publish it to a distribution source like a container registry or Git, and then it's ready for use. This extra layer adds maintenance burden, version drift risk, and a barrier to adoption. + +**Direct Recipe Module Support removes this requirement.** Platform engineers can now point a Recipe's `source` directly at any standard Bicep or Terraform module — no wrapper needed. Radius automatically resolves input values into the module's parameters at deploy time and maps the module's outputs back to the Resource Type's properties. The result is a simpler workflow: find a module, reference it, configure the parameter and output mappings, and deploy. + +This also enables us to scale the `resource-types-contrib` repository by leveraging existing, well-maintained community modules without having to write everything from scratch. Instead of authoring and maintaining custom wrapper Recipes, we maintain Recipe Packs with tested modules per platform validated against Radius Resource Types. + +### Top level goals + +- Enable platform engineers to leverage community maintained Terraform registry modules and Azure Verified Modules directly as Radius Recipes +- Eliminate the need to write a Recipe wrapper, publish, and maintain these Recipes separately from the underlying module +- Scale the `resource-types-contrib` repository with tested Recipes by referencing community modules directly, eliminating wrapper maintenance overhead + +### Non-goals (out of scope) + +1. Local filesystem paths as `source` values. Modules must be fetched from remote registries or Git; local paths are not supported. +2. Automatic module version bumping. Platform engineers explicitly pin module versions in their Recipe Packs. Auto-upgrading could introduce breaking changes silently. +3. Custom retry logic for module fetch failures. Fetch retries are delegated to the underlying IaC engine (Terraform CLI / Bicep). Radius does not add its own retry layer. +4. New observability infrastructure. This feature uses existing Radius logging, tracing, and metrics. No new telemetry systems are introduced. + +## User profile and challenges + +### User persona(s) + +**Platform Engineer**: Responsible for defining infrastructure Recipes that application developers consume. Manages Recipe Packs, environments, and provider configurations. Typically works across multiple teams and maintains a catalog of infrastructure patterns. + +**Application Developer**: Consumes Recipes by declaring resources with properties (e.g., `database: 'mydb'`, `size: 's'`). Does not need to understand the underlying IaC module or infrastructure details. + +### Challenge(s) faced by the user + +1. **Adoption friction** — Every module requires a custom Recipe wrapper that must be published and versioned separately before it can be used. Using a community Terraform module requires writing a Recipe wrapper, publishing the wrapper to a distribution source and then adding it to Recipe Packs for consumption. + +2. **Maintenance burden** — Upstream module updates require wrapper changes, validation, and republishing, creating version drift over time. Platform engineers must track upstream releases and update wrappers accordingly. + +3. **Ecosystem lockout** — Thousands of production-ready community modules (Terraform registry, Azure Verified Modules) cannot be used directly, limiting the value proposition of Radius Recipes. + +### Positive user outcome + +Platform engineers can easily configure a Recipe using any existing module setting `source` directly — zero wrapping, zero republishing. The entire Terraform and Bicep module ecosystem becomes immediately usable as Radius Recipes. + +## Key scenarios + +### Scenario 1: Terraform Registry Module + +A platform engineer sets `source` to a Terraform registry module source (e.g., `terraform-aws-modules/rds/aws`) with a `version` field (e.g., `5.9.0`) and the system deploys it by automatically resolving developer set properties via `context` as Terraform input variables, and mapping module outputs to Resource Type properties via the `outputs` field. + +### Scenario 2: Azure Verified Modules (OCI) + +A platform engineer sets `source` to an Azure Verified Module OCI reference (e.g., `br:mcr.microsoft.com/bicep/avm/res/storage/storage-account:0.14.3`) and the system deploys it via ARM, passing resolved parameters and mapping outputs to Resource Type properties via the `outputs` field. + +### Scenario 3: Private hosted Module + +A platform engineer references an internal module hosted in a private OCI registry (e.g., `oci://my-registry.com/my-namespace/my-module:1.0.0`). This supports organizations that maintain proprietary modules outside the public Terraform Registry. The system clones the module at the specified ref and executes it using the same parameter resolution and output mapping as registry modules. + +## Key dependencies and risks + +- **Community module coverage** — Not every Radius Resource Type has a suitable community module. Where gaps exist, platform engineers will still author custom modules. For the `resource-types-contrib` repository, we will continue to write and maintain Recipes for Resource Types that lack community module coverage. +- **Terraform version compatibility** — Direct modules may use features from newer Terraform versions. Mitigation: Terraform settings feature enables specifying the Terraform version. +- **Module API stability** — Direct modules expose platform engineers to upstream breaking changes. Mitigation: version pinning via `?ref=` for Git or registry version constraints. +- **Expression resolution correctness** — Unresolved `{{context.*}}` expressions are left as literal strings, which may cause cryptic downstream errors from the IaC engine. Mitigation: clear documentation of available expression paths. + +## Current state + +Today, Radius only supports Recipes that are purpose-built for Radius. Both Terraform and Bicep modules must include a `context` input variable (carrying Radius runtime metadata) and a structured `result` output (returning resource values back to Radius). Any community or third-party module that lacks these conventions cannot be used as a Recipe without first wrapping it. + +Recipe Packs (`Radius.Core/recipePacks`) already enable platform engineers to group Recipe definitions by Resource Type. This direct module support extends that foundation by adding an `outputs` mapping to `RecipeDefinition` and broadening `source` to accept standard module sources that do not follow Radius wrapper conventions. + +## Details of user problem + +When I want to use a community Terraform module (like `terraform-aws-modules/rds/aws`) as a Radius Recipe, I have to write a wrapper module that adds the `context` variable and `result` output, publish my wrapper to a Git repository or OCI registry based on the IaC engine, and then reference my wrapper in the Recipe definition. If the upstream module releases a new version, I have to update my wrapper to pin the new version, validate that the wrapper still passes through the correct inputs/outputs, and republish it — even if the module's interface hasn't changed. This creates a maintenance burden that scales with the number of modules in my catalog. + +## Desired user experience outcome + +After this feature, I can set `source` directly to `terraform-aws-modules/rds/aws` and `version` to `5.9.0` on my Recipe definition, configure `parameters` with `{{context.*}}` expressions for dynamic values, and set `outputs` to map the module's outputs to my Resource Type's properties. The module doesn't need to know about Radius. When my application developer deploys a `Radius.Data/mySqlDatabases` resource, the system resolves expressions, passes parameters to the module, executes it, and maps outputs — all without any wrapper. + +### Detailed user experience + +>**Note**: This user experience incorporates the [Rename recipeKind and recipeLocation to kind and source in Recipe Packs](https://github.com/radius-project/radius/issues/11879) changes. + + +1. Platform engineer creates a Recipe Pack pointing directly at an upstream module: + + ```bicep + resource recipepack 'Radius.Core/recipePacks@2025-08-01-preview' = { + name: 'data-pack' + properties: { + recipes: { + 'Radius.Data/mySqlDatabases': { + kind: 'terraform' + source: 'terraform-aws-modules/rds/aws' + version: '5.9.0' + // Specify the module's input parameters + parameters: { + identifier: '{{context.resource.name}}' + db_name: '{{context.resource.properties.database}}' + engine: 'mysql' + instance_class: 'db.t3.micro' + } + // Map the module's outputs to the resource's properties + outputs: { + host: 'db_instance_address' + port: 'db_instance_port' + database: 'db_instance_name' + secretName: 'db_instance_master_user_secret_arn' + } + } + } + } + } + + resource env 'Radius.Core/environments@2025-08-01-preview' = { + name: 'aws-dev' + properties: { + recipePacks: [recipepack.id] + recipeParameters: { + 'Radius.Data/mySqlDatabases': { + backup_retention_days: 2 + geo_redundant_backup_enabled: false + } + } + } + } + ``` + +2. Application developer deploys a resource with properties: + + ```bicep + resource mysql 'Radius.Data/mySqlDatabases@2025-08-01-preview' = { + name: 'orders-db' + properties: { + environment: environment + application: app.id + database: 'ordersdb' + } + } + ``` + +3. System resolves expressions (`{{context.resource.name}}` → `'orders-db'`, `{{context.resource.properties.database}}` → `'ordersdb'`), merges parameters, and executes Terraform against the upstream module +4. Module outputs are mapped to resource properties via the `outputs` definition +5. Application reads `resource.properties.host`, `resource.properties.port` etc as the resolved values from the module outputs. + +6. (Optional) For private modules, the platform engineer configures registry or Git authentication using the settings resources (`Radius.Core/terraformConfigs` or `Radius.Core/bicepConfigs`) referenced from the environment. The Radius driver resolves credentials by parsing the hostname from the module `source` and looking up the matching key in the credentials map. + + **Private Terraform registry (e.g., Terraform Cloud):** + + ```bicep + resource terraformConfig 'Radius.Core/terraformConfigs@2025-08-01-preview' = { + name: 'my-terraform-config' + properties: { + terraformrc: { + credentials: { + 'app.terraform.io': { + secret: tfcSecrets.id + } + } + } + } + } + + resource tfcSecrets 'Radius.Core/secrets@2025-08-01-preview' = { + name: 'tfc-credentials' + properties: { + type: 'generic' + data: { + token: { value: tfcToken } + } + } + } + + resource env 'Radius.Core/environments@2025-08-01-preview' = { + name: 'aws-dev' + properties: { + recipePacks: [recipepack.id] + terraformConfig: terraformConfig.id + } + } + ``` + + **Private Bicep/OCI registry:** + + ```bicep + resource bicepConfig 'Radius.Core/bicepConfigs@2025-08-01-preview' = { + name: 'my-bicep-config' + properties: { + registryAuthentications: { + 'corp.acr.io': { + authenticationMethod: 'BasicAuth' + basicAuthSecretId: acrSecrets.id + } + } + } + } + + resource acrSecrets 'Radius.Core/secrets@2025-08-01-preview' = { + name: 'acr-credentials' + properties: { + type: 'generic' + data: { + username: { value: acrUsername } + password: { value: acrPassword } + } + } + } + + resource env 'Radius.Core/environments@2025-08-01-preview' = { + name: 'aws-dev' + properties: { + recipePacks: [recipepack.id] + bicepConfig: bicepConfig.id + } + } + ``` + +## Key investments + +### Feature 1: Direct Module Execution + +Use any standard Bicep or Terraform module directly as a Recipe by pointing `source` at the module source. The system downloads the module, and executes it through the existing driver. + +### Feature 2: Template Expression Resolution + +A `{{context.*}}` expression system that resolves Radius application runtime context values into Recipe parameters at deploy time. Supports resource metadata, application/environment info, Kubernetes runtime, Azure, and AWS provider context. Includes single-level ternary expressions for conditional value mapping. + +### Feature 3: Output Mapping + +An `outputs` field on `RecipeDefinition` that maps module output names to resource property names. Provides a stable property interface for resource consumers regardless of the underlying module's output naming. + +### Feature 4: Private Registry/Repository Support + +Validate that direct recipe modules work with private registries and repositories using the existing `Radius.Core/terraformConfigs` and `Radius.Core/bicepConfigs` authentication mechanisms. Since direct modules use the same `source` field and driver download path as wrapper-based recipes, this is expected to be a no-op, the existing credential resolution logic should work. The investment here is primarily testing and documentation. + +## Usage Examples + +### Terraform Registry Module (AWS RDS) + +```bicep +resource recipepack 'Radius.Core/recipePacks@2025-08-01-preview' = { + name: 'aws-data-pack' + properties: { + recipes: { + 'Radius.Data/mySqlDatabases': { + kind: 'terraform' + source: 'terraform-aws-modules/rds/aws' + version: '5.9.0' + parameters: { + identifier: '{{context.resource.name}}' + db_name: '{{context.resource.properties.database}}' + engine: 'mysql' + engine_version: '8.0' + instance_class: 'db.t3.micro' + manage_master_user_password: true + create_db_subnet_group: true + subnet_ids: subnetIds + vpc_security_group_ids: vpcSecurityGroupIds + } + outputs: { + host: 'db_instance_address' + port: 'db_instance_port' + database: 'db_instance_name' + secretName: 'db_instance_master_user_secret_arn' + } + } + } + } +} +``` + +### Azure Verified Module (PostgreSQL) with T-Shirt Sizing + +```bicep +resource recipepack 'Radius.Core/recipePacks@2025-08-01-preview' = { + name: 'azure-postgres-pack' + properties: { + recipes: { + 'Radius.Data/postgreSqlDatabases': { + kind: 'terraform' + source: 'br:mcr.microsoft.com/bicep/avm/res/db-for-postgre-sql/flexible-server:0.4.0' + parameters: { + name: 'pg-{{context.resource.name}}' + location: 'eastus2' + sku_name: '{{context.resource.properties.size == "s" ? "B_Standard_B1ms" : "GP_Standard_D2s_v3"}}' + storage_mb: '{{context.resource.properties.size == "s" ? "32768" : "65536"}}' + tags: { + environment: '{{context.environment.name}}' + application: '{{context.application.name}}' + } + } + outputs: { + host: 'fqdn' + port: 'port' + database: 'database_name' + username: 'administrator_login' + } + } + } + } +} +``` + +### Environment-Level Parameter Overrides + +```bicep +resource devenv 'Radius.Core/environments@2025-08-01-preview' = { + name: 'dev' + properties: { + recipePacks: [recipepack.id] + recipeParameters: { + 'Radius.Data/mySqlDatabases': { + backup_retention_days: 2 + geo_redundant_backup_enabled: false + } + } + } +} +``` + +Environment parameters merge with Recipe-level parameters. Environment takes precedence for overlapping keys. + +### Secrets Output Mapping + +Properties marked as `x-radius-sensitive` in the Resource Type definition are automatically handled by Radius — their values are encrypted at rest and not exposed in plain text through the API. When a module output is mapped to a property that has `x-radius-sensitive: true`, Radius encrypts the value before storing it: + +```bicep +resource recipepack 'Radius.Core/recipePacks@2025-08-01-preview' = { + name: 'aws-data-pack' + properties: { + recipes: { + 'Radius.Data/mySqlDatabases': { + kind: 'terraform' + source: 'terraform-aws-modules/rds/aws' + version: '5.9.0' + parameters: { + identifier: '{{context.resource.name}}' + db_name: '{{context.resource.properties.database}}' + engine: 'mysql' + instance_class: 'db.t3.micro' + manage_master_user_password: true + } + outputs: { + host: 'db_instance_address' + port: 'db_instance_port' + database: 'db_instance_name' + password: 'db_master_user_secret_arn' // mapped to x-radius-sensitive property → encrypted by Radius + } + } + } + } +} +``` + +Radius determines sensitivity from the Resource Type schema — any output mapped to a property with `x-radius-sensitive: true` is automatically encrypted. + +## Edge cases + +1. **Module with no outputs** — A module that produces no outputs results in an empty Values map on the resource. The resource is still created successfully but has no populated properties from the Recipe. Radius logs a warning indicating that the module produced no outputs to help platform engineers catch misconfiguration early. +2. **Output name mismatch** — If the `outputs` mapping references a module output that does not exist, the mapped property is left empty (nil) on the resource. Radius logs a warning indicating the unresolved output name, so platform engineers can identify and fix the mapping without blocking deployment. +3. **Required parameters not supplied** — If the module requires an input variable that is not provided through `parameters` or environment-level `recipeParameters`, Radius surfaces the error from IaC engine indicating which variable is missing. +4. **Invalid `{{context.*}}` expression path** — If a template expression references a context path that does not exist (e.g., `{{context.resource.properties.nonexistent}}`), the expression resolves to an empty string. The IaC engine may then fail if the parameter requires a non-empty value. +5. **Large module with many unused variables** — Some community modules expose dozens of input variables with defaults. Only variables explicitly mapped in `parameters` are passed; all others use the module's defaults. This is expected behavior. +6. **Module version yanked or deleted** — If the pinned module version is removed from the registry, the Recipe deployment fails at the module download step with a fetch error. Mitigation: platform engineers should monitor upstream module availability. + +## Open Questions/ Designs (Reviewed on 6/10) + +1. Follow up with a separate design/gist to determine the approach for handling secrets in the context of Recipe modules. +2. Describe the user experience for platform engineers using pre-canned recipe packs from resource type contrib in the feature spec +3. Depending on external modules carries a supply-chain security risk even with version pinning, since bad actors could potentially overwrite pinned versions. SHA/digest-based pinning is the more secure approach. Investigate which source types support this (Git via `?ref=`, OCI via `@sha256:...`) and whether Radius should recommend or enforce it.