Fix corrupt state archive: self-heal the default group and guard rad shutdown

[sk593]

Description

The repo-based deploy pipeline (run-rad-commands-azure.yml / run-rad-commands-aws.yml, and the delete-* workflows) restores control-plane state at setup with rad startup and persists it at teardown with rad shutdown. rad startup restores the previous run's snapshot with pg_dump --clean --if-exists — a wholesale drop-and-replace of the control-plane PostgreSQL databases, not a merge.

Two independent problems corrupted the shared state archive and made deploys fail deterministically with resource group "default" not found:

1. A corrupt (group-less) archive never repaired itself. After Revert default resource group and environment creation from rad install kubernetes #12829 stopped rad install kubernetes from re-seeding the default group server-side, the pipeline created it before rad startup. Since startup restores the databases wholesale, a group created before it is dropped whenever the restored snapshot lacks the group — so once the archive was group-less it stayed that way, and every deploy failed:

   21:43:55  rad group create default                 # created...
   21:43:57  Restoring control-plane databases...      # ...then dropped by the wholesale restore
   21:44:21  State restored successfully.
   21:44:22  rad deploy $ENV_BICEP  ->  NotFound: resource group ".../resourcegroups/default" not found
   

2. A pre-startup failure overwrote the good archive with blank state. rad shutdown ran unconditionally under if: always(). A workflow that failed before rad startup completed (setup, cluster connect, OIDC, credential registration, or startup itself) still hit teardown, backed up the uninitialized databases, and overwrote the archive. The next run restored that blank state wholesale and re-persisted it — self-perpetuating corruption.

Fix

A two-part fix; neither half is sufficient alone.

1. Create the default group after rad startup (self-heal). Moving rad group create/switch below the restore means the group is (re)created on top of whatever the restore produced, so it is present for the deploy and persisted by the next rad shutdown. A group-less archive now heals itself on the next successful run. rad startup has no resource-group dependency, so nothing needs the group earlier.

2. Guard rad shutdown on a successful restore. restore-state exposes a state-restored output, set to true only after rad startup succeeds. The workflow passes it into teardown's state-restored input; when it is not true, teardown skips rad shutdown (emitting a ::warning::) so a run that never reached startup cannot overwrite the archive. This replaces the earlier ${RUNNER_TEMP} file marker with an explicit action output→input, so the dependency is visible in the workflow YAML and fails safe if the steps are ever split across jobs.

This preserves the behaviors we do want:

• First-run seeding. rad startup is a no-op on the first run (empty archive) but still succeeds and sets the output, so rad shutdown runs and seeds the archive.
• Persist after a later failure. If startup succeeded and a later step fails (e.g. a partially-applied Terraform deploy), the output is true, so rad shutdown still runs and that work is not lost.

The guard is pure workflow orchestration — it does not couple rad startup and rad shutdown as commands.

Also in this PR

• teardown: rad app list → rad app list --preview. The pipeline deploys Radius.Core applications, but the teardown status step queried the legacy Applications.Core plane and listed nothing. --preview matches the surface the rest of the pipeline uses, and a failed listing now surfaces a ::warning:: instead of being swallowed by || true.

Type of change

• Bugfix

Fixes

Fixes #12838

Testing

• .github/extension/actions/teardown/teardown_test.sh (wired into build/test.mk as test-teardown) extracts the actual run: blocks from the restore-state and teardown composite actions and executes them with stubbed rad/git (no cluster/CLI), asserting real behavior:
  • restore-state sets state-restored=true on $GITHUB_OUTPUT after rad startup, and creates the default group after startup (order asserted from the invocation log).
  • Negative path: when rad startup fails, the block exits non-zero and never sets the output.
  • First run: a no-op rad startup still sets the output.
  • teardown runs rad shutdown only when state-restored == "true", and otherwise skips with a ::warning::.
  • teardown lists apps via rad app list --preview and warns (does not swallow) on failure.
  • All four workflows wire restore-state → teardown within the same job.
    Verified by mutation that each invariant, when broken, fails the test.
• make test-teardown and make test-extension-action-shell-syntax pass; the test is shellcheck-clean.

Follow-ups (out of scope)

Raised in review and intentionally deferred — neither is required to fix #12838:

• rad shutdown should refuse to persist a degenerate/empty state snapshot #12847 — rad shutdown should refuse to persist a degenerate/empty snapshot (defense-in-depth for control-plane degradation after a successful startup).
• Add concurrency: group to repo-based deploy/delete workflows to serialize shared state-archive access #12848 — add a concurrency: group to serialize workflows that share the state archive.

Notes

Example failing runs (before this fix): https://github.com/sk593/aks-store-demo/actions/runs/33015265288, https://github.com/sk593/aks-store-demo/actions/runs/33016468544

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Copilot]

Pull request overview

Fixes a deterministic failure in the repo-based deploy pipeline where the default resource group was being created before rad startup and then wiped when rad startup restored the control-plane database snapshot.

Changes:

• Stop creating/switching the default resource group in the “Configure Radius workspace” step (leave workspace creation only).
• Create and switch to the default resource group after rad startup completes, with expanded inline documentation explaining why.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Unit Tests

    2 files  ±0    459 suites  ±0   8m 45s ⏱️ +27s
6 518 tests ±0  6 516 ✅ ±0  2 💤 ±0  0 ❌ ±0 
7 794 runs  ±0  7 792 ✅ ±0  2 💤 ±0  0 ❌ ±0 

Results for commit 99dedbd. ± Comparison against base commit f00bb86.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.98%. Comparing base (f00bb86) to head (99dedbd).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #12840   +/-   ##
=======================================
  Coverage   59.97%   59.98%           
=======================================
  Files         776      776           
  Lines       46410    46410           
=======================================
+ Hits        27836    27837    +1     
+ Misses      18574    18573    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Functional Tests - corerp-noncloud

188 tests  ±0   186 ✅ ±0   1h 12m 50s ⏱️ -57s
  3 suites ±0     2 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 2eef265. ± Comparison against base commit f00bb86.

♻️ This comment has been updated with latest results.

[radius-functional-tests]

Radius functional test overview

🔍 Go to test action run

Click here to see the test run details

Name	Value
Repository	radius-project/radius
Commit ref	99dedbd
Unique ID	func9288c505fe
Image tag	pr-func9288c505fe

• Dapr: 1.14.4
• Azure KeyVault CSI driver: 1.4.2
• Azure Workload identity webhook: 1.3.0
• Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func9288c505fe
• Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
• applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func9288c505fe
• dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func9288c505fe
• controller test image location: ghcr.io/radius-project/dev/controller:pr-func9288c505fe
• ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func9288c505fe
• deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded