Code Club Walkthrough v2-aug14

[nitya]

Tracking issue for the full pre–Code Club walkthrough on branch v2-testing-aug14.

Goal: verify every lab end-to-end along two self-contained paths — a UI (portal-first) walkthrough and a CLI (azd/Copilot-first) walkthrough — hardening Fundamentals + Core first, then Capstone, then More Labs.

How to use: pick a path, then work top-to-bottom. Each path is ordered so it runs end-to-end in a single sitting with no forward references. Path A is listed first because it needs no local toolchain; Path B reuses the same subscription but a fresh azd env name.

Legend: ✅ = verified working as written · edits needed get a linked commit/PR.

---

🔗 Related issues

This walkthrough is the verification vehicle for the open issues below. Each is anchored to the checklist item that proves it out — close the issue when that item is checked and the fix has landed.

Issue	What it covers	Verified by
#1 — Remove Web Search Tool from Prompt Agent	Web search conflicts with grounded data; wants a default → grounded → optimize progression	A4, B5, A10 / B10
#2 — UI Path can cause duplicate RG creation after Step 5	UI provisioning + CLI hosted-agent deploy creates a second RG; ordering / resource reuse	A5, A5a, A5b, B1, B1a
#3 — Deploying hosted agent in Microsoft Internal tenant	Expired refresh token + undocumented RBAC needed for azd deploy	0.3, 0.6, B6b

---

0️⃣ Pre-flight (do once, before either path)

• 0.1 Clone / worktree is on v2-testing-aug14 and clean
• 0.2 pip install -r requirements-dev.txt && pytest -q passes on a clean tree — 14 passed in 0.67s in Codespace
• 0.3 az login + azd auth login against the target tenant (both, independently cached) — ref Deploying hosted agent in Microsoft Internal tenant issue #3
• 0.4 Subscription has quota/region availability for gpt-5.4-mini + gpt-5.4-judge
• 0.5 specs/course.yaml matches the actual lab set on disk — all 18 IDs resolve both ways, no orphans
• 0.5a Fix azd up → azd provision / azd deploy in the course.yaml titles for 01-provision-azd and 05-deploy-hosted-agent; reconcile the lone azd up at 01-provision-azd.md:100
• 0.6 Document the exact RBAC roles needed for hosted-agent deploy, and where a learner checks them — ref Deploying hosted agent in Microsoft Internal tenant issue #3

---

🖥️ Path A — UI walkthrough (Foundry portal first)

Run in this order.

• A1 fundamentals/00-overview.md — orientation; mermaid renders, all links resolve
• A2 fundamentals/02-provision-portal.md — provision Foundry project in the portal; record the RG name for reuse in A5
• A3 fundamentals/03-deploy-models.md — deploy both models via the portal
• A4 fundamentals/04-create-prompt-agent.md — create the Prompt Agent in the portal — ref Remove Web Search Tool from Prompt Agent #1
  • A4a Default agent (web search on) answers ungrounded — confirm the lab shows this
  • A4b After attaching data/*.csv, answers are a mix of web + grounded — confirm the lab shows this
  • A4c Lab states the fix is deferred to the optimize lab (A10), not done here
• A5 fundamentals/05-deploy-hosted-agent.md — confirm the "portal cannot create hosted agents" callout is accurate; portal is view/manage only
  • A5a Hand-off check — the CLI deploy step reuses the RG/project from A2 and does not create a second RG — ref Using UI Path can cause duplicate RG creation after Step 5 #2
  • A5b Mixed-path linker — run ./scripts/link-portal-rg.sh before Lab 05 Step 1; confirm it discovers the RG + Foundry account + project, seeds the azd env, and azd provision reuses the existing RG (no second RG created) — ref Using UI Path can cause duplicate RG creation after Step 5 #2
• A6 fundamentals/06-verify.md — Fundamentals checkpoint passes from the portal
• A7 core/00-overview.md — map table + prerequisites match what A1–A6 produced
• A8 core/01-observe-portal.md — invoke the agent, first traces appear in the portal
• A9 core/02-evaluate-portal.md — sample dataset created + quality evaluator runs
• A10 core/03-optimize-skills.md — optimizer produces a new prompt; review results in portal — ref Remove Web Search Tool from Prompt Agent #1 (this is where web search gets removed or scoped by instruction)
• A11 core/04-monitor-portal.md — post-change evidence shows the improvement
• A12 core/05-capstone-hosted.md — capstone loop completes, verified in the portal
• A13 Teardown — azd down --purge --force (or portal delete); confirm no orphaned RG is left behind — ref Using UI Path can cause duplicate RG creation after Step 5 #2

---

⌨️ Path B — CLI walkthrough (azd + Copilot CLI)

Run in this order. Use a new azd env name rather than reusing Path A's.

• B1 azd env new <fresh-name> — avoids stale server-side state; lab should also explain how to point an env at an existing portal-created RG for the mixed UI→CLI case — ref Using UI Path can cause duplicate RG creation after Step 5 #2
  • B1a scripts/link-portal-rg.sh runs cleanly against a portal-created RG (idempotent, no destructive side effects) and correctly sets AZURE_RESOURCE_GROUP, AZURE_AI_ACCOUNT_NAME, USE_EXISTING_AI_PROJECT=true, AI_PROJECT_DEPLOYMENTS="[]" — ref Using UI Path can cause duplicate RG creation after Step 5 #2
• B2 fundamentals/00-overview.md — orientation (CLI framing)
• B3 fundamentals/01-provision-azd.md — azd provision succeeds
  • B3a Gotcha: invalid character 'n' after object key:value pair — the four *Json params in infra/main.parameters.json; workaround azd env set AI_PROJECT_DEPLOYMENTS "[]"
• B4 fundamentals/03-deploy-models.md — bicep defaults deployed (gpt-5.4-mini, gpt-5.4-judge)
• B5 fundamentals/04-create-prompt-agent.md — prompt agent via CLI / agent.yaml — ref Remove Web Search Tool from Prompt Agent #1 (tool config matches the portal path)
• B6 fundamentals/05-deploy-hosted-agent.md — azd deploy creates the hosted agent
  • B6a Gotcha: 409 modified concurrently → azd down --purge --force → azd env new → re-provision → re-deploy
  • B6b Gotcha: expired/stale token — 404 Subdomain does not map to a resource and refresh token has expired due to inactivity; re-auth both az and azd — ref Deploying hosted agent in Microsoft Internal tenant issue #3
• B7 fundamentals/06-verify.md — Fundamentals checkpoint passes from the CLI
• B8 core/01-observe-portal.md — drive invocations from the CLI to generate traces
• B9 core/02-evaluate-portal.md — batch eval run from the CLI
• B10 core/03-optimize-skills.md — Copilot + microsoft-foundry skill prompt optimization — ref Remove Web Search Tool from Prompt Agent #1
• B11 core/04-monitor-portal.md — CLI re-eval + before/after comparison
• B12 core/05-capstone-hosted.md — capstone loop end-to-end from the CLI
• B13 scripts/reset.sh restores src/ from src.original/ after the optimize labs
• B14 Teardown — azd down --purge --force

---

🧪 More Labs (explore after both paths land)

• M1 more/README.md — index is accurate and links resolve
• M2 more/continuous-eval.md
• M3 more/trace-driven-datasets.md
• M4 more/red-teaming.md
• M5 more/troubleshooting.md — every gotcha hit in Path A/B is captured here (incl. Using UI Path can cause duplicate RG creation after Step 5 #2 duplicate-RG and Deploying hosted agent in Microsoft Internal tenant issue #3 auth/RBAC)

---

🔧 Repo-level checks (after walkthroughs, before Code Club)

• R1 Every lab conforms to labs/_template/lab-template.md (H1 Lab NN — …, 🎯 🧭 📋 ✅ 🧠 ➡️ sections)
• R2 Every generative step ships a reference/ artifact under artifacts/
• R3 Lab gotcha callouts stay in sync with AGENTS.md known-gotchas
• R4 Workshop-coach agent locates the learner correctly via progress-tracker
• R5 S1–S11 + T1–T7 complete (all TODO(nitya) items either resolved or converted to real tickets)
• R6 pytest -q still passes after all walkthrough edits
• R7 Remove Web Search Tool from Prompt Agent #1, Using UI Path can cause duplicate RG creation after Step 5 #2, Deploying hosted agent in Microsoft Internal tenant issue #3 all resolved and closed
• R8 Decide: add a mermaid-syntax guardrail to the test suite (30 diagrams, currently zero coverage — a broken one is invisible to pytest and shows as a red error box on GitHub). Validated manually this pass via mermaid@11 parse(); a permanent check needs a Node dep in a Python repo, or a lightweight regex-based structural test.

---

📸 Screenshots to capture

One item per lab; count in () = number of <!-- TODO(nitya): screenshot --> markers currently in that file. Tick when all placeholders in the file have been replaced with real images under labs/<phase>/images/.

• S1 fundamentals/00-overview.md (1) — architecture diagram (Concierge → specialists → CSVs)
• S2 fundamentals/01-provision-azd.md (2) — device-code prompt, RG in Azure portal
• S3 fundamentals/02-provision-portal.md (3) — Foundry landing, Create-project form, Project-endpoint field
• S4 fundamentals/03-deploy-models.md (3) — Deploy button, Succeeded state, Models list
• S5 fundamentals/05-deploy-hosted-agent.md (2) — azd deploy success, azd ai agent invoke output
• S6 fundamentals/06-verify.md (2) — Prompt-agent playground, Hosted-agent playground with trace panel
• S7 core/01-observe-portal.md (4) — playground + logs, metrics dialog, response with AI Quality, polite-refusal + scope score
• S8 core/02-evaluate-portal.md (3) — New-evaluation flow, evaluator-selection panel, low-scoring row with rationale
• S9 core/03-optimize-skills.md (3) — Copilot Chat + MCP setup, streaming observe workflow, redeploy before/after
• S10 core/04-monitor-portal.md (2) — Monitor tab (all charts), Tokens chart + AI-generated analysis
• S11 more/red-teaming.md (1) — safety-evaluator selection panel

🧹 Non-screenshot TODO(nitya) items

Confirm each of these is resolved (or deliberately deferred with an updated marker) before Code Club.

• T1 infra/main.bicep — structural fix for the four *Json params (change to array/object, drop json(), un-quote in main.parameters.json)
• T2 infra/README.md — confirm region + SKU choices for the standardized models
• T3 scripts/provision-portal.md — fill in the portal screenshots + step-by-step instructions (or link to Lab 02 and delete)
• T4 src/README.md — confirm the exact azd ai agent run incantation
• T5 labs/more/continuous-eval.md (line 47) — swap in real command once the eval runner is finalized
• T6 labs/more/continuous-eval.md (line 89) — confirm runner + secrets story (prefer OIDC)
• T7 labs/more/trace-driven-datasets.md (line 43) — paste the canonical KQL query

🎨 Visual navigation spot-checks

For each Fundamentals lab, confirm the compact "📍 You are here" mermaid banner renders on GitHub and that the highlighted (teal) node matches the lab file.

• V1 fundamentals/00-overview.md banner highlights 00
• V2 fundamentals/01-provision-azd.md banner highlights 01
• V3 fundamentals/02-provision-portal.md banner highlights 02
• V4 fundamentals/03-deploy-models.md banner highlights 03
• V5 fundamentals/04-create-prompt-agent.md banner highlights 04
• V6 fundamentals/05-deploy-hosted-agent.md banner highlights 05
• V7 fundamentals/06-verify.md banner highlights 06
• V8 The RG-context diagram at the bottom of 00-overview.md (➡️ Next section) still renders and its ⚠️ warning box names scripts/link-portal-rg.sh

---

This issue is kept up to date as the walkthrough progresses — items are added, split, or checked off as they're verified.

[nitya]

Pre-flight progress

0.5 — `specs/course.yaml` matches the lab set on disk ✅

All 18 lab IDs resolve to files, and every lab file is registered — no orphans in either direction (fundamentals 7, core 6, more 4).

One drift found (not blocking, worth fixing before Code Club): course.yaml still titles two labs with azd up:

• 01-provision-azd → "Provision Foundry with azd up"
• 05-deploy-hosted-agent → "Deploy the Hosted Agent (azd up)"

The lab bodies use azd provision + azd deploy consistently (as do the AGENTS.md gotchas). azd up appears exactly once in 01-provision-azd.md (line 100) and never in lab 05. Since these titles surface in nav/index, a learner can read azd up and type it. Tracking as 0.5a below.

• 0.5a Fix azd up → azd provision / azd deploy in specs/course.yaml titles, and reconcile the lone azd up mention in 01-provision-azd.md:100

Remaining pre-flight (0.1, 0.2, 0.3, 0.4, 0.6) is Codespace/subscription-specific — running there.

[nitya]

Handoff — session paused, continuing in Codespaces

Environment: Codespace on branch v2-testing-aug14.

Pre-flight so far

Item	Status	Evidence
0.1 branch + clean tree	✅	on v2-testing-aug14, no dirty files
0.2 pytest -q	✅	14 passed in 0.67s
0.3 az + azd login	⬜ next	Codespaces needs --use-device-code for both
0.4 model quota / region	⬜	gpt-5.4-mini, gpt-5.4-judge
0.5 course.yaml ↔ labs	✅	18 IDs resolve both ways
0.5a azd up title drift	⬜ open	fix in specs/course.yaml + 01-provision-azd.md:100
0.6 RBAC roles documented	⬜	blocks #3

Next up: 0.3, then 0.4 and 0.6, then start Path A (portal) at A1.

Commands for the remaining pre-flight:

az login --use-device-code
azd auth login --use-device-code
az account show --query "{sub:name, id:id, tenant:tenantId}" -o table

az cognitiveservices usage list -l eastus2 -o table | grep -i -E "gpt|token" | head

az role assignment list --assignee $(az ad signed-in-user show --query id -o tsv) \
  --all --query "[].roleDefinitionName" -o tsv | sort -u

Watch for: the two known auth failure modes on azd deploy — refresh token has expired due to inactivity and 404 Subdomain does not map to a resource. Both mean stale tokens, both require re-auth of az and azd (independent caches). See #3 / item B6b.

[nitya]

0.1 verified — ran git rev-parse --abbrev-ref HEAD && git status --porcelain | head in Codespace; branch is v2-testing-aug14 and the worktree is clean.

[nitya]

0.2 verified — ran pip install -r requirements-dev.txt && pytest -q in Codespace: 14 passed in 0.40s.

[nitya]

Pre-flight — devcontainer dependencies verified (per Quickstart):

Command	Output
python --version	Python 3.13.5
az version	azure-cli 2.89.1
azd version	1.31.1 (stable)
copilot --version	GitHub Copilot CLI 1.0.80 (latest)
gh --version	2.97.0 (2026-07-31)

[nitya]

0.3 verified — followed the "Steps" in labs/fundamentals/01-provision-azd.md:

Command	Result
az login --use-device-code	completed
az account show	verified active subscription
azd auth login --use-device-code	completed
azd auth status	verified logged in

Both az and azd tokens are cached independently in this Codespace.

[nitya]

0.4 verified via microsoft-foundry quota sub-skill.

Learner-runnable prompt (paste into a Copilot chat with the microsoft-foundry skill + Azure MCP available):

Use the microsoft-foundry quota skill to confirm I have GlobalStandard capacity 100 in eastus2 for gpt-5.4-mini@2026-03-17 and gpt-5.4@2026-03-05. If not, recommend the best of eastus2 / swedencentral / northcentralus.

Process the skill drives (quota/quota.md → workflow #3 "Check Quota Before Deployment"):

1. az account show --query id -o tsv → capture subscription id.
2. For each candidate region, call GET https://management.azure.com/subscriptions/{subId}/providers/Microsoft.CognitiveServices/locations/{region}/usages?api-version=2023-05-01 via az rest.
3. Filter rows to the target model family (OpenAI.GlobalStandard.gpt-5.4-mini, OpenAI.GlobalStandard.gpt-5.4) and compute Available = limit - currentValue.
4. Pass criterion: Available ≥ capacity requested in infra/main.bicep (bicep asks capacity 100 per model, units are K TPM).
5. If eastus2 fails, compare against swedencentral / northcentralus and pick the largest headroom.

Cross-checked with Foundry MCP model_catalog_list: both model asset IDs and versions in bicep match the AOAI registry exactly (gpt-5.4-mini@2026-03-17, gpt-5.4@2026-03-05).

Final quota confirmation lands at azd provision (task B3).

[nitya]

0.5a done — reconciled the azd up drift between specs/course.yaml and the lab bodies.

Changes:

File / Line	Before	After
specs/course.yaml #L27	title: Provision Foundry with \azd up``	title: Provision Foundry with \azd provision``
specs/course.yaml #L59	title: Deploy the Hosted Agent (\azd up`)`	title: Deploy the Hosted Agent (\azd deploy`)`

Left intentionally unchanged: the > 💡 callout in labs/fundamentals/01-provision-azd.md that defines azd up = azd provision + azd deploy and explains why the workshop splits them. That mention of azd up is the pedagogical justification for the split, not stale drift.

Guardrail: pytest -q → 14 passed.

[nitya]

Mixed UI→CLI path + per-lab navigation — pre-fixes landed

Two changes to make the pre-Code-Club walkthrough smoother. New validation items added above (A5b, B1a, V1–V8).

1. Fix the "two resource groups" gotcha (ref #2)

• NEW scripts/link-portal-rg.sh — auto-discovers a portal-created RG + Foundry account + project and binds them into an azd env (idempotent). Sets USE_EXISTING_AI_PROJECT=true and AI_PROJECT_DEPLOYMENTS="[]" so Lab 05's provisioning gotcha is preempted.
• Lab 05 gets a "🧭 Path check" preface: portal-first learners run ./scripts/link-portal-rg.sh once before Step 1; CLI-first learners skip it.
• Lab 02 now has a forward-reference tip pointing to the linker.
• Lab 00 overview mermaid warning box now names the script explicitly.
• AGENTS.md — new bullet under Known gotchas describing the mixed-path scenario and the recovery.

2. Per-lab navigation banner (color-coded)

Added a compact "📍 You are here" mermaid banner right after each Fundamentals lab's "This lab covers" callout. Shows the 7-lab flow (00 → 01/02 → 03 → 04 → 05 → 06) with the current node highlighted in the workshop teal (#0ea5e9, matching the DevOps-loop active node style). Purely visual — no lab content changed.

Guardrail

pytest -q → 14 passed.

Validation notes

• A5b / B1a: run ./scripts/link-portal-rg.sh end-to-end on a Path-A portal-provisioned RG and confirm no second RG appears after azd provision in Lab 05.
• V1–V8: open each Fundamentals lab on GitHub, confirm the mermaid banner renders and the teal-highlighted node matches the file.

[nitya]

V1–V8 verified ✅ — progress banners + RG-context diagram

V1–V7 — “📍 You are here” banners. All 7 fundamentals labs carry a structurally identical banner, each highlighting the correct node via :::active:

Lab	Active node
00-overview.md	L0 00 overview
01-provision-azd.md	L1 01 azd
02-provision-portal.md	L2 02 portal
03-deploy-models.md	L3 03 models
04-create-prompt-agent.md	L4 04 prompt agent
05-deploy-hosted-agent.md	L5 05 hosted agent
06-verify.md	L6 06 verify

The branch/merge topology is correct and consistent in all 7: L0 forks to L1 (CLI) and L2 (portal), and both converge on L3 — so the two provisioning doors visibly rejoin before Lab 03. classDef active is defined in every block (teal #0ea5e9, matching the DevOps-loop active-node style).

V8 — RG-context diagram in the ➡️ Next section of 00-overview.md: renders, and the ⚠️ warning node explicitly names ./scripts/link-portal-rg.sh with the reason ("Otherwise azd deploy creates a second RG"). Correctly wired as a dotted watch out edge off the rg-contoso-travel subgraph. Ref #2.

---

Bonus: all 30 mermaid diagrams validated with the real parser

Rather than eyeballing, I parsed every mermaid block in the repo through mermaid@11’s own parse() (jsdom, securityLevel: strict — same setting GitHub renders with):

=== 30 diagrams parsed, 0 failed ===

So no diagram will render as an error box during Code Club.

⚠️ Gap worth noting: the test suite has no mermaid coverage — grep -rn mermaid tests/ returns nothing, despite 30 diagrams now carrying real pedagogical weight. A broken diagram is invisible to pytest and only shows up as a red error box in the browser. Adding a guardrail would mean a Node dep in a Python repo, so flagging as a decision rather than doing it — tracked as R8 below.

[nitya]

R1 / R3 sweep — findings (⚠️ not yet fixed)

Two issues found, no changes made — handing off.

🔴 R3-a — Lab 01 invites the learner straight into the JSON gotcha

labs/fundamentals/01-provision-azd.md L154–156:

💡 Want different models? The golden path deploys gpt-5.4-mini and a gpt-5.4-judge. To swap them, set the AI_PROJECT_DEPLOYMENTS env var before azd provision — see Lab 03.

This tells the learner to set the exact variable that breaks azd provision (invalid character 'n' after object key:value pair) — with no warning. The gotcha is documented in Lab 03 and Lab 05, but not at the point of first exposure, and Lab 01 is where a CLI-path learner first runs azd provision.

Suggested fix: add a > ⚠️ warning to that callout pointing at azd env set AI_PROJECT_DEPLOYMENTS "[]", or defer the customization advice entirely to Lab 03.

Gotcha coverage matrix (all 5 AGENTS.md entries are present somewhere):

Gotcha	Labs covering it	Gap?
409 modified concurrently	05	—
Portal cannot create hosted agents	05 (L138)	—
Mixed UI→CLI second RG	00, 02, 05	—
invalid character 'n'	03, 05	⚠️ missing in 01
404 Subdomain does not map	05	—

🟡 R1-a — H1 naming is inconsistent across phases

test_lab_content.py enforces the six pedagogy sections (🎯 🧭 📋 ✅ 🧠 ➡️) and all 18 labs pass. But the AGENTS.md rule "H1 title starting with Lab NN — ..." is not tested, and the titles have drifted:

File	H1	Note
fundamentals/00–06	# Lab NN — …	consistent
core/00-overview.md	# Lab 00 — Core Labs overview	⚠️ collides with fundamentals/00
core/01–05	# Core Lab NN — …	consistent
more/*.md	# More Lab — …	unnumbered; deviates from Lab NN rule

Two learners saying "I'm on Lab 00" can mean different labs. Suggested fix: core/00 → # Core Lab 00 — Core Labs overview; decide whether the Lab NN rule in AGENTS.md should formally exempt More Labs.

ℹ️ Devcontainer note (relates to R4)

.devcontainer/devcontainer.json has the coach-progress mount commented out:

// "mounts": [ "source=${localWorkspaceFolderBasename}-coach,target=/home/vscode/.contoso-coach,type=volume"]

So workshop-coach progress-tracker state does not survive a container rebuild. Worth deciding before Code Club.

---

Status: V1–V8 ✅ · R1/R3 swept (findings above, unfixed) · continuing in Codespaces.