Skip to content

fix: harden PVC permissions and paid smoke flows#615

Merged
bussyjd merged 3 commits into
research/hermes-configmap-skills-deliveryfrom
research/pv-permission-smoke-hardening
Jun 9, 2026
Merged

fix: harden PVC permissions and paid smoke flows#615
bussyjd merged 3 commits into
research/hermes-configmap-skills-deliveryfrom
research/pv-permission-smoke-hardening

Conversation

@bussyjd

@bussyjd bussyjd commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
  • keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
  • harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
  • update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

  • bash -n flows/*.sh
  • git diff --check
  • go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
  • full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

@bussyjd

bussyjd commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Summary

What changed:

  • Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
  • Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
  • Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
  • Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
  • Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

  • Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
  • Reduces fragile permission scaffolding while keeping runtime writes available.
  • Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

  • Code
  • Charts / manifests
  • Flows / QA scripts
  • Docs / skills
  • Images / dependencies
  • Other:

Validation

CI checks:

Check Status Link
CI not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1

Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow Network QA machine label Worktree Result Artifacts
flow-01-prerequisites local remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-02-stack-init-up local k3d remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-03-inference local k3d remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-04-agent local k3d remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-05-network Base Sepolia RPC remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-06-sell-setup Base Sepolia x402 pricing remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-07-sell-verify x402 tunnel route remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-10-anvil-facilitator Base Sepolia Anvil fork remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-08-buy Base Sepolia Anvil fork / USDC remote ARM64 QA host redacted PASS receipt retained on QA host
flow-09-lifecycle local k3d remote ARM64 QA host redacted PASS release-smoke report retained on QA host
flow-11-dual-stack Base Sepolia / USDC remote ARM64 QA host redacted SKIP only for optional registration receipt receipt summary retained on QA host
flow-14-live-obol-base-sepolia live Base Sepolia / OBOL remote ARM64 QA host redacted SKIP only for optional registration receipt receipt summary retained on QA host
flow-13-dual-stack-obol Base Sepolia Anvil fork / OBOL remote ARM64 QA host redacted PASS receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

  • Base Sepolia
  • Base Sepolia Anvil fork

RPC/provider:

  • Paid Base Sepolia RPC, redacted
  • Local Anvil fork for forked flows

Facilitator:

  • Public Obol facilitator for live Base Sepolia flows, hostname omitted
  • Local x402-rs facilitator for fork flow

Contracts and tokens:

Name Address Version / notes
Base Sepolia OBOL 0x0a09371a8b011d5110656ceBCc70603e53FD2c78 live token
Base Sepolia USDC 0x036CbD53842c5426634e7929541eC2318f3dCF7e live/fork token
ERC-8004 identity registry 0x8004A818BFB912233c491871b3d84c89A494BD9e Base Sepolia
Fork OBOL 0x210BBd033630e5e611B7922D70b0Caabe64636d9 flow-13 deployed token

Wallet roles:

Role Address Source
Alice / seller / register 0xC0De030F6C37f490594F93fB99e2756703c4297E runtime-derived from configured signer
Bob / buyer / payer 0x57b0eF875DeB5A37301F1640E469a2129Da9490E deterministic second-derived buyer
Single-stack flow-08 seller 0x70997970C51812dc3A010C7d01b50e0d17dc79C8 local flow seller

Balances:

Token Address Before After Expected delta Actual delta
Live OBOL, Alice 0xC0De030F6C37f490594F93fB99e2756703c4297E 30000000000000000 31000000000000000 +1000000000000000 +1000000000000000
Live OBOL, Bob 0x57b0eF875DeB5A37301F1640E469a2129Da9490E 4947000000000000000 4946000000000000000 -1000000000000000 -1000000000000000
Fork OBOL, Alice 0xC0De030F6C37f490594F93fB99e2756703c4297E 10000000000000000000 10001000000000000000 +1000000000000000 +1000000000000000
Fork OBOL, Bob 0x57b0eF875DeB5A37301F1640E469a2129Da9490E 10000000000000000000 9999000000000000000 -1000000000000000 -1000000000000000

Transaction receipts:

Purpose Tx hash From To Amount / event Status
ERC-8004 registration 0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310 Alice registry Agent ID reflected in ServiceOffer PASS
Metadata / service offer Metadata tx unavailable, ServiceOffer status reflected Agent ID SKIP
Live OBOL approval / permit 0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c Bob Permit2 Max allowance confirmed PASS
Live OBOL settlement transfer 0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981 Bob Alice 1000000000000000 wei OBOL PASS
Fork OBOL buyer funding 0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de deployer Bob 10 OBOL minted PASS
Fork OBOL approval / permit 0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8 Bob Permit2 Max allowance confirmed PASS
Fork OBOL settlement transfer 0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d Bob Alice 1000000000000000 wei OBOL PASS
Flow-08 USDC settlement transfer 0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2 buyer seller 1000 micro-USDC PASS
Flow-11 USDC settlement transfer 0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df Bob Alice 1000 micro-USDC PASS

Runtime Evidence

QA environment:

Item Value
OS / arch Linux / arm64 remote QA host
Backend k3d
Tool versions branch-built obol CLI, Foundry tools available
QA agent/model qwen36-deep via redacted OpenAI-compatible endpoint

Images:

Component Image Tag / digest Source
serviceoffer-controller local dev image rebuilt during smoke branch source
x402 facilitator ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay 1.4.9 flow-13 preflight

Kubernetes / stack:

Item Value
Stack IDs omitted to avoid host-local details
Namespaces llm, x402, erpc, hermes-obol-agent
Pod readiness all smoke-gated pods Ready during flow checks
Cleanup result no k3d clusters remaining after final smoke

Model and routing:

Item Value
Agent/model used qwen36-deep
LiteLLM route paid/qwen36-deep
Paid endpoint status HTTP 200 with non-empty response
Auth token source PurchaseRequest plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact Location / link Notes
Release smoke report retained on QA host, path omitted result PASS
Flow-11 receipts retained on QA host, path omitted USDC settlement
Flow-14 receipts retained on QA host, path omitted live OBOL settlement
Flow-13 receipts retained on QA host, path omitted fork OBOL settlement

Demo readiness:

Item Status Notes
Seller visible / registered PASS ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works PASS Buyer discovered service catalog / registry data
Paid route works PASS paid/qwen36-deep returned HTTP 200
Settlement visible on-chain PASS Transfer receipts and exact deltas verified

Review Notes

Known gaps:

  • flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
  • PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

  • None required for the validated smoke path.

Reviewer focus:

  • UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
  • LiteLLM ConfigMap reload behavior for paid routes.
  • Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
  • Post-stack Bob wallet import in dual-stack flows.

bussyjd added a commit that referenced this pull request Jun 9, 2026
@bussyjd
chore(plans): drop shipped/absorbed plan docs ahead of v0.10.0
c07c441 
Removed plans whose work landed and whose durable learnings already live in
the skill references:
- obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI)
- post-490-integration-20260513.md — integration landed in May
- release-smoke-hardening-20260513.md — session retro; learnings folded into
  obol-stack-dev/references/release-smoke-debugging.md
- inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA
  findings live in release-smoke-debugging.md §10 (c2dddc1)

Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending
walk-through), openapi-redoc-storefront.md (phase 2 deferred),
storefront-buy-inference-cta-handoff.md (active frontend handoff),
volume-permission-hardening.md (PR #615 owns its rewrite).
bussyjd added 2 commits June 10, 2026 02:27
@bussyjd
fix(pvc): legacy hostPath-PV compat, scoped Reloader, flow alignment …
c667432 
…with #614

Applies the verified findings from the cross-review against PR #614 (every
item adversarially confirmed against the refs; union merge-tree clean).

PVC / upgrade path:
- llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer.
  Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where
  kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json
  written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on
  `load state`, and takes every paid/<model> route down. On fresh local-type
  PVs the explicit UID is harmless (fsGroup 65532 grants group access).
  embed_buyer_state_test.go updated to pin the new contract.
- plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12"
  section — supported path is cluster recreation (wallet backup/restore),
  with a documented k3d chown escape hatch. troubleshooting.md gets the
  symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be
  patched at runtime without reintroducing the chown machinery this PR
  removes, so it is a documented breaking change instead.

Paid-route availability:
- llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer
  ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the
  controller on every buy, top-up, auto-refill, and tombstone cleanup;
  with strategy Recreate + 1 replica the previous annotation bounced the
  entire inference gateway (all Hermes traffic, in-flight SSE streams) on
  every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback,
  not the default buy path). The buyer hot-reloads via /admin/reload.
  stack_test.go updated to pin litellm-config-only.

Flow alignment with #614:
- lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's
  flow-16 (now last in the single-stack array) intentionally leaves a live
  agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate
  refuses, graceful down is silently skipped (`|| true`), and teardown
  degrades to the raw k3d-delete fallback on every release-smoke run.
- flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical
  live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling).

Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N
assertions on rare partial failures; flow-11 lacks flow-14's remote-signer
rolled guard; aztec PVC has no permission story (runs as root today);
post-merge controller repin so released sub-agents pick up this PR's
UID-1000 render (tracked in #614's pin-test note).
@bussyjd
docs(plans): multi-node roadmap for the volume-permission direction
d8a390a 
Records the options ladder for 'nodes joining my cluster' (far-future, not
scheduled) so near-term permission decisions don't foreclose it: the
host-FS-canonical coupling and its four conditions, the agent-homes-only
scope, what breaks on day one of a join (WaitForFirstConsumer + no
nodeSelector in any render), options 0-4 (home-node pattern -> NFS ->
distributed storage -> API-mediated access -> hybrid by data class), join
mechanics (native k3s server + remote agents; macOS stays k3d), and the
decisions that bind today: group-1000 sharing over render-time UID
matching, home-node pinning as a join prerequisite, inputs to API objects
per Remaining Debt.
@bussyjd
bussyjd requested a review from OisinKyne June 9, 2026 22:58
@bussyjd
bussyjd merged commit 888515b into research/hermes-configmap-skills-delivery Jun 9, 2026
This was referenced Jun 10, 2026
Closed
Merged
OisinKyne pushed a commit that referenced this pull request Jun 10, 2026
@bussyjd @OisinKyne
chore(plans): drop shipped/absorbed plan docs ahead of v0.10.0
7cfd463 
Removed plans whose work landed and whose durable learnings already live in
the skill references:
- obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI)
- post-490-integration-20260513.md — integration landed in May
- release-smoke-hardening-20260513.md — session retro; learnings folded into
  obol-stack-dev/references/release-smoke-debugging.md
- inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA
  findings live in release-smoke-debugging.md §10 (c2dddc1)

Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending
walk-through), openapi-redoc-storefront.md (phase 2 deferred),
storefront-buy-inference-cta-handoff.md (active frontend handoff),
volume-permission-hardening.md (PR #615 owns its rewrite).
@OisinKyne
OisinKyne deleted the research/pv-permission-smoke-hardening branch July 1, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@OisinKyne OisinKyne Awaiting requested review from OisinKyne

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd