fix(pvc): kubelet-owned volume permissions — local-type PVs + fsGroup, no root chowns; harden paid smoke flows

[bussyjd]

Summary

Replaces the stack's volume-permission patchwork (provisioner chown -R 1000:1000, root chown init containers, the internal/k8sperm helper, per-workload UID drift) with kubelet-owned permissions: local-path PVs are now provisioned as local-type volumes so the kubelet actually applies pod fsGroup at mount, and every PVC consumer runs non-root with an explicit, documented identity. Also hardens the paid smoke flows (flow-06/08/11/13/14 + libs) and moves the QA runbook to CLI-first surfaces.

Why permissions were "intermittent" for months: Docker Desktop's VirtioFS mounts the data dir with fakeowner — it fakes ownership and skips POSIX permission checks, so every UID mismatch was invisible on macOS k3d. On any real Linux filesystem (native k3s, CI, remote QA boxes) the same workloads hit Permission denied. Green locally, red everywhere that matters. Verified empirically on a live node (see plans/volume-permission-hardening.md).

Ownership model: before → after

flowchart TB
    subgraph BEFORE["Before — chown patchwork (masked by VirtioFS fakeowner on macOS)"]
        P1["local-path provisioner<br/>chown -R 1000:1000 (hardcoded)"] --> V1["hostPath-typed PV<br/>(kubelet IGNORES fsGroup)"]
        V1 --> H1["Hermes UID 10000<br/>root chown init re-owns /data<br/>EVERY start"]
        V1 --> A1["Spawned agents UID 10000<br/>NO chown init → stuck<br/>Provisioning on Linux"]
        V1 --> B1["x402-buyer UID 65532<br/>nothing → EACCES on<br/>consumed.json (Linux)"]
        S1["CLI sync chowns tree to<br/>HOST uid (501 on macOS)"] -.->|flapping| V1
    end

    subgraph AFTER["After — kubelet-owned (this PR)"]
        P2["provisioner: mkdir -m 0770 only<br/>StorageClass defaultVolumeType: local"] --> V2["local-typed PV<br/>kubelet applies fsGroup at mount"]
        V2 --> H2["Hermes + spawned agents<br/>UID/GID 1000, fsGroup 1000,<br/>fsGroupChangePolicy: Always<br/>(host-seeded editable .hermes state)"]
        V2 --> B2["litellm pod fsGroup 65532<br/>buyer container UID 1000<br/>(legacy hostPath-PV state compat)"]
        N2["no root containers anywhere<br/>internal/k8sperm deleted"]
    end

    BEFORE ==> AFTER

Loading

Fresh agent-home lifecycle under the new model

sequenceDiagram
    participant CLI as obol CLI (host)
    participant Prov as local-path provisioner
    participant Kub as kubelet
    participant Pod as Hermes pod (UID 1000)

    CLI->>CLI: SeedHostFiles — SOUL.md, skills,<br/>.no-bundled-skills into $DATA_DIR
    Pod->>Prov: PVC bound (WaitForFirstConsumer)
    Prov->>Prov: mkdir -m 0770 (no chown)
    Note over Prov: PV typed "local", not hostPath
    Pod->>Kub: mount
    Kub->>Kub: fsGroup 1000 ownership walk<br/>(fsGroupChangePolicy: Always —<br/>host re-seeds between mounts)
    Kub->>Pod: volume writable for 1000:1000
    Pod->>Pod: non-root config-seed / profile-seed inits<br/>(replace the root chown init)
    Note over CLI,Pod: config change → checksum/hermes-config<br/>annotation rolls the pod

Loading

⚠️ Breaking: clusters created on ≤ v0.10.0-rc12

PV specs are immutable — existing clusters keep hostPath-typed PVs where the kubelet skips fsGroup, and their Hermes data is owned 10000:10000 by the removed root init. An in-place obol stack up with this code breaks Hermes and the buyer on those clusters.

flowchart LR
    Q{"Cluster created<br/>on ≤ rc12?"} -->|no| OK["nothing to do —<br/>fresh local-type PVs"]
    Q -->|yes| R["recreate (supported path):<br/>wallet backup → down → purge -f →<br/>init → up → wallet restore"]
    Q -->|"yes, must keep cluster"| E["k3d escape hatch:<br/>docker exec node chown -R 1000:1000<br/>over the PV backing dirs + restart pods"]

Loading

Full instructions: plans/volume-permission-hardening.md § Upgrading from ≤ v0.10.0-rc12 and the new troubleshooting entry.

Changes

Permission model

• local-path.yaml: StorageClass defaultVolumeType: local; setup hook reduced to mkdir -m 0770.
• hermes.go + agent_render.go: UID/GID 10000 → 1000, fsGroup: 1000, fsGroupChangePolicy: Always, root chown inits removed, non-root config-seed/profile-seed inits, checksum/hermes-config annotation rolls pods on config change. Host-side ownership repair now targets the container UID (1000) on both backends.
• llm.yaml: litellm pod fsGroup 65532; buyer keeps container UID/GID 1000 — pre-v0.10.0 state PVs hold consumed.json written 0600 by UID 1000, a 65532 sidecar crashloops on load state and takes every paid/* route down.
• internal/k8sperm deleted; tests pin every contract (embed_local_path_test, embed_buyer_state_test, agent_render_test, hermes_test, stack_test).

Paid-route availability (cross-review fix)

• Reloader annotation scoped to litellm-config only. The controller rewrites the buyer ConfigMaps on every buy/top-up/auto-refill/tombstone; with strategy: Recreate + 1 replica the broad annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE) on every purchase event — inverting CLAUDE.md pitfall 7. The buyer hot-reloads via /admin/reload.

Smoke flows

• flow-06 JSON offer path, flow-08 buy retry + rollout settling, flow-11 register block (Ready poll aligned to flow-14's 300s for pitfall-13 RPC throttling), Bob wallet seeding, PurchaseRequest readiness gates, qwen36-deep targeting.
• lib.sh reset: stack down --yes — feat: agent purchase type within hermes #614's flow-16 leaves a live offer, and the non-TTY safety gate otherwise refuses graceful teardown on every release-smoke run.

Docs

• obol-stack-dev runbook → CLI-first QA; troubleshooting entry for the upgrade breakage; plans/volume-permission-hardening.md rewritten with the fakeowner root cause, empirical validation, upgrade section, and a multi-node roadmap (host-FS coupling analysis, options ladder, decisions that bind today).

Compatibility with #614

git merge-tree of this branch × #614 is clean, and the union was built and tested: Hermes v2026.6.5, the buyer 04bebbc pin, flow-16, and this PR's securityContext changes all survive. The stack down --yes and Reloader scoping above are the two semantic interactions the cross-review surfaced.

Post-merge follow-up (shared with #614): the embedded serviceoffer-controller pin predates this PR — released clusters render sub-agents with the old UID-10000 model until the controller is rebuilt from the merge commit and repinned (tracked in embed_image_pin_test.go).

Validation

• go build ./..., full go test ./... green on the branch and on the simulated union with feat: agent purchase type within hermes #614.
• bash -n over all changed flows; full release smoke with OBOL_LLM_MODEL=qwen36-deep including USDC, live OBOL, and fork OBOL paid flows (fix: harden PVC permissions and paid smoke flows #615 run).
• Permission model proven on a live node via docker exec -u matrix (UID 10000/1000/65532 × chown states) — table in the plan doc.

History

This branch first carried a ConfigMap-delivered config+skills experiment (the original PR title); #615 — merged into this branch — reverted that in favor of host-seeded files + kubelet-owned permissions after the editability tradeoff proved wrong for operator workflows. The net diff contains no ConfigMap delivery; the experiment's useful remnants (checksum-roll annotations, non-root seed inits) survive. The ConfigMap direction returns as option 3 of the multi-node roadmap when single-node stops being an invariant.

🤖 Generated with Claude Code

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.