release: v0.10.0-rc14 (consolidate #609/#610/#614 + review hardening)#616
Merged
Conversation
Ship the agent config.yaml (hermes-config) and the Obol skills tree (new hermes-skills ConfigMap, a deterministic gzipped tarball in binaryData) to the Hermes pod, and extract them into the PVC from the init-hermes-data init container running as the container UID (10000). The host no longer writes the data PVC, so there is no wrong-owned scaffolding for the root init-hermes-perms chown to repair, removing the host-write + chown-repair pattern behind the recurring PVC-ownership bugs. - embed.SkillsTarball(): deterministic (sorted paths, fixed mode, zero mtime), excludes __pycache__/.pyc, so the ConfigMap and the Deployment hash change only when the skills themselves change. - Delete the host-side syncRuntimeFiles / syncObolSkills / removeLegacyHeartbeat path; config + skills now arrive via read-only ConfigMap volumes mounted into the init container. Skills still land at /data/.hermes/obol-skills (external_dirs), preserving the existing read-only consumption semantics.
Module-path migration (not a version bump): the SDK renamed its path in the post-2026-03-31 releases. `github.com/coinbase/x402/go` → `github.com/x402-foundation/x402/go` @ v0.0.0-20260529172747; the foundation module is the maintained path (and where new schemes land). Pure path swap of the single `.../go/types` import across the tree; types API unchanged. Unlocks the upto + batch-settlement schemes already shipped under mechanisms/evm/. Folded in from #613 so this branch is self-contained on the latest module.
Runs a local x402-paid MCP (Model Context Protocol) server. The paid tool forwards the buyer's JSON arguments to a backend HTTP service and returns the response, so any real service can be resold to agents per call. Buyers settle in-band via the MCP request _meta["x402/payment"] field (per specs/transports-v2/mcp.md); verify -> execute -> settle runs inside the tool call, so a caller is never charged for a failed tool. Mirrors the canonical x402 paid-MCP example (a paid get_weather tool), generalising the upstream from a stub to a real backend. - internal/x402mcp: PaymentWrapper over the official modelcontextprotocol/go-sdk; proxyTool forwards args (+ optional backend auth header, set server-side, never sent to buyers) and returns the service response; free ping tool + /health. - cmd/obol/sell.go: `sell mcp` subcommand (--pay-to/--price/--chain/--tool-name/ --port/--upstream/--upstream-header/--description/--facilitator). Uses the x402-foundation/x402/go SDK.
…ardening, doc alignment Synthesis of a multi-agent review of this PR (every finding adversarially verified by two independent checkers; build+786 unit tests green throughout). Release-blocking: - flow-16 ended with undefined `summary` → exit 127 under set -euo pipefail, so every release-smoke run would report FAIL the moment this PR wires flow-16 into the unconditional flows array. Replaced with emit_metrics (the lib.sh idiom every other flow ends with). - Image pins at 04bebbc are this PR's own merge base: the shipped controller still renders sub-agents with Hermes v2026.5.28 (compiled-in default), the verifier lacks the tx-hash + maxTimeoutSeconds changes, the buyer lacks the settled-but-failed ConfirmSpend branch. Cannot be fixed pre-merge (no image containing this PR exists yet) — documented the mandatory post-merge rebuild+repin in embed_image_pin_test.go. Dev clusters mask this via OBOL_DEVELOPMENT rebuilds. Money-path hardening: - buyer proxy: fire OnConfirmSpendFailure (confirm_spend_failure_total) when the settled-but-failed branch fails to persist the consumed nonce — the alert purpose-built for "chain debited but consumed.json missed it"; the 2xx path already did this for the identical error. - buy.py: paid requests (pay, pay-agent) no longer follow redirects. urllib converts a redirected POST into a body-less GET and re-sends every header — including the signed X-PAYMENT voucher — to the Location target. - buy.py: mid-stream read failures in pay-agent print an actionable warning (payment already accepted; do not blindly retry) instead of a traceback. - buy.py: the settled-on-chain warning fires on any failed status (>=400) with a tx-hash settle header, matching the Go buyer condition; non-dict settle JSON no longer raises AttributeError. - buy.py: auth-TTL floor raised 300s → 600s to track the new default settle window (x402.DefaultMaxTimeoutSeconds). Consistency: - docs/observability.md + release-smoke-debugging.md: corrected the "5xx" claims to the actual conditions (buyer: >=400; verifier: non-200 /settle). - Replaced all 9 references to never-committed plans/rc13report.md (docs, code comments, test comments, buy.py runtime output) with pointers to docs/observability.md ("Verify settlement against the chain"). - renovate.json: Hermes pre-merge gate referenced nonexistent flow-17-agent-bundled-skills.sh → flow-16-sell-agent.sh §1.1.1/§1.3.1. - openapi.go: /api/services.json now advertises the clamped maxTimeoutSeconds the 402 wire actually enforces. - SKILL.md: corrected "nothing settles on-chain until that one request succeeds" (disproven by the rc13 incident this PR handles) and the stale 300s TTL-floor rationale. - discover.go: the pay-agent pointer also fires for storefront-base probes of agent-only sellers, not just exact /services/<name> URLs. - flow-16: pod gate on the Ready condition (not phase Running) so the exec-based bundled-skills assertions cannot false-pass mid-boot. Intentionally skipped: buy.py wire-echo fallback maxTimeoutSeconds=60 for sellers that omit the field (changing the echo risks requirement matching); PaymentEventUnsettled naming conflation (cosmetic); verifier-side metric for settle-error-with-tx (follow-up). Note the "default 600s" applies to static routes + the standalone gateway — ServiceOffer paths pin 300 via the CLI --max-timeout default and the CRD schema default.
Removed plans whose work landed and whose durable learnings already live in
the skill references:
- obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI)
- post-490-integration-20260513.md — integration landed in May
- release-smoke-hardening-20260513.md — session retro; learnings folded into
obol-stack-dev/references/release-smoke-debugging.md
- inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA
findings live in release-smoke-debugging.md §10 (c2dddc1)
Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending
walk-through), openapi-redoc-storefront.md (phase 2 deferred),
storefront-buy-inference-cta-handoff.md (active frontend handoff),
volume-permission-hardening.md (PR #615 owns its rewrite).
…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).
Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.
…-hardening fix: harden PVC permissions and paid smoke flows
…t fix The Shipped Changes bullet still described the pre-c667432 model (buyer inherits 65532). c667432 restored container-level UID/GID 1000 on the buyer for legacy hostPath-PV state compat; the bullet now matches llm.yaml and embed_buyer_state_test.go.
CLAUDE.md (+6 lines net): `sell mcp` in the command tree + one-line description; `pay`/`pay-agent` added to the buy.py command table; pitfall 7 notes the rc14 Reloader behavior (litellm-config restarts, buyer CMs stay hot-reload); new pitfalls 15 (settled-but-failed tx-hash semantics) and 16 (<= rc12 hostPath-PV upgrade breakage). plans/: dropped sell-agent-perf.md (max_turns 30 + reasoning_effort low shipped in agent_render.go; streaming shipped in #614); trimmed openapi-redoc-storefront.md to the deferred phase-2 design (phase 1 — Scalar UI + /openapi.json — is live); openapi-402-followups.md intro updated: its trigger (oisin/openapi landing) has fired, items are now actionable.
… private The <= rc12 upgrade guide ships in the v0.10.0 release notes (paste-ready block added to the #616 PR body); plans/ is heading to a private location. Every reference from public, durable surfaces is now self-contained: - CLAUDE.md pitfall 16 + troubleshooting.md: full fix inline, pointer to the release notes instead of plans/volume-permission-hardening.md - flows/lib.sh: stale-binary pitfall described inline (was citing the deleted v1337 buy report) - agent_contract_integration_test.go: drops the deleted sell-agent-perf.md citation (the reasoning survives in the comment) - cmd/obol/sell_agent.go: demo-label TODO self-contained (was citing openapi-402-followups.md) - docs/observability.md: deleted QA reports referenced via git history Remaining plans/ mentions are intentional: the CLAUDE.md conventions line, observability's explicit git-history pointer, and the SKILL.md path table.
Rebuild x402-verifier, serviceoffer-controller, and x402-buyer from release/v0.10.0-rc14 HEAD (docker-publish-x402 workflow_dispatch run 27265470511) and repin the embedded manifests to the new multi-arch digests. Resolves the FOLLOW-UP gate recorded in embed_image_pin_test.go: the prior 04bebbc pins were this train's merge base and contained none of the train's verifier/controller/buyer changes (settled-but-failed ConfirmSpend, settle tx hash on the error path, maxTimeoutSeconds on the 402 wire, UID-1000 sub-agent rendering, Hermes v2026.6.5 default).
OisinKyne
approved these changes
Jun 10, 2026
OisinKyne
pushed a commit
that referenced
this pull request
Jun 10, 2026
… private The <= rc12 upgrade guide ships in the v0.10.0 release notes (paste-ready block added to the #616 PR body); plans/ is heading to a private location. Every reference from public, durable surfaces is now self-contained: - CLAUDE.md pitfall 16 + troubleshooting.md: full fix inline, pointer to the release notes instead of plans/volume-permission-hardening.md - flows/lib.sh: stale-binary pitfall described inline (was citing the deleted v1337 buy report) - agent_contract_integration_test.go: drops the deleted sell-agent-perf.md citation (the reasoning survives in the comment) - cmd/obol/sell_agent.go: demo-label TODO self-contained (was citing openapi-402-followups.md) - docs/observability.md: deleted QA reports referenced via git history Remaining plans/ mentions are intentional: the CLAUDE.md conventions line, observability's explicit git-history pointer, and the SKILL.md path table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release train for v0.10.0-rc14, consolidating the three open PRs ≥ #609 (the rc12/#600 pattern). Each constituent was reviewed and cross-reviewed before assembly; the union builds clean (
go build ./...) and passes the full test suite (go test ./..., zero failures).flowchart LR M["main @ 04bebbc"] --> R["release/v0.10.0-rc14"] P614["#614 agent purchase type<br/>+ rc13 review fixes<br/>(settled-but-failed tx-hash,<br/>maxTimeoutSeconds, pay-agent,<br/>flow-16)"] --> R P610["#610 kubelet-owned PVC<br/>permissions + paid-smoke<br/>hardening (incl. #615)"] --> R P609["#609 obol sell mcp —<br/>paid MCP tool over<br/>x402 in-band _meta"] --> R R --> REPIN["post-merge gate:<br/>rebuild + repin verifier/<br/>controller/buyer images"] --> TAG["v0.10.0-rc14"]What each PR brings
agentpurchase type within hermes #614 —agentpurchase type within hermes (+ applied review synthesis550e820): streamingpay-agentbuy side, settlement tx-hash forensics on facilitator 5xx (verifierX-PAYMENT-RESPONSE+ buyerConfirmSpend+ buy.py warning),maxTimeoutSecondsplumbed offer→402 wire (default 600s, 24h cap), Hermesv2026.6.5(bundled-skills marker honored), flow-16 wired into release-smoke (and made functional:emit_metrics, Ready gate), paid-request redirect hardening in buy.py, image pins →04bebbc.c667432):defaultVolumeType: localso fsGroup actually applies, root chown inits andinternal/k8spermremoved, Hermes/agents at UID 1000, buyer keeps UID 1000 for legacy hostPath-PV state compat, Reloader scoped tolitellm-configonly, smoke flows hardened, multi-node roadmap recorded. Carries the breaking change (see below).obol sell mcp— paid MCP tool over x402 in-band _meta #609 —obol sell mcp: paid MCP tool surface over x402 in-band_meta(internal/x402mcp).Cross-PR verification
git merge-treeunions are textually clean; the assembled tree keeps every load-bearing invariant (spot-checked: Hermesv2026.6.5, buyer04bebbcdigest pin +runAsUser: 1000,reload: "litellm-config"only, flow-16emit_metrics,stack down --yes,x402mcppackage present).stack down --yes), and the Reloader annotation vs the controller's per-purchase buyer-CM writes.Clusters created on ≤ v0.10.0-rc12 keep hostPath-typed PVs; the kubelet cannot manage their ownership and the new non-root pods cannot read the legacy data. Slot the block below into the release template's NOTE/WARNING area:
Warning
Breaking — clusters created on v0.10.0-rc12 or earlier must be recreated.
This release moves volume permissions to kubelet-managed ownership (local-type PVs +
fsGroup). Pre-existing clusters hold hostPath-typed PVs the kubelet cannot manage, owned by the old UID. After upgrading the CLI:Symptoms if skipped: Hermes crashloops on
state.db/config.yaml; the x402-buyer exitsload state:at startup and everypaid/<model>route returns 503. k3d-only escape hatch (no recreation):docker exec <k3d-node> chown -R 1000:1000 /var/lib/rancher/k3s/storage/pvc-*then restart the pods.Release gate — status
Rebuild + repinDone: images built fromx402-verifier/serviceoffer-controller/x402-buyerfrom the release commitc19ffafviadocker-publish-x402workflow_dispatch (run 27265470511), repinned in2db429b, fullgo test ./...green. Tagv0.10.0-rc14cut on the repin commit and published as a pre-release: https://github.com/ObolNetwork/obol-stack/releases/tag/v0.10.0-rc14RELEASE_SMOKE_INCLUDE_OBOL=true), which now includes flow-16's agent/bundled-skills contract.Supersedes #614, #610, #609.