feat: improve persistent Cloudflare tunnel setup and domain onboarding

[bussyjd]

Summary

What changed:

• Added guided persistent Cloudflare tunnel/domain onboarding: obol tunnel setup, obol domain search, obol domain check, and obol domain register.
• Expanded tunnel lifecycle/state semantics so persistent local-managed vs remote-managed tunnels are tracked/restored correctly across stack up / tunnel restart.
• Hardened cloudflared runtime/chart defaults and improved smoke-flow reliability on the maintainer macOS environment.

Why it matters:

• Makes the persistent tunnel path more production-friendly and easier to adopt for custom domains.
• Prevents stale local-managed resources from shadowing remote-managed tunnel state.
• Improves tunnel HA/health reporting and keeps release-smoke coverage green for the affected CLI/runtime paths.

Risk level: medium

Commit under test:

• b98cc83b820050624b6c974525fd94ee3ce0c915

Base branch:

• main

Scope

• Code
• Charts / manifests
• Flows / QA scripts
• Docs / skills
• Images / dependencies
• Other:

Validation

CI checks:

Check	Status	Link
Analyze (actions)	PASS	https://github.com/ObolNetwork/obol-stack/actions/runs/25429528556/job/74591812891
Analyze (go)	PASS	https://github.com/ObolNetwork/obol-stack/actions/runs/25429528556/job/74591812889
Analyze (javascript-typescript)	PASS	https://github.com/ObolNetwork/obol-stack/actions/runs/25429528556/job/74591812878
Analyze (python)	PASS	https://github.com/ObolNetwork/obol-stack/actions/runs/25429528556/job/74591813014
CodeQL	PASS	https://github.com/ObolNetwork/obol-stack/runs/74591937505

Unit tests:

go test ./...
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

Integration tests:

go vet ./cmd/obol ./internal/tunnel ./internal/stack ./internal/hermes
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

helm lint internal/embed/infrastructure/cloudflared
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

helm template internal/embed/infrastructure/cloudflared --set mode=auto
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

helm template internal/embed/infrastructure/cloudflared --set mode=remote
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

helm template internal/embed/infrastructure/cloudflared --set mode=local
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

bash -n flows/flow-02-stack-init-up.sh
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

bash -n flows/flow-04-agent.sh
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

bash -n flows/flow-11-dual-stack.sh
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

obol tunnel setup --help
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

obol domain register --help
PASS on b98cc83b820050624b6c974525fd94ee3ce0c915

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
flow-01-prerequisites	local host	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-02-stack-init-up	local k3d + Base Sepolia RPC checks	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-03-inference	local k3d + Ollama	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-04-agent	local k3d + Hermes	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-05-network	local k3d + eRPC	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-06-sell-setup	local k3d + x402 setup	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-07-sell-verify	local k3d + paid route verification	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-10-anvil-facilitator	Base Sepolia fork via Anvil	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-08-buy	local k3d + local facilitator path	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-09-lifecycle	local k3d + tunnel lifecycle	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded in full release-smoke report below
flow-11-dual-stack	Base Sepolia live	maintainer-macos-arm64	feat/tunnel-domain-setup	PASS	Embedded below in full release-smoke report and Live Chain Evidence

Release smoke:

Command:
TMP_DOCKER_CFG=$(mktemp -d)
printf '{}\n' > "$TMP_DOCKER_CFG/config.json"
DOCKER_CONFIG="$TMP_DOCKER_CFG" \
OBOL_REUSE_LOCAL_DEV_IMAGES=true \
OBOL_NONINTERACTIVE=true \
OBOL_DEVELOPMENT=true \
OBOL_CONFIG_DIR="$PWD/.workspace/config" \
OBOL_BIN_DIR="$PWD/.workspace/bin" \
OBOL_DATA_DIR="$PWD/.workspace/data" \
bash flows/release-smoke.sh

Result:
# Obol Stack Release Smoke Report
Date: 2026-05-06T10:21:47Z
Commit: b98cc83b820050624b6c974525fd94ee3ce0c915
Artifacts ID: release-smoke-20260506-182147

| Flow | Result | FAIL lines | Exit code |
| --- | --- | ---: | ---: |
| `flow-01-prerequisites` | PASS | 0 | 0 |
| `flow-02-stack-init-up` | PASS | 0 | 0 |
| `flow-03-inference` | PASS | 0 | 0 |
| `flow-04-agent` | PASS | 0 | 0 |
| `flow-05-network` | PASS | 0 | 0 |
| `flow-06-sell-setup` | PASS | 0 | 0 |
| `flow-07-sell-verify` | PASS | 0 | 0 |
| `flow-10-anvil-facilitator` | PASS | 0 | 0 |
| `flow-08-buy` | PASS | 0 | 0 |
| `flow-09-lifecycle` | PASS | 0 | 0 |
| `flow-11-dual-stack` | PASS | 0 | 0 |

Notes:
- The runner uses the real `obol` CLI and the flow scripts as black-box release checks.
- Any `FAIL:` line is release-gating, even when a child script exits zero.
- The live dual-stack receipt summary is embedded below in this PR body instead of referencing a local temp path.

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

• Base Sepolia (84532)

RPC/provider:

• https://sepolia.base.org
• Both Alice and Bob stacks also added Base Sepolia into in-cluster eRPC during the live flow.

Facilitator:

• https://x402.gcp.obol.tech

Contracts and tokens:

Name	Address	Version / notes
USDC	0x036CbD53842c5426634e7929541eC2318f3dCF7e	Base Sepolia USDC, 6 decimals
ERC-8004 registry	0x8004A818BFB912233c491871b3d84c89A494BD9e	Agent registration / metadata registry

Wallet roles:

Role	Address	Source
Alice / seller / register	0xC0De030F6C37f490594F93fB99e2756703c4297E	.env signer used by live flow
Bob / buyer / payer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	deterministic Bob wallet derived by flow-11-dual-stack.sh
Facilitator / receiver	N/A	facilitator is the HTTPS service above; on-chain settlement proof is the Bob-signer -> Alice USDC transfer

Balances:

Token	Address	Before	After	Expected delta	Actual delta
USDC	Alice 0xC0De030F6C37f490594F93fB99e2756703c4297E	8639000	8590000	-49000	-49000
USDC	Bob signer 0x1E4f1136ec1f3Faa164184c7cF8Ac0591339Abc9	0	49000	+49000	+49000

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	0x599c81220a08e179cc79b71f4b75917c2c6ac03b04d9107da7bcf43b1b9990f3	Alice	ERC-8004 registry	agent 5603 registered; route URL recorded	success
Metadata / service offer	0x45e08f2ef7468063df8b6cff81a6adac041748493ddd97cac2cf575a202803a1	Alice	ERC-8004 registry	x402 metadata written for agent 5603	success
Buyer funding	0xea6c42b82340b41896be9568dca9fe654b82b9842b9f38f9d57801a76a40260c	Alice	Bob signer (USDC transfer)	50000 micro-USDC	success
Approval / permit	N/A	N/A	N/A	no separate approval tx in this recorded live path	N/A
Purchase request	off-chain	Bob Hermes agent	Alice paid route	PurchaseRequest CR became Ready	ready
Settlement transfer	0x42419f2b880583ed690d75d368b4de15f1e19f4a9ec39e31fb423f043d3d321f	Bob signer (USDC authorization signer; tx sender 0xd744494e28b01073514ebc89987b305001ed257a)	Alice	transferWithAuthorization / 1000 micro-USDC	success

Runtime Evidence

QA environment:

Item	Value
OS / arch	macOS 26.3.1 arm64
Backend	Docker Desktop 28.5.1 + k3d v5.8.3 / k3s v1.33.6-k3s1
Tool versions	go1.25.5, kubectl v1.34.1, helm v4.0.4
QA agent/model	Bob Hermes agent successfully bought and used paid/qwen3.5:9b; platform default agent model remained gpt-oss:120b-cloud

Images:

Component	Image	Tag / digest	Source
cloudflared	cloudflare/cloudflared	2026.1.2	internal/embed/infrastructure/cloudflared/values.yaml
x402 verifier	ghcr.io/obolnetwork/x402-verifier	latest	internal/embed/infrastructure/base/templates/x402.yaml
Hermes agent	nousresearch/hermes-agent	latest	internal/hermes/hermes.go

Kubernetes / stack:

Item	Value
Stack IDs	single-stack release-smoke coverage plus dual-stack Alice/Bob live flow
Namespaces	llm, x402, erpc, monitoring, hermes-obol-agent
Pod readiness	flow-02 passed 19/19 platform checks; flow-11 Alice/Bob x402 pods and Bob Hermes agent all reached ready states
Cleanup result	PASS: tunnel stopped, Alice stack down, Bob stack down

Model and routing:

Item	Value
Agent/model used	Bob agent bought and invoked paid/qwen3.5:9b
LiteLLM route	http://litellm.llm.svc.cluster.local:4000/v1
Paid endpoint status	STATUS=200, TIME=13.1s, MODEL=paid/qwen3.5:9b
Auth token source	obol hermes token obol-agent in the buyer stack

Artifacts and logs:

Artifact	Location / link	Notes
Release-smoke report	Embedded in Validation > Release smoke above	full 11-flow matrix is written in this PR body
Flow-11 receipt summary	Embedded in Live Chain Evidence above	registration / metadata / funding / settlement hashes written here
Fresh rerun confirmation	#429 (comment)	notes that the older failing smoke notification was pre-fix and superseded

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	agent 5603 registered and ServiceOffer became Ready
Buyer discovery works	PASS	Bob agent discovered Alice via ERC-8004
Paid route works	PASS	Bob bought 5 auths and inference returned STATUS=200 with paid/qwen3.5:9b
Settlement visible on-chain	PASS	funding + settlement receipts captured; settlement tx 0x42419f2b880583ed690d75d368b4de15f1e19f4a9ec39e31fb423f043d3d321f

Review Notes

Known gaps:

• Full smoke evidence in this PR is from the maintainer macOS/arm64 environment only; I did not rerun the entire release-smoke suite on Linux in this session.
• The live validation intentionally did not execute a real Cloudflare Registrar purchase; domain purchase coverage here is command-surface / implementation coverage, not a paid registrar side-effect test.

Follow-ups:

• If desired, add a Linux release-smoke rerun before merge for cross-platform smoke evidence.
• If desired, perform a deliberate manual validation of the registrar purchase flow with an expendable domain budget / operator confirmation.

Reviewer focus:

• Tunnel state migration/restore and remote-vs-local management mode precedence.
• cloudflared chart/runtime changes (replicas, probes, PDB, management-mode wiring).
• macOS noninteractive DNS fail-open behavior vs unattended smoke expectations.
• flow-only reliability fixes staying scoped to the QA harness rather than widening product defaults.

[bussyjd]

Reran bash flows/release-smoke.sh on pushed HEAD b98cc83 in the maintainer macOS worktree environment.

Validation update:

• status: PASS
• report: .tmp/release-smoke-20260506-182147/RELEASE_REPORT.md
• live dual-stack receipts: .tmp/release-smoke-20260506-182147/flow-11-receipts/receipt-summary.json
• flow-11 settlement tx: 0x42419f2b880583ed690d75d368b4de15f1e19f4a9ec39e31fb423f043d3d321f

Note: the earlier failing notification (release-smoke-20260506-171059) was from a pre-fix run on base commit 69cd156; the current rerun on PR head passed.

[bussyjd]

Superseded by #452.

Closing this PR in favor of the integrated follow-up so review and validation stay consolidated in one place.

Reasoning:

• integration: validate tunnel onboarding with live OBOL faucet flow #452 is the active PR carrying the final integrated validation story.
• This branch overlaps heavily with the newer tunnel/domain work but also carries older unrelated branch history.
• Keeping feat: improve persistent Cloudflare tunnel setup and domain onboarding #429 open would split review across parallel PRs for substantially the same area.