buzz-arcadia deploy assets: wrapper chart, ecr ci, ops runbooks - #1
Open
bse-ai wants to merge 10 commits into
Open
buzz-arcadia deploy assets: wrapper chart, ecr ci, ops runbooks#1bse-ai wants to merge 10 commits into
bse-ai wants to merge 10 commits into
Conversation
…red chart The Arcadia clusters have no internet egress, so `helm dependency build` can never resolve `oci://registry-1.docker.io/cloudpirates` — the OCI dependency is what blocks consuming the upstream chart as-is and is the whole reason this fork exists (thin fork, deploy assets only, zero diff on crates/, web/, desktop/). Both subcharts are eval-only anyway: `postgresql.enabled` and `redis.enabled` stay false on our clusters. Postgres is a Crossplane RDS claim and Redis is a single in-cluster Deployment, both supplied by the buzz-arcadia wrapper chart, which also hard-fails the render if either flag is ever flipped back on. Chart.lock only recorded the two removed dependencies, so it goes too. `helm dependency build deploy/charts/buzz` (the helm-chart workflow step) stays green on a chart with no dependencies, and all three upstream fixtures still render. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wraps the vendored upstream chart (file://../buzz) with everything an
Arcadia EKS deploy needs and upstream deliberately leaves to the
operator. The relay Deployment/Service/SA/PDB still come from the
subchart unchanged.
New templates:
externalsecret.yaml ESO -> ClusterSecretStore/ssm-clustersecretstore,
producing the one Secret named in
buzz.secrets.existingSecret. Mandatory under
Argo: the subchart's own Secret path uses
lookup + randAlphaNum and would regenerate the
relay identity on every sync.
database-url.yaml ESO Kubernetes-provider SecretStore + a
ServiceAccount/Role scoped to the single
Crossplane connection Secret. Crossplane emits
username/password/endpoint and never a URL, and
the endpoint does not exist before the first
sync, so DATABASE_URL cannot be an SSM param —
it is composed in the ExternalSecret target
template instead (same for REDIS_URL).
rds-claim.yaml Crossplane RDS claim, full required schema,
Postgres 17.4, non-burstable class per env
(m6g dev / r6g multi-AZ prd) because the
litellm-internal readiness RCA applies verbatim
to a relay whose /_readiness touches the DB.
redis.yaml Single in-cluster Redis, litellm-module shape,
ECR-mirrored Chainguard image. Accepted SPOF in
both envs: buzz takes one redis_url and has no
Sentinel support. prd ElastiCache is a tracked
follow-up, not a phase-0 blocker.
networkpolicy.yaml Ingress scoped to the environment's real VPC
CIDRs (10.54.x dev / 10.52.x prd, never
10.0.0.0/8); Redis reachable from relay pods only.
ingress.yaml Internal ALB, idle_timeout 3600 for long-lived
WebSockets, cert + hostname built from the
injected globals. Health checks target :8080
/_readiness, NOT "/" on :3000: the relay binds a
community from the Host header before the upgrade
and 404s an unmapped host, and an ALB health
check sends the target IP as Host.
Value nesting is a hard requirement — Helm passes only .Values.buzz to
the subchart, and Helm cannot copy a parent value into a subchart
subtree (parent values.yaml is plain YAML; subchart templates render
first). So the wrapper consumes the injected globals in the resources it
renders itself and *enforces* buzz.image.repository against
<account>.dkr.ecr.<region>.amazonaws.com/buzz/relay, failing the render
on a cross-account or docker.io reference instead of leaving an
ImagePullBackOff after sync. There is no image.registry key to set: the
vendored schema declares image.additionalProperties=false.
_validate.tpl encodes the rest of the deployment decisions as render-time
failures: explicit image tag (an empty tag silently falls back to
appVersion 0.1.0), existingSecret must match the ExternalSecret target,
ingress host must equal the relayUrl authority, no bundled subcharts, no
S3 (the startup conformance probe is env-gated off until media lands with
IRSA), no ServiceMonitor (no Prometheus Operator on these clusters), RDS
claim inputs present, NetworkPolicy CIDRs present and not a /8.
ci/dev-ai-values.yaml and ci/prd-ai-values.yaml are the rendering
fixtures and the copyable reference for each cluster's Argo CD
Application; they carry the nine injected globals so the chart renders
standalone, which is also how chart-testing lints it.
Verified: helm lint + helm template green for both fixtures; the
parent-level --set form renders byte-identically to the fixture form;
rendered output has no lookup/randAlphaNum, no cloudpirates reference,
no chart-managed Secret, no PVC and no BUZZ_S3_ENDPOINT; 12 negative
cases (schema violation, flat values, wrong-account image, /8 CIDR,
missing globals, ...) all fail the render with a named reason.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dev-ai and prd-ai EKS clusters have no internet egress and cannot pull ghcr.io, so upstream's public image is unusable there. This builds the same Dockerfile into 258174056699 and 572630832277 ECR on every main push, tagged main-<short10-sha> — the tag shape the ArgoCD Application carries. Notable choices: * amd64 only, no PR trigger (a PR build would OIDC into a production account from a potentially fork-authored ref). * Registry build cache in ECR with image-manifest=true,oci-mediatypes=true — ECR rejects buildkit's default cache media type, and a failed cache export fails the build. * Second AWS assume passes unset-current-credentials, or configure-aws- credentials role-chains from the dev-ai keys and fails against a role that does not trust them. * Read-only ECR preflight instead of ecr:CreateRepository — provisioning is SRE's (node N-1), so a missing repo fails fast with the fix, not 15 minutes later as an opaque push error. The deploy-dev job seds the image tag in cloud-config-infrastructures ai-dev, guarded so it stops rather than guesses: exactly one line must match tag: "main-<10 hex>", and the resulting diff must be exactly 1/1. A missing Application (node N7 not merged yet) is a documented no-op, not a red run. Its concurrency group is shared by name with aria-frontend's bridge bump job, but GitHub scopes concurrency to a repository, so that pairing is documentation only — the fetch/rebase/retry push loop is what actually makes two repos writing one branch safe. Also adds deploy/charts/buzz-arcadia/** to helm-chart.yml's pull_request paths; without it a PR touching only the wrapper chart gets no chart CI at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ARIA bridge has two egress dependencies outside its namespace: aria-bot's task API (cross-account, off-cluster, and nothing on dev-ai has ever reached it) and the in-cluster LiteLLM gateway. A VPC-peering or transit-gateway gap on the first is multi-week SRE work; discovering it during the long-task node would stall that lane after every other piece has landed. So the probe runs in phase 0, before the bridge exists. In-pod curl is the default mode — the relay image already ships curl, so nothing new is needed — and --launch is opt-in precisely because it is a kubectl write. A failure here is information, not a bug: each target carries the note that says what it blocks and who owns the fix, and the transcript is meant to be archived on the ticket either way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
External Secrets resolves an ExternalSecret all-or-nothing, so one missing SSM parameter leaves buzz-secrets unwritten and every relay Pod crash-loops with nothing in any log naming the key that was absent. That makes "did we create them all" a machine check, not a checklist item. provision-secrets.sh --verify is the gate. It derives the relay key list from the chart's externalSecret.data rather than retyping it, so a chart change that outruns this script goes red on drift instead of passing and letting the deploy find out. It reads types only, never --with-decryption: pulling relay identities into a terminal to prove they exist is exposure with no benefit. --apply refuses to run outside the cluster's own account, never overwrites an existing parameter, and passes private keys as --value file:// so they stay out of the process table. Documents what is deliberately NOT in SSM and why each was wrong earlier: database-url (the RDS endpoint does not exist pre-sync and Crossplane emits components, not a URL), redis-url (composed in-chart), the S3 keys (unprovisionable and unnecessary — IRSA), litellm keys (runtime, per human), and the owner private key, which stays in a password manager because buzz-admin writes the roster straight to Postgres and so losing it costs invite links only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
smoke-e2e.sh checks readiness on every Pod, that the public host actually terminates at the relay through the internal ALB, that the roster contains the pilots and agents, and that an @mention comes back as a correctly threaded kind:9 reply. --team runs the cross-mention matrix: each agent must answer only its own mentions. It does not reimplement a Nostr client. Steps 4-6 shell out to the bridge's own smoke_mention.py in aria-frontend and read its --json result; the CLI and JSON contract is written down in SMOKE.md so the other repo has something to build against. If that client is absent the steps SKIP loudly — and skips are counted separately and called out, because a green run full of skips does not satisfy the phase-0 exit criteria. soak-metrics.sh gets the WebSocket-through-ALB evidence with no platform dependency. These clusters have no Prometheus Operator, so there is no ServiceMonitor path at all and real scraping is an SRE-ticketed Alloy change; putting that on the phase-0 critical path would gate the spike on another queue. It scrapes :9102/metrics through the API-server Pod proxy — a pure read, nothing needed inside the container — and its --summary classifies each drop in buzz_ws_connections_active as explained (a matching backpressure disconnect) or unexplained, which is the signature the soak is actually hunting. Every metric column was checked against the relay source. There is no reconnect counter; buzz_auth_attempts_total is documented as the proxy rather than reported as a literal count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every chat message is an event row with its body in plaintext, replicated to every channel member, full-text indexed and retained indefinitely. ARIA replies are ordinary kind:9 messages and get exactly that treatment, and ARIA skills reach claims-adjacent systems — so this is the phase-2 gate it is marked as. The FTS statement is the sharp part and is verified rather than assumed: fresh installs (which both clusters are) take the positive allowlist from migration 0008, kind IN (0, 9, 40002, 45001, 45003). kind:9 is in it and is not in the privacy-excluded set from 0005. ARIA replies are searchable. Search is scoped to the caller's accessible channels, so the index widens how easily a member finds something inside channels they are already in, not who can read what. Retention is stated honestly instead of being dressed up. The relay has no global event retention and no event reaper. The one knob that exists, ephemeralTtlOverride, is set to 7 days idle and is documented as three things it is not: not a cap (it overrides a shorter client request), not a purge (the reaper sets channels.archived_at; event rows survive and stay indexed), and not applicable to permanent channels — which is what the ARIA team channels are. The resulting decision, purge job versus accepted indefinite retention, is Security's and is written up as the open question that gates prd. Chart changes use the subchart's own values (ephemeralTtlOverride, requireMediaGetAuth, uploadRecords) rather than relay.extraEnv, so nothing depends on last-duplicate-wins env semantics; only BUZZ_AUDIT_ENABLED, which has no subchart value, goes through extraEnv. requireAuthToken is pinned explicitly because it is a security boundary and a silent upstream default flip would be invisible in a values diff. All of it renders identically in both environments — a privacy control that is weaker in dev is one that gets tested in the wrong configuration. The prd fixture deliberately does not override extraEnv, and says why: Helm replaces lists rather than merging them, so one override there would drop the startup-fatal BUZZ_GIT_CONFORMANCE_PROBE=false along with these controls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bridge chart's ExternalSecret consumes `aria-db-password` (its Postgres role's password, composed into ARIA_DATABASE_URL) and `pilot-allowlist` (BUZZ_PILOT_ALLOWLIST, the phase-0 execution gate) under its defaults, and both CCI Applications keep those defaults. Neither was in this inventory, so `--verify dev-ai` — the gate that must be green before the Application merges — passed while two required parameters did not exist. ESO then resolves buzz-bridge-secrets all-or-nothing, fails on the gap, and the bridge pod never starts with nothing in any log naming the key: exactly the failure this node exists to prevent. Adds both to BRIDGE_PARAMS and to PREFLIGHT.md, generates aria-db-password in --apply (nothing else mints it — Crossplane's connection Secret holds the RDS master credentials, which the bridge Deployment must never see), and documents pilot-allowlist's format. The drift gate only compared the relay tier against the relay chart, which is why the gap could open at all. It now also checks the bridge tier against aria-frontend/deploy/k8s/aria-buzz-bridge/values.yaml when a sibling checkout is present (or ARIA_FRONTEND_REPO points at one), requiring the inventory to cover every key that chart resolves — including the two the template hardcodes rather than reading from values, which is how they were missed. Extra agent-key-<slug> params warn instead of failing: one environment can host several agent releases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Author
|
Code review — verdict: ship (once the RDS connection-secret key contract is confirmed). ✅ Ran Pre-merge verification (external contract, not verifiable in this repo): confirm the Crossplane |
chart-testing's yamllint (comments.min-spaces-from-content: 2) rejected the single space before the argocdNamespace inline comment. Two spaces now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Resolve mesh-llm rev" step read the root Cargo.lock, but the build step immediately after runs `cargo fetch --manifest-path desktop/src-tauri/Cargo.toml`. desktop/src-tauri is a separate cargo workspace with its own lockfile, and the two pin different mesh-llm revs: root Cargo.lock -> 43103c5 (tag v0.73.1) desktop/src-tauri/Cargo.lock -> c441ea7 (rev pin) cargo therefore checked out c441ea7 while the lookup searched git/checkouts for a directory named 43103c5, which never exists, and the job failed with "mesh-llm checkout for 43103c5 not found after cargo fetch". The macOS desktop build could not pass on any commit that triggered it. Read the lockfile that cargo actually resolves. This also corrects the mesh-llama cache key, which was keyed on a rev unrelated to the artifacts being cached. Same defect and same fix in release.yml and signed-macos-canary.yml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B2W1JSegBJQsYyQR5j2N9y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thin-fork deploy assets for hosting buzz on the Arcadia AI clusters (spec D1 — zero application-code diff; only
deploy/**and.github/**):deploy/charts/buzz-arcadia/— wrapper chart over the vendored relay chart: re-maps the global-patch values into thebuzz:subtree, composes the ECR image ref, ExternalSecret to SSM, Crossplane RDS claim, in-cluster Redis, NetworkPolicy, internal-ALB ingress with a 3600s idle timeout; the two cloudpirates OCI deps removed..github/workflows— relay ECR build (dual-account) + dev-ai tag-bump.POC work — no change-control ticket.
🤖 Generated with Claude Code