[jayleaton]

Sits on #114. There, the storage bucket and keys are live secrets but are read only at boot, so rotating a key or moving buckets meant a restart. Now a change in the secrets provider takes effect live, and only after the new set has proven it works. Source: jayleaton/auto-visuals#41 (storage part only).

Summary

secrets manager poll ─▶ Secrets.OnStorage(fn)        once per distinct set of the 4 STORAGE_S3_* names
                          │
                          ▼  cmd/server hook
             same fingerprint? ─ yes ─▶ nothing
             all four removed? ─ yes ─▶ keep current client (warn)
             storage.Live.Swap(ctx, cfg)
               ├─ New(cfg) + Probe (signed HEAD on the bucket)
               │    fail ─▶ old client stays; manager logs "consumer rejected change" (fingerprints only), retries after each later fetch
               └─ ok   ─▶ atomic.Pointer swap; in-flight calls finish on the old one; its idle conns closed
                          log "storage: bucket/keys rotated from=<fp> to=<fp>"

Files: packages/storage/live.go (new, mirrors database.Pool.Swap), config.Secrets.OnStorage, the hook in cmd/server/storage.go, and the server.ObjectStore adapter now over *storage.Live. Docs: docs/storage.md "Rotate keys" (the zero-downtime key runbook) and the row in docs/secrets.md. If storage was off at boot and turns on upstream, that is logged: mounting the routes needs a restart.

Evidence

The real binary (make build, static), one process (PID 1258316) for the whole run, on SECRETS_PROVIDER=doppler with SECRETS_REFRESH_INTERVAL=10s. DOPPLER_API_URL pointed at a local stand-in that serves Doppler's download JSON from a file, so the real manager polled, validated and ran the hooks. Two local S3 servers (RustFS) had different root keys: S3 #2 answers S3 #1's keys with InvalidAccessKeyId. Everything was removed afterwards.

Before (#114): a new set was used only after a restart.
After:

stage (pushed upstream)	log	where the next upload landed
A: S3 #1 / app	storage: bucket reachable storage=endpoint:fp:e368ecdf,bucket:fp:95426130,…	S3 #1/app (9 objects)
B: wrong secret	ERROR secrets: consumer rejected change, keeping its previous client … err="storage: new bucket/keys …secret:fp:81a9f3c3 failed their probe, keeping …secret:fp:c2e30b05: storage: access denied: head 403 Forbidden"	still S3 #1/app (10)
C: S3 #2 endpoint + keys, bucket app2	storage: bucket/keys rotated from=endpoint:fp:e368ecdf,… to=endpoint:fp:21a3ccdc,bucket:fp:7174bae6,key:fp:75e6372e,secret:fp:a7ad4dc8	S3 #2/app2 (1); S3 #1 stays 10
D: bucket move app2 → app2-moved	rotated … to=…bucket:fp:154d36cd…	S3 #2/app2-moved (1)
E: all four removed	WARN storage: STORAGE_S3_* removed upstream; keeping the current bucket client until restart	S3 #2/app2-moved (2)

Same PID at the end, /healthz {"status":"ok"}. A grep of both server logs for every secret value, key id, bucket name and endpoint host:port: 0 hits.

Checks: make check-fast ok. After the naming pass (JJ code rules: no single-letter names, no magic numbers) and the rebase onto #114 at 0828bd2, the run was repeated on the rebuilt binary (PID 1789590): the wrong secret was rejected with the old client kept, then a new endpoint + keys + bucket swapped live (rotated from=endpoint:fp:f5533671,… to=endpoint:fp:6aefb589,…), the upload landed on S3 #2, and 0 values were in the log. No new tests: the swap lives in packages/storage, and cmd/config are outside the test boundaries (the same as database.Pool.Swap). It was proven by the run above.

Merge Danger

Door: two-way. No state; a revert puts back the read-at-boot behaviour.

Blast Radius: storage. This only matters when storage is on. A bad upstream value can no longer break uploads (it is probed first). A good one switches every instance at its next poll, so copy the objects before moving to a new bucket (documented).

Rebased onto #114 at c29526e (banded resize, the image routes on the access gate, main 24be189). newImages now takes the gate; the rotation code is unchanged. Re-checked live on the rebuilt binary (one process): a wrong secret pushed upstream was rejected with access denied: head 403 Forbidden and the old client kept; a new endpoint + keys + bucket rotated (storage: bucket/keys rotated from=endpoint:fp:d9be5b40,… to=endpoint:fp:71791828,…), the next upload landed on the second server, and the log carried 0 secret values. CI green on 71c2eac (second attempt; the first hit the same packages/auth scrypt test-time flake as #114).

🤖 Generated with Claude Code