[jayleaton]

The template had no way to keep files: no bucket client, and nothing safe for taking an image from a user or fetching one from a URL they name. This adds optional S3-compatible object storage (Railway buckets, R2, S3, MinIO) and safe image ingestion. It is off unless configured. Source: jayleaton/auto-visuals#9 (storage, SSRF-safe fetch, pure-Go imaging) and #26 (storage stays optional; partial config is refused). Only the generic parts are ported.

Summary

POST /api/core/v1/images            raw body ─┐
POST /api/core/v1/images/import     {"url"}  ─┤ api.Images (mounted only when STORAGE_S3_BUCKET is set)
GET|DELETE /api/core/v1/images/{id}           │
                                              ▼
services.ImageService ── rules (tested): sniff magic bytes, 10 MB, 50 MP, per-user keys
  │                       users/<uid>/images/<random>.webp, PublicAddr + FetchURLAllowed
  ├─ ObjectStore  ──▶ server adapter ──▶ packages/storage            S3 + SigV4, no SDK; boot HEAD probe
  ├─ ImageFetcher ──▶ server adapter ──▶ packages/storage/safefetch  AllowAddr on every dialled IP (DNS + Control),
  │                                                                  CheckURL on every redirect, ≤3 hops, 10 MB, 20 s, no proxy
  └─ ImageCodec   ──▶ server adapter ──▶ packages/storage/imaging    JPEG/PNG/GIF/WebP/AVIF → upright, ≤2048 px (banded resize), metadata gone, WebP

The routes register through the central access gate (#124): import and delete are Endpoints (access.SignedIn); the raw upload and the streamed download use two small route.go variants, RouteLimit (10 MB body cap) and RouteFile (file instead of the envelope). They share one admit helper with Route, so 401/403/500 behave identically, and Admit stays in route.go.

• Config: STORAGE_S3_ENDPOINT, _BUCKET, _ACCESS_KEY_ID and _SECRET_ACCESS_KEY are live secrets (secrets.Secret, validated as all-or-none in validateLive, accessor Secrets.Storage(), revealed only in cmd/server). STORAGE_S3_REGION and _PATH_STYLE are plain env. In this PR the server reads them at boot; live rotation is the next PR in the stack.
• Layers: depguard keeps packages/storage out of services and transports, and keeps it from importing apps, database, auth or secrets. make lint-layering has four new probes.
• Deps: gen2brain/webp v0.6.4 (libwebp transpiled to Go; decode and encode) and gen2brain/avif v0.6.0 (libavif/dav1d in WASM under wazero). Both are current, released in July 2026. golang.org/x/image is no longer needed: the resize is our own (below), and the copy uses the standard library's image/draw. They are the only pure-Go WebP encoder and AVIF decoder; the alternative is libvips over cgo, which rules out the static binary.
• Build tag: without -tags nodynamic these codecs link purego, and a CGO_ENABLED=0 binary comes out dynamically linked, which crash-loops on distroless/static. make build and the Dockerfile pass the tag, and the Docker build fails (go version -m check) if it is dropped. CI's own go build/go test work without it.
• Banded resize (memory). x/image's CatmullRom.Scale holds the whole horizontally scaled image: target width × source height × 32 B = 198 MB for a 12 MP photo, 396 MB for 48 MP. A server held about 540 MB after a few uploads, too much for Railway's 512 MB. imaging.resize does the same separable Catmull-Rom (weights from the same newDistrib algorithm, the same 16-bit premultiplied arithmetic, the same ftou rounding) one band of output rows at a time. Each band scales only the source rows it reads, with the kernel-support rows it shares with its neighbours, and scratch is capped at 8 MiB. The output is identical byte for byte, so bands leave no seams. No lasting test was added: by construction there is no seam risk, and imaging sits outside the test boundaries (services see the codec only as a mocked port); the comparison below proves it.
• Code rules (JJ): none of the new code uses single-letter names, and every literal that carries meaning is a named, commented const (maxErrorBody, orientRotate90, imageIDBytes, bandBudget, …). Checked with mnd + varnamelen (min length 2, receivers and params included) on the new files: 0 findings. Tests use test *testing.T, as main now does. After that pass the outputs were byte-identical (same sizes, same 39.54 dB orientation PSNR, same SSRF refusals).
• Docs: docs/storage.md (turn on, use, safety, delete), one README "Learn more" line, the names in docs/secrets.md and .env.example, and one line in AGENTS "Where code lives".

Evidence

Real binary (make build) against a throwaway Postgres and a local S3 server (RustFS: minio/minio is no longer pullable from Docker Hub, and quay returned 401). Both were on free ports and removed afterwards.

• Static binary. Before: a probe build of the codecs without the tag gave dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2. After: bin/server: ELF 64-bit LSB executable, x86-64, … statically linked, with build info -tags=nodynamic CGO_ENABLED=0. Size (stripped, linux/amd64, decimal MB), measured on drop-one builds of the real server: 24.5 MB vs 16.9 MB on main e80a772, +7.6 MB: AVIF + wazero 4.13 MB, WebP 3.08 MB, the rest of imaging 0.37 MB, S3 client + safefetch + service/API under 0.1 MB. Dropping x/image saved 0.15 MB. README's Performance table now says 24.5 MB, notes in one sentence that 7.6 MB of it is the image codecs, and gives the image as about 11 MB compressed (0.86 MB distroless base from its manifest + the binary, which gzips to 9.9 MB; on main it gzips to 6.0 MB, so the old "about 5 MB" was already stale).

• Boot. Before: no storage. After:

  • storage: bucket reachable storage=endpoint:fp:957fb5c0,bucket:fp:ff1bc3c0,key:fp:c6d09e1a,secret:fp:b22e42ac.
  • Wrong secret: serve failed … storage: access denied: head 403 Forbidden, exit 1.
  • Missing bucket: … access denied: no such bucket.
  • Bad endpoint: … unavailable: head: dns lookup failed. No values are logged, fingerprints only.
  • Partial set: invalid config: STORAGE_S3_ACCESS_KEY_ID is required when any STORAGE_S3_* secret is set.
  • Storage off: storage: off …, so POST /images is a 404 while GET /users/me is a 200.

• Uploads (POST /api/core/v1/images):

  input	result
  exif-orient-1.jpg 320×240	webp 320×240
  exif-orient-6.jpg (same pixels, tag 6)	webp 240×320; vs orient-1 output rotated 90° CW: PSNR 39.5 dB (−90°: 8.2 dB); no EXIF in output
  4032×3024 JPEG	webp 2048×1536, 81 766 B
  2000×1500 RGBA PNG	WebP image, with alpha, 2000x1500
  sample.avif	webp 800×600
  animated.webp	first frame 400×300
  20000×20000 PNG bomb	Image has too many pixels
  truncated JPEG / text / HEIC	Image could not be read / Unsupported image type… / Unsupported image type…

  The bucket held only users/<uid>/images/<32 hex>.webp. GET serves Content-Type: image/webp and Content-Security-Policy: default-src 'none'; sandbox. Another user's id, ..%2F.., uppercase ids and no session get 404/401. Delete, then GET, is a 404. Browser headers get the bare 404.

• Banded resize vs today (x/image), same decoded images (throwaway program):

  image	bytes that differ	max diff	band edges checked	resize-only peak RSS today → banded	time today → banded
  4032×3024 JPEG (YCbCr) → 2048×1536	0 of 12 582 912	0	24, all 0	228 → 43 MB	604 → 502 ms
  8064×6048 JPEG (48 MP)	0	0	52, all 0	476 → 97 MB (73 MB is the decoded JPEG)	1.75 → 1.69 s
  4000×3000 RGBA PNG (NRGBA)	0	0	24, all 0	256 → 72 MB	543–913 → 593–837 ms (noisy)

  On the real server (fresh process per scenario, after sign-up ≈ 148 MB, which includes argon2): the stored WebP is byte-identical (same SHA-256, 81 766 B).

  scenario	peak RSS today → banded	time today → banded (median of 3, old/new alternated)
  12 MP uploads in a row	564 MB (settles at 539) → 177 MB (settles at 138)	2 669 → 2 578 ms
  one 48 MP upload	590 → 219 MB	5 308 → 4 847 ms
  two concurrent 12 MP	532 → 267 MB	2 816 → 2 866 ms wall

  The comment on imageConcurrency and docs/storage.md give this per-upload memory: about 30 MB above idle for 12 MP, 70 MB for 48 MP, 120 MB for two 12 MP at once.

• Codec idle cost (measured for JJ before this change): about +1.7 MB RSS; startup within noise; browser bytes identical. Nothing runs at boot: AVIF compiles its wasm on the first AVIF image (1.4 s, 6.5 MB, once).

• Import / SSRF (POST /api/core/v1/images/import, live):

  • https://www.gstatic.com/webp/gallery/1.webp → stored 550×368. A redirect via httpbin to a public image → stored.
  • Refused (This URL cannot be fetched): 127.0.0.1, localhost, [::1], [::ffff:127.0.0.1], 169.254.169.254, [fd00:ec2::254], 10.0.0.1, 100.100.100.200, 2130706433, 0x7f.1, 127.1, the Postgres and S3 ports, user:pw@, and file: (400 from the validator).
  • Dial-time only: https://localtest.me/ (resolves to 127.0.0.1; it passes the URL rule, so only the dial check can refuse it) is refused. A redirect hop to https://localtest.me/a.png or https://169.254.169.254/… is refused.
  • DNS rebinding, with the same name 7f000001.08080808.rbndr.us four times: the first answer was 8.8.8.8 (dialled, 10 s dial timeout → could not be fetched); the next three were 127.0.0.1 → refused at dial.
  • httpbin.org/redirect/5 → refused (over 3 hops). A 100 MB file → Image is larger than 10 MB in 1.2 s. httpbin.org/html → Unsupported image type.

• Tests (boundaries only):

  • services: TestPublicAddr (56 addresses, including NAT64/6to4/Teredo/IPv4-mapped/CGNAT/metadata), TestFetchURLAllowed (26 URLs), sniffing for every format plus HEIC/SVG/HTML/MP4, limits, port error mapping, the fetch deadline, per-user keys and id tricks. 100.0% coverage.
  • api: the four cases per route (through apitest.Gate()), the upload cap, the import validator and the sandboxed file response. 97.5% coverage.

• Checks: rebased on main 24be189 (merged beside the new Google/Apple sign-in: both features' routes, config, live secrets and server wiring are kept); before that on e80a772 (no migrations in this PR, so nothing to renumber); make check-fast ok (full run, because go.mod changed) and make lint-layering ok, including main's new Admit probes. After the rebase, the gate-registered routes on the real binary: upload 200, signed out 401, another user 404, get with the sandbox CSP, localtest.me refused, public import 200, file:// 400, delete then get 404, browser headers 404.

• CI: green on c29526e, but only on the second attempt. The first failed the test-time budget in packages/auth (TestBetterAuthScryptHash 10.34 s, budget 10 s; on the re-run 5.97 s). The cause is CI contention, not this code: scripts/ci-check.sh runs lint and the race tests in parallel, and the PR build cache comes from main, which has never built gen2brain, so its 2.7 MB transpiled libwebp.go compiles cold during the tests (the same test took 3.08 s on another branch). After merge, main's cache includes it. If it recurs, the CI owner could give that test more headroom or compile before testing.

Not covered by a lasting test: packages/storage/** (the S3 client, safefetch, imaging). Tests are not allowed there, so they were proven by the run above.

Merge Danger

Door: two-way. No migrations and no stored state outside the bucket; storage is off by default, and a revert removes the routes.

Blast Radius: build. Every server binary gains 7.6 MB and needs -tags nodynamic to stay static (Makefile and Dockerfile updated, guarded). A deploy pipeline that runs its own go build for distroless images must add the tag. Known follow-up: deleting an account leaves that user's images in the bucket (users/<id>/images/). It is documented in docs/storage.md. The fix is a DeletePrefix (ListObjectsV2 + delete) on the ObjectStore port, called from UserService.DeleteAccount, with service tests. Deploy follow-up: neither apps/web (Dockerfile) nor deploy/platform.yaml sets GOMEMLIMIT, and the web service has memory_mb: 512. Setting GOMEMLIMIT to about 80% of the service memory lets Go hand freed upload memory back sooner (docs/storage.md says so). That belongs to the deploy owner; this PR does not touch deploy/.

Stack: this is the bottom layer; key rotation follows on top.

🤖 Generated with Claude Code