Repository navigation
Phase 6: redirect zarr bytes for non-browser clients - #1202
Merged
neuromechanist merged 9 commits intoSep 2, 2026
Merged
neuromechanist merged 9 commits into
neuromechanist merged 9 commits into
Conversation
Non-browser GET requests to a zarr store object (no allowlisted Origin) now 302 to the public S3 object instead of proxying through the Worker, never touching D1, the edge cache, or an upstream fetch. index.json stays proxied/edge-cached/D1-gated, HEAD is never redirected, and the rate limiter exempts redirect candidates from the data-ip bucket via the same predicate serve() uses to route. Closes #1061. Part of epic #1181.
s3PublicUrl interpolated the decoded key unescaped, so a literal '#' truncated the redirect Location at the fragment, '?'/space broke the request line, and a non-Latin-1 character made workerd refuse to send the header at all (Bun tolerated it, so tests stayed green). It now shares canonicalCacheUrl's per-segment encodeURIComponent treatment via a new encodeS3KeyPath() helper, so the fetch target, the cache key, and the Location all agree. Exported s3PublicUrl for a direct unit test of its real (non-s3Base) branch. PR #1202 review item 1.
ZARR_OBJECT_PATH_RE's \/zarr\/?(.*)$ matched "bogus" as rest for /on000001/zarrbogus, exempting a path that never reaches serve()'s route from the rate limiter. Anchored on \/zarr(?:\/(.*))? like ZARR_PATH_RE, requiring end-of-string or a literal slash after zarr. isRedirectCandidate never validated the dataset id either, so zz000001 (wrong prefix), nm1234567 (too many digits), and nm999999 (over the 99999 cap) got redirected to a garbage S3 URL and wrote an invalid id into Analytics Engine, while the same request with a browser Origin correctly 404'd. The regex now captures the id and isValidDatasetId gates it; a malformed id falls through to the normal proxied 404 path instead of a separate D1-free branch. Also drops the stale "catalog.json will need the same exemption" comment: catalog.json is a separate route with no <id>/zarr/ segment and can never match this regex at all. PR #1202 review items 2, 3, 7 (comment only; item 7's tests follow).
"bytes=100-50" is syntactically a valid single range (parseCacheableRange only checks digit shape, not end >= start), so it computed -49 and that landed in the Analytics Engine data point. Real S3 would 416 an inverted range like any other out-of-range request, so there is never a real byte count for one; fall back to the same "unknown" 0 as every other unmeasurable case. PR #1202 review item 4.
Replaces the unconditional per-request console.info the zarr redirect exemption used to emit with an observe-only mode on rateLimiter itself (a new RateLimiterOptions.observeOnly). A redirect candidate now counts toward the SAME IP-keyed data-ip bucket a proxied request from that client would be enforced against, but never blocks -- so a later proxied request from a client that's been hammering the bucket via redirects still 429s correctly. Logs at most one console.warn per window, the first time the bucket crosses the point enforcement would have tripped, with a hash of the IP (not the raw address) rather than a log line per exempted request. Updates AGENTS.md's zarr paragraph to match, and corrects a related inaccuracy: catalog.json is proxied and edge-cached but has no D1 gate of its own (it lists only already-public datasets), unlike index.json. PR #1202 review item 5.
Route-level companion to the isRedirectCandidate unit tests added earlier: a bare GET /catalog.json still gets a 200 from S3 via serveCatalog, never a 302 -- this route has no <id>/zarr/ segment and never consults isRedirectCandidate at all. PR #1202 review item 7 (tests).
- /zarrproxy + allowed Origin + store object: proxied bytes with CORS (item 8) - HEAD tests assert the fake upstream actually received method HEAD, not just a matching path/range (item 9) - Private dataset redirect safety: follow the 302 Location for real and confirm S3 itself 403s an unauthenticated request under the private prefix (item 10) - 8 concurrent redirect-candidate GETs: all 302, zero upstream, zero D1 (item 13) - Re-verify range-key hit/miss, the full-object size cap, and the 206-without-Content-Range fail-closed path via a no-Origin index.json request, since those mechanics moved to allowed-Origin store objects earlier in this PR (item 12) Items 1-7 and 11 landed in earlier commits on this branch. PR #1202 review items 8, 9, 10, 12, 13.
neuromechanist
force-pushed
the
feature/issue-1061-phase6-redirect-contract
branch
from
September 2, 2026 18:23
52c2f89 to
8b35ec7
Compare
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1202Version: 🤖 Updated on each push. |
neuromechanist
merged commit Sep 2, 2026
04ec7fa
into
feature/issue-1181-epic-zarr-serving
12 checks passed
neuromechanist
deleted the
feature/issue-1061-phase6-redirect-contract
branch
September 2, 2026 18:35
This was referenced Sep 2, 2026
Closed
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Non-browser requests to zarr store objects now redirect straight to the
public S3 object instead of proxying through the Worker, so Cloudflare
never carries the bulk of the byte traffic while every request stays
counted.
index.jsonremains the fully-managed contract entry point.isRedirectCandidate(method, path, origin)predicate inbackend/src/routes/zarr-data.ts, used identically byserve()'s ownrouting fork and by the zarr sub-app's rate-limit middleware wrapper, so
routing and rate-limiting can never disagree about which requests reach
D1/the edge cache/S3 through the Worker.
the edge cache or performs an upstream fetch. The bucket's own
NotResourcedeny-list (backend/src/services/bucket-policy.ts) is thereal enforcement point for a private dataset on that path -- a redirect
that 403s at S3 leaks nothing the proxied 404 does not.
the data-ip bucket (logs a counter via
console.infoinstead ofenforcing), without touching the shared
rateLimiterused by the API app.recordAccessfires on the redirect path withdetail: "chunk-redirect"(the access-metrics blob slots are fixed, sothis reuses the existing
detailslot rather than adding a column) andbytescomputed from the client's Range header (bytesFromRangeHeader) --bounded and suffix ranges get an exact count, open-ended/missing/malformed
ranges fall back to 0.
Routing table
GET /<id>/zarr/index.jsonGET /<id>/zarr/<store>/...with an allowlistedOriginGET /<id>/zarr/<store>/...with noOrigin, or a non-allowlisted one302to the public S3 URL (Location,Cache-Control: public, max-age=3600,Vary: Origin; noContent-Length, no ACAO).HEADanythingOPTIONSBoth entry points (
zarr.nemar.orghost fork and the/zarrproxypathmount) are served by the same code, redirect included.
Test plan
cd backend && bun test-- 1410 pass, 0 fail (includes 41 new/updatedtests in
zarr-data-cache.test.tscovering: the redirect routing forkat every edge case, zero D1 reads / zero upstream requests on the
redirect path (including for a private dataset), the
/zarrproxymount, telemetry bytes for bounded/suffix/open-ended/missing/malformed
ranges, the
isRedirectCandidateandbytesFromRangeHeaderpurefunctions, and the rate-limiter exemption against a real (non-bypassed)
rateLimiterwith a real in-memorycaches.defaultstand-in)Origin (now redirect-eligible) to carry an allowlisted Origin, or
retargeted onto
index.jsonwhere the scenario specifically needed ablocked Origin to still reach the proxied/cached code path
bun test test/zarr-data-route.test.tsfrom the repo root -- 10 pass(pure
allowedOrigin/corsHeaders/cacheControlFortests, unchangedby this phase)
tsc --noEmit(root and backend) andbiome checkclean on bothchanged source files
moved onto the redirect path, telemetry
detail/bytes arithmeticbroken, rate-limit exemption removed) each reverted after confirming
the expected new tests -- and only those -- failed
Closes #1061.
Part of epic #1181.