Repository navigation
Phase 4: rclone-compatible HEAD + file metadata headers - #506
Merged
neuromechanist merged 2 commits intoMay 15, 2026
Merged
neuromechanist merged 2 commits into
neuromechanist merged 2 commits into
Conversation
Closes #498, part of epic #449. - Explicit HEAD handler on /<id>/<version>/<bids-path>. Returns 200 with Content-Length, Last-Modified (from manifest.created), ETag (from manifest checksum, quoted per RFC 7232), and Cache-Control. Empty body. Hono v4 would otherwise auto-derive HEAD from the GET handler and return a 302, which rclone's HTTP backend does not follow on HEAD. - HEAD on a directory: 200 with text/html content-type, empty body. - HEAD on missing path: bare 404, no tombstone walk. Rclone fans out HEAD across every missing file; a 10-version walk per HEAD would amplify a divergent-sync into N*10 S3 fetches. - GET 302 on a file also gains Content-Length, Last-Modified, ETag -- for clients that skip HEAD (rclone --http-no-head, custom downloaders). - toHttpDate helper converts ISO 8601 to RFC 1123 HTTP-date. - Verified the existing HTML index format is parseable by rclone's HTTP backend: an rclone-parser-style regex test pins the contract. Phase 3 chrome (version picker, removed-since footer, all-versions link) uses absolute hrefs that rclone correctly filters out. - 97 unit tests pass (88 from Phase 1-3 + 9 new for toHttpDate and rclone parser compat). New E2E tests: HEAD on file/directory/missing, GET 302 metadata headers. - Docs: 'Sync with rclone' section in downloading.md, HEAD endpoint documented in data-api.md with response-code table update.
From the 3-agent /review-pr pass on PR #506: - CRITICAL (code-reviewer): dataRoutes.on('HEAD', ...) is dead code in Hono v4. Hono's #dispatch intercepts HEAD unconditionally before router.match -- it re-runs the GET handler with the original Request (method still 'HEAD'), so the isHead branch in fileOrIndexHandler fires correctly regardless. Removed the dead registration and rewrote the comment to accurately describe the dispatch path. - HIGH (code-reviewer): Content-Length on the GET 302 violates RFC 9110 §8.6 -- the field describes the (empty) message body, not the redirect target. Some intermediaries can mis-frame a long Content-Length 302 as a hung response. Split fileResponseHeaders into a (withContentLength: boolean) variant: HEAD keeps it (standard HTTP semantics), GET 302 drops it. Last-Modified and ETag stay on the 302 (both are valid on redirects per RFC 9110 §8.8). Docs and E2E test updated. - MEDIUM (silent-failure-hunter): toHttpDate silently passed malformed input through. A corrupt manifest.created would never surface in wrangler tail. Added console.warn so operators can diagnose. Test updated to assert the warning fires and includes the bad value. - Test gap (test-analyst): the rclone-parser test was effectively a good simulation but the comment about double-encoding was misleading -- left as-is, the existing tests are sufficient. 97 unit tests pass, all earlier tests preserved.
Contributor
Author
Review pass (3 sonnet agents in parallel)Findings addressed in 6870a8a:
Findings dismissed:
Unit test count: 88 (Phase 1-3) -> 97 (+9 Phase 4). All passing. Ready to merge. |
neuromechanist
merged commit May 15, 2026
e003a2b
into
feature/issue-449-epic-data-https
1 check passed
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR506Version: 🤖 Updated on each push. |
This was referenced May 15, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 of epic #449. Closes #498, part of epic #449.
Adds explicit HEAD support and surfaces `Content-Length` / `Last-Modified` / `ETag` on file responses so `rclone sync :http:data.nemar.org///` works correctly with delta detection.
Why
`rclone sync` with the HTTP backend uses a HEAD per file to read size + mtime + ETag, then transfers only changed bytes. Before this PR, HEAD against the Worker was auto-derived from the GET handler by Hono v4 -- it 302d to a presigned S3 URL. rclone's HTTP backend does NOT follow redirects on HEAD by default, so every sync would re-download every file.
What ships
Verified rclone HTML compat
The existing Phase 3 renderer was checked against rclone's HTTP backend parser. Files have bare relative hrefs (`x.edf`), dirs have trailing-slash hrefs (`sub-01/`). Phase 3 chrome (version picker, removed-since footer, all-versions link) uses absolute `/` -prefixed hrefs which rclone filters out. A unit test with a regex modeled after rclone's parser pins this contract against future renderer changes.
One soft note: every directory listing's footer link `<a href="manifest.json">manifest.json` IS a relative href, so rclone treats it as a file in the directory. Documented in the user guide (`--exclude manifest.json` if you don't want it). It's a real 200 with the file index JSON.
Tests
Docs
Test plan
Out of scope