Skip to content

Phase 4: rclone-compatible HEAD + file metadata headers - #506

Merged
neuromechanist merged 2 commits into
feature/issue-449-epic-data-httpsfrom
feature/issue-498-phase4-rclone
May 15, 2026
Merged

neuromechanist merged 2 commits into
feature/issue-449-epic-data-httpsfrom
feature/issue-498-phase4-rclone

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Phase 4 of epic #449. Closes #498, part of epic #449.

Adds explicit HEAD support and surfaces `Content-Length` / `Last-Modified` / `ETag` on file responses so `rclone sync :http:data.nemar.org///` works correctly with delta detection.

Why

`rclone sync` with the HTTP backend uses a HEAD per file to read size + mtime + ETag, then transfers only changed bytes. Before this PR, HEAD against the Worker was auto-derived from the GET handler by Hono v4 -- it 302d to a presigned S3 URL. rclone's HTTP backend does NOT follow redirects on HEAD by default, so every sync would re-download every file.

What ships

  • Explicit HEAD handler on `///`:
    • File: `200` + `Content-Length`, `Last-Modified`, `ETag`, `Cache-Control: public, max-age=300`, empty body.
    • Directory: `200` + `Content-Type: text/html`, empty body.
    • Missing path: bare `404`. Tombstone walk skipped on HEAD -- rclone fans out HEAD across every file it doesn't have locally; a 10-version walk per missing-path HEAD would balloon a divergent-sync into N*10 S3 fetches.
  • GET 302 also carries `Content-Length`, `Last-Modified`, `ETag` -- for clients that skip HEAD (rclone with `--http-no-head`, custom downloaders). Cheap: all three come from the manifest entry already in hand.
  • `toHttpDate` pure helper converts ISO 8601 to RFC 1123 HTTP-date. Used for the `Last-Modified` header.
  • ETag is the manifest checksum verbatim (`"sha256:"` for git-annex files, `"git:"` for inline git content), quoted per RFC 7232. Content-addressed; stable across re-publications of identical content. Makes `rclone check --checksum` work too.

Verified rclone HTML compat

The existing Phase 3 renderer was checked against rclone's HTTP backend parser. Files have bare relative hrefs (`x.edf`), dirs have trailing-slash hrefs (`sub-01/`). Phase 3 chrome (version picker, removed-since footer, all-versions link) uses absolute `/` -prefixed hrefs which rclone filters out. A unit test with a regex modeled after rclone's parser pins this contract against future renderer changes.

One soft note: every directory listing's footer link `<a href="manifest.json">manifest.json` IS a relative href, so rclone treats it as a file in the directory. Documented in the user guide (`--exclude manifest.json` if you don't want it). It's a real 200 with the file index JSON.

Tests

  • 97 unit tests pass (88 Phase 1-3 + 9 new). New tests: `toHttpDate` ISO/invalid/HTTP-date-passthrough/sub-second, rclone-parser regex against the renderer with empty / picker / removed-since / nested / URL-encoded cases.
  • New E2E: HEAD on a real file (asserts `Content-Length` matches manifest, `Last-Modified` parses as RFC 1123, `ETag` matches `/^"(sha256|md5|sha1|git):/` and empty body); HEAD on a directory; HEAD on a missing path (empty body, no tombstone); GET 302 with metadata headers.

Docs

  • `docs/guides/downloading.md` -- new "Sync with rclone" section with first-time / re-sync / version-switch / checksum-mode / `lsl` recipes. Adds `rclone sync` to the parallel-downloader table.
  • `docs/reference/data-api.md` -- new `HEAD` endpoint subsection, response-code table updated.

Test plan

  • `bun test test/data-route.unit.test.ts` -- 97 pass, 0 fail
  • `bun x tsc --noEmit` from `backend/` -- clean
  • After this and epic merge: `rclone lsl :http:data.nemar.org/nm000103/v1.0.0/` against the deployed Worker -- expect a flat listing with sizes and dates
  • `rclone sync :http:data.nemar.org/nm000103/v1.0.0/ ./local/` twice -- expect first run transfers everything, second run reports 0 files transferred (delta detection working)

Out of scope

  • `If-Modified-Since` / `If-None-Match` -> `304` responses. rclone uses HEAD to compare; not needed.
  • WebDAV (`PROPFIND`) -- separate consumer if asked for.
  • `?format=json` on directory listings -- rclone HTTP backend only parses HTML.

Closes #498, part of epic #449.

- Explicit HEAD handler on /<id>/<version>/<bids-path>. Returns 200 with
  Content-Length, Last-Modified (from manifest.created), ETag (from
  manifest checksum, quoted per RFC 7232), and Cache-Control. Empty
  body. Hono v4 would otherwise auto-derive HEAD from the GET handler
  and return a 302, which rclone's HTTP backend does not follow on HEAD.
- HEAD on a directory: 200 with text/html content-type, empty body.
- HEAD on missing path: bare 404, no tombstone walk. Rclone fans out
  HEAD across every missing file; a 10-version walk per HEAD would
  amplify a divergent-sync into N*10 S3 fetches.
- GET 302 on a file also gains Content-Length, Last-Modified, ETag --
  for clients that skip HEAD (rclone --http-no-head, custom
  downloaders).
- toHttpDate helper converts ISO 8601 to RFC 1123 HTTP-date.
- Verified the existing HTML index format is parseable by rclone's
  HTTP backend: an rclone-parser-style regex test pins the contract.
  Phase 3 chrome (version picker, removed-since footer, all-versions
  link) uses absolute hrefs that rclone correctly filters out.
- 97 unit tests pass (88 from Phase 1-3 + 9 new for toHttpDate and
  rclone parser compat). New E2E tests: HEAD on file/directory/missing,
  GET 302 metadata headers.
- Docs: 'Sync with rclone' section in downloading.md, HEAD endpoint
  documented in data-api.md with response-code table update.
From the 3-agent /review-pr pass on PR #506:

- CRITICAL (code-reviewer): dataRoutes.on('HEAD', ...) is dead code in
  Hono v4. Hono's #dispatch intercepts HEAD unconditionally before
  router.match -- it re-runs the GET handler with the original Request
  (method still 'HEAD'), so the isHead branch in fileOrIndexHandler
  fires correctly regardless. Removed the dead registration and
  rewrote the comment to accurately describe the dispatch path.
- HIGH (code-reviewer): Content-Length on the GET 302 violates RFC
  9110 §8.6 -- the field describes the (empty) message body, not the
  redirect target. Some intermediaries can mis-frame a long
  Content-Length 302 as a hung response. Split fileResponseHeaders
  into a (withContentLength: boolean) variant: HEAD keeps it (standard
  HTTP semantics), GET 302 drops it. Last-Modified and ETag stay on
  the 302 (both are valid on redirects per RFC 9110 §8.8). Docs and
  E2E test updated.
- MEDIUM (silent-failure-hunter): toHttpDate silently passed malformed
  input through. A corrupt manifest.created would never surface in
  wrangler tail. Added console.warn so operators can diagnose. Test
  updated to assert the warning fires and includes the bad value.
- Test gap (test-analyst): the rclone-parser test was effectively a
  good simulation but the comment about double-encoding was misleading
  -- left as-is, the existing tests are sufficient.

97 unit tests pass, all earlier tests preserved.
@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review pass (3 sonnet agents in parallel)

Findings addressed in 6870a8a:

  • CRITICAL (code-reviewer): `dataRoutes.on("HEAD", ...)` was dead code in Hono v4 -- `#dispatch` intercepts HEAD unconditionally before `router.match` and re-runs the GET handler with the original Request (method still `HEAD`), so my `request.method === "HEAD"` branch fires correctly regardless. Removed the dead registration and rewrote the comment to accurately describe the dispatch path.
  • HIGH (code-reviewer): `Content-Length` on the GET 302 violates RFC 9110 §8.6 -- the field describes the (empty) message body, not the redirect target. Some intermediaries can mis-frame a long-`Content-Length` 302 as a hung response. Split `fileResponseHeaders` to omit `Content-Length` for the 302 while keeping it on HEAD (standard HTTP semantics). `Last-Modified` and `ETag` remain on the 302 (RFC 9110 §8.8 explicitly allows both on redirects). Docs and E2E test updated to match.
  • MEDIUM (silent-failure-hunter): `toHttpDate` silently passed malformed input through. A corrupt `manifest.created` would never surface in `wrangler tail`. Added `console.warn` so operators can diagnose; test asserts the warning fires and includes the bad value.

Findings dismissed:

  • CRLF injection via `file.checksum` / `manifest.created` -- false positive. Both are regex-constrained at parse time and Cloudflare's Response constructor throws on CRLF. (hunter tagged this themselves.)
  • `Last-Modified` semantic mismatch (manifest `created` vs file mtime) -- false positive for this use case. `ETag` is content-addressed (SHA-256 / git blob SHA), so rclone with `--checksum` uses it and is unaffected by mtime fudge.
  • "HEAD-skips-tombstone-walk not pinned at unit level" (test-analyst criticality 7) -- the route handler returns BEFORE the tombstone walk on HEAD; refactoring that out would also delete the `if (isHead)` branch, which would break the existing E2E test. The unit-level instrumentation would add fixture machinery for a path the existing test already covers behaviorally.
  • rclone-parser regex fidelity (test-analyst criticality 5) -- the simulation matches what rclone's HTTP backend actually filters (<a href> + skip-absolute + skip-parent). Double-encoding concern is moot given encodeURIComponent use in the renderer.
  • Content-Length NaN/Infinity guard (test-analyst criticality 3) -- JSON cannot represent NaN/Infinity. Won't happen.

Unit test count: 88 (Phase 1-3) -> 97 (+9 Phase 4). All passing.

Ready to merge.

@neuromechanist
neuromechanist merged commit e003a2b into feature/issue-449-epic-data-https May 15, 2026
1 check passed
@github-actions

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR506

Version: 0.8.15-PR506.6870a8a
Commit: 6870a8a


🤖 Updated on each push.

This branch was previously deployed

1 inactive deployment
npm — 6870a8a1 Deployed May 15, 2026 by neuromechanist via publish #916
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant