Repository navigation
Description
Motivation
Right now, accessing a NEMAR dataset's files requires understanding either git-annex (via DataLad/nemar-cli) or the content-addressed objects/ layout (an internal git-annex implementation detail). Both are walls for:
- Casual users who just want to download a few files in a browser
- Downstream tools/pipelines that want plain HTTP paths
- Machines/scripts that don't want to install nemar-cli or git-annex
- Users coming from OpenNeuro who expect a simple cloud-like file layout
Arno (co-owner) and Bruno raised this as a UX gap. The existing archives/v1.0.0.zip covers "give me everything in one zip" but not "let me browse / selectively download / incrementally sync".
Proposal
Expose a tool-agnostic, BIDS-shaped view of every public dataset over HTTPS at https://data.nemar.org/<datasetId>/<version>/..., served by a Cloudflare Workers route that resolves logical paths to backing-store objects via the existing version/vX.Y.Z.json manifest.
No S3 duplication. Nothing new is materialized in S3. The dataset/ namespace is virtual; the Workers proxy is the only "pointer layer".
https://data.nemar.org/nm000103/latest/ # HTML index of latest published version
https://data.nemar.org/nm000103/latest/sub-01/eeg/...edf # 302 -> presigned objects/SHA... URL
https://data.nemar.org/nm000103/v1.0.0/... # same, addressed by explicit version
https://data.nemar.org/nm000103/latest/manifest.json # [{path, size, sha256, url}, ...]
Underlying S3 layout is unchanged:
s3://nemar/nm00xxxx/
objects/ # git-annex content-addressed blobs (SHA-named); internal, no public listing
version/ # version manifests (vX.Y.Z.json); existing
archives/ # downloadable zip snapshots; existing (still the canonical 'whole dataset' path)
Worker behavior
The Workers proxy 302-redirects to presigned URLs for actual file bytes; it does NOT stream data through itself. Consequences:
- Parallel downloads scale to S3's request rate, not Worker CPU/bandwidth
- Worker only handles manifest lookup, HTML index rendering, and redirect: minimal compute cost
- Public datasets only. Private/unpublished datasets are NOT exposed via this HTTPS path; the existing
nemar-cli+ git-annex flow remains the only access route for those. This keeps Phase 1 simple (no auth gating, no token-in-URL/header concerns) and matches the OpenNeuro pattern.
Why direct objects/ access is discouraged
The objects/ prefix MUST NOT be the public access path, even though it physically holds the bytes:
- Content-addressed, not human-readable: files are named
SHA256-s12345--abcd1234.../.... You can't findsub-01_task-rest_eeg.edfwithout first reading the git-annex pointer files. - No directory structure: flat blob storage; no BIDS hierarchy.
- Internal git-annex implementation detail: git-annex's object key format is subject to change (
MD5E-,SHA256E-,SHA256-, etc.). Code that hardcodes object paths breaks across git-annex versions or backend changes. - No version semantics:
objects/mixes all versions of all files. You don't know which file belongs to v1.0.0 vs v1.1.0 without the manifest. - No deletion safety:
objects/is governed by git-annex GC rules. Direct consumers risk dangling references when old content is dropped. - Fragile against migration: if NEMAR ever reorganizes the bucket layout,
objects/paths break.data.nemar.org/...is the stable contract.
Therefore: objects/ will have public listing disabled at the bucket policy level. Direct GETs may still work for now (git-annex needs them) but are unsupported as a stable API.
Why aws s3 cp s3://... is intentionally not supported for dataset/
The natural follow-up question is "can I aws s3 cp s3://nemar/nm000103/dataset/latest/ ./?" — no, by design. Reasons:
- It would require materializing every published version into S3, ~doubling storage for the latest snapshot of all ~300 datasets (50 TB → +50 TB). NEMAR's S3 is hosted under the AWS Open Data Program; AWS sponsors storage and egress, but our quota/bookkeeping footprint is still ours to manage, and doubling it just to support
s3://muscle memory is the wrong tradeoff. - Materialization is fragile: partial states during publish, orphans on failed delete, drift between manifest and bytes. Single source of truth (the manifest +
objects/) is dramatically simpler. - Two equivalent UX paths already exist that don't require materialization:
archives/vX.Y.Z.zip(singleaws s3 cpof a zip) and HTTPS access viadata.nemar.org(works with every parallel downloader; see table below).
If a real audience surfaces later for s3://nemar/.../dataset/latest/ access specifically, materialization can be added without changing the public HTTPS contract.
Parallel download support (-j 16 style)
Parallelism is a client concern, not a protocol concern. All major parallel-download tools work natively over HTTPS because the Worker 302s to presigned URLs and parallel clients hit S3 directly:
| Tool | Command |
|---|---|
aria2c -j 16 |
curl -sL https://data.nemar.org/nm000103/latest/manifest.json | jq -r '.[].url' | aria2c -j 16 -i - |
rclone --transfers 16 |
rclone copy --transfers 16 --multi-thread-streams 4 :http:data.nemar.org/nm000103/latest/ ./ |
wget --mirror |
wget -r -np https://data.nemar.org/nm000103/latest/ |
curl + xargs -P 16 |
xargs -P 16 -n 1 curl -O < urls.txt |
| Whole-dataset zip | aws s3 cp s3://nemar/nm000103/archives/v1.0.0.zip ./ (unchanged, still works) |
The manifest endpoint makes any parallel downloader work in one line, and clients can verify SHA-256 checksums for free (content-addressing inherited from git-annex).
This epic does NOT introduce or modify any nemar-cli commands. The existing git-annex contract (nemar dataset clone / get / save / push / drop) is unchanged and untouched. A NEMAR-specific CLI wrapper for the HTTPS path can be considered as a separate issue later if users actually ask for one; until then, the existing third-party tools above cover the use case.
Storage backing
NEMAR's S3 bucket is hosted under the AWS Open Data Program; AWS sponsors storage and egress. This epic does NOT migrate the backing store and does NOT introduce a CDN layer. The Workers route reads existing S3 objects via presigned URLs; the only new infrastructure is the Worker route, the manifest endpoint, and the HTML index renderer. Keeping S3 as-is is required to maintain Open Data Program eligibility.
The cost rationale for not materializing dataset/latest/ is therefore operational, not financial: avoid drift, partial states, and doubling our Open Data quota footprint.
Phasing
Phase 1: Workers route + manifest + HTML index
- New Workers route:
GET /data/:datasetId/:version/*pathresolves via manifest, 302s to presigned S3 URL :versionacceptslatest,vX.Y.Z- HTML index style: Apache-style directory listing with minimal CSS (similar to AWS S3 static website listings; functional, no theme branding)
- Manifest JSON endpoint:
GET /data/:datasetId/:version/manifest.jsonwith presigned URLs and SHA-256s - Public subdomain:
data.nemar.org(CNAME to Workers) - Docs:
docs/guides/downloading.mdsection on HTTPS access + parallel-download tooling - Public datasets only; private/unpublished datasets return 404
Phase 2: Versioned access UX + tombstone indication
- Version picker in HTML index; sitemap-style listing of available versions at
/data/:datasetId/ /latestresolves to canonical version- Tombstone / file-removed-in-newer-version indication: when a path exists in an older version but not in the requested version (typically
latest), return a structured 404 that points the user to the last version where the file existed. Example:- Request:
GET https://data.nemar.org/nm000103/latest/sub-99/eeg/...edf - Response: HTML 404 / JSON 404 with
{ status: 404, reason: "removed", last_seen_version: "v1.0.0", last_seen_url: "https://data.nemar.org/nm000103/v1.0.0/sub-99/eeg/...edf" } - HTML index pages also show a "Files removed since v1.0.0" footer when relevant.
- Request:
- Optional: emit 0-byte placeholder objects with
x-amz-website-redirect-locationfors3://nemar/.../dataset/vX.Y.Z/only if a user explicitly asks for it (defer)
Phase 3: Sync compatibility (optional)
- List endpoint compatible enough that
rclone syncagainsthttps://data.nemar.org/nm.../latest/works correctly with delta detection (ETags, Last-Modified, size)
Out of scope
- Replacing git-annex / DataLad workflow for uploaders. This is download-side only.
- Any changes to
nemar-clicommands. The existingnemar dataset get(and the surroundingclone/save/push/dropcontract) is part of an interconnected git-annex workflow and is intentionally left untouched. A separate CLI wrapper for HTTPS access can be considered later as its own issue if there is real demand. - Private/unpublished dataset access via
data.nemar.org. Usenemar-cli+ git-annex for those (auth, SSH, collaborator model already in place). - Making
objects/public-browseable. - Materialization of
dataset/latest/as real S3 objects (explicitly rejected; see "Whyaws s3 cp s3://...is intentionally not supported" above). - Backing-store migration (R2, CF cache, etc.). NEMAR stays on AWS Open Data sponsored S3.
- Migration of
archives/orversion/off S3 (also stays on Open Data S3).
Open questions
- Unpublish/delete flow specifics: how Workers handles a 404 on a previously-published version where the manifest itself is gone (full version unpublished) vs the tombstone-style case in Phase 2 (manifest still there, file just isn't in this version). Likely just: missing manifest → plain 404 with "this version is not published"; missing file in present manifest → structured tombstone response per Phase 2.
References
- Conversation context: Arno + Bruno feedback on s3://nemar/nm00xxxx/ structure; AWS Open Data Program sponsorship
- Existing prefixes documented in
nemar-cli/CLAUDE.md(S3 Bucket Structure section) - Related: hallu-sync uses the existing
archives/prefix; this proposal does not change that
Activity
arnodelorme commentedon May 12, 2026
bruAristimunha commentedon May 12, 2026
I liked the solution, Yahya! Something that came to mind was the CDN Cloudflare solution for indexing, and I was wondering if we could somehow combine it with the metadata; it would be perfect!
I was playing with eegdash to overcome the s3://object current limitation and implement the eeg/emg/meg/ieeg viewer (https://electrodes.eegdash.org/ and https://eegdash.github.io/eegdash-viewer/). I deployed an experimental index here, and it was very easy with CodeFlare!
or even:
https://cdn.eegdash.org/nm000148/objects/SHA256E-s15851104--74283925ddb8087a9561e2be85ddadd0f44518c120287318c05662b0072b6b1b.bdf
and this unlocks me to do this type of deployment with something only running at the terminal.
https://eegdash.github.io/eegdash-viewer/?dataset=nm000166&sub=021&ses=02&task=ssvepSA&ext=vhdr
neuromechanist commentedon May 14, 2026
Thanks @bruAristimunha — that's the right pattern, and
data.nemar.orgis literally the same trick: Cloudflare in front of S3. The only reason it can't be pure path-passthrough like yourcdn.eegdash.org/ds002893/sub-001/...is that our bucket only hasobjects/SHA...on disk (no BIDS shape — we don't materialize, see the proposal). The Worker does the manifest → object hop, then 302s to the bytes, so to your viewer it looks identical to a passthrough CDN.On the "combine with metadata" point: yes — I'll add a sibling
https://data.nemar.org/<id>/metadata.jsonnext to the per-versionmanifest.json. It'll expose the existing enrichment catalog (authors, modalities, MeSH, DOI, available versions) in one fetch so your viewer can resolvedataset → subjects/tasks → fileswithout crawling BIDS by trial-and-error. That's now Phase 2 in the epic.Question to confirm the contract: does
https://data.nemar.org/<id>/<version>/<bids-path>(Worker-translated, stable, 302 under the hood) work foreegdash-viewer, or do you specifically need a literal passthrough URL with no redirect?bruAristimunha commentedon May 14, 2026
About the contract, it would be better if mirrored the act filesystem under the .
In other words, can we have the sidecars too?
Phase 1: data.nemar.org route + manifest + HTML index (#502)
Phase 3: versioned access UX + file-removed tombstones (#505)
neuromechanist commentedon May 15, 2026
Quick follow-up on the sidecars ask, @bruAristimunha (and now that Phase 3 just landed on the epic branch, the relevant context is concrete enough to point at):
Sidecars already work today via the per-path route. JSON sidecars are git-backed entries in every version manifest, so a URL like:
302s to the actual JSON file (raw.githubusercontent.com under the hood, pinned to the version tag). Same one-request cost as fetching the EDF/BDF.
manifest.jsonenumerates every sidecar. The per-version manifest at/<id>/<version>/manifest.jsonreturns every file in the dataset — including all*.json,*.tsv,participants.tsv, etc. — with its URL. So your viewer can list every sidecar for a dataset in one fetch and resolve them client-side without crawling BIDS by trial-and-error.metadata.jsonstays dataset-level — and that's deliberate. It follows neuroschema'sdataset.schema.json, which models dataset-level metadata (authors, modalities, MeSH keywords, DOI, BIDS subjects/sessions/tasks/runs index). Inlining sidecar contents would:datasetdocument);The existing
extensions.nemar.bids_indexalready gives you{ subject -> sessions, modalities, tasks, runs }, which is enough to construct any sidecar URL deterministically. So the flow is:GET /<id>/metadata.jsononce → catalog + version list + BIDS index.(sub, ses, task, run, modality), build the canonical BIDS path →data.nemar.org/<id>/<version>/<bids-path>(302 to bytes for the JSON sidecar, EDF, etc.).GET /<id>/<version>/manifest.jsononce → flat enumeration of every URL with size + SHA-256.If there's a real use case for a
?type=sidecarfilter on the manifest endpoint (or anything else that meaningfully cuts roundtrips foreegdash-viewer), happy to add it — but I'd want to see the access pattern first so the filter matches what you'd actually use.Phase 3 also added a couple of things you might find useful:
GET /<id>/(default JSON for machine clients) now returns the full version list inline, so you can drop themetadata.jsonround-trip if all you need is "what versions exist + where's the manifest".{ reason: "removed", last_seen_version, last_seen_url }instead of a bare{ error: "File not found" }. The viewer can detect path drops and follow up automatically.Once the epic merges and deploys,
data.nemar.orgstarts serving. Will ping the issue when it's live.bruAristimunha commentedon May 15, 2026
And what about the events and the participants.tsv?
Phase 4: rclone-compatible HEAD + file metadata headers (#506)
neuromechanist commentedon May 15, 2026
Epic complete -- shipped via PR #507 (regular-merged at e101294 on dev). All four phases live:
Plus a cross-phase polish commit (ae95791) addressing a version-format drift across the metadata.json fields, and Cache-Control on the 308 redirect.
Next: deploy to dev Worker, smoke-test data.nemar.org// end-to-end, then ride along on the next dev -> main release.