Skip to content

S3 dataset/ prefix: tool-agnostic cloud-like access to NEMAR datasets #449

Closed
4 of 4 issues completed
Closed

Description

@neuromechanist

Motivation

Right now, accessing a NEMAR dataset's files requires understanding either git-annex (via DataLad/nemar-cli) or the content-addressed objects/ layout (an internal git-annex implementation detail). Both are walls for:

  • Casual users who just want to download a few files in a browser
  • Downstream tools/pipelines that want plain HTTP paths
  • Machines/scripts that don't want to install nemar-cli or git-annex
  • Users coming from OpenNeuro who expect a simple cloud-like file layout

Arno (co-owner) and Bruno raised this as a UX gap. The existing archives/v1.0.0.zip covers "give me everything in one zip" but not "let me browse / selectively download / incrementally sync".

Proposal

Expose a tool-agnostic, BIDS-shaped view of every public dataset over HTTPS at https://data.nemar.org/<datasetId>/<version>/..., served by a Cloudflare Workers route that resolves logical paths to backing-store objects via the existing version/vX.Y.Z.json manifest.

No S3 duplication. Nothing new is materialized in S3. The dataset/ namespace is virtual; the Workers proxy is the only "pointer layer".

https://data.nemar.org/nm000103/latest/                   # HTML index of latest published version
https://data.nemar.org/nm000103/latest/sub-01/eeg/...edf  # 302 -> presigned objects/SHA... URL
https://data.nemar.org/nm000103/v1.0.0/...                # same, addressed by explicit version
https://data.nemar.org/nm000103/latest/manifest.json      # [{path, size, sha256, url}, ...]

Underlying S3 layout is unchanged:

s3://nemar/nm00xxxx/
    objects/    # git-annex content-addressed blobs (SHA-named); internal, no public listing
    version/    # version manifests (vX.Y.Z.json); existing
    archives/   # downloadable zip snapshots; existing (still the canonical 'whole dataset' path)

Worker behavior

The Workers proxy 302-redirects to presigned URLs for actual file bytes; it does NOT stream data through itself. Consequences:

  • Parallel downloads scale to S3's request rate, not Worker CPU/bandwidth
  • Worker only handles manifest lookup, HTML index rendering, and redirect: minimal compute cost
  • Public datasets only. Private/unpublished datasets are NOT exposed via this HTTPS path; the existing nemar-cli + git-annex flow remains the only access route for those. This keeps Phase 1 simple (no auth gating, no token-in-URL/header concerns) and matches the OpenNeuro pattern.

Why direct objects/ access is discouraged

The objects/ prefix MUST NOT be the public access path, even though it physically holds the bytes:

  1. Content-addressed, not human-readable: files are named SHA256-s12345--abcd1234.../.... You can't find sub-01_task-rest_eeg.edf without first reading the git-annex pointer files.
  2. No directory structure: flat blob storage; no BIDS hierarchy.
  3. Internal git-annex implementation detail: git-annex's object key format is subject to change (MD5E-, SHA256E-, SHA256-, etc.). Code that hardcodes object paths breaks across git-annex versions or backend changes.
  4. No version semantics: objects/ mixes all versions of all files. You don't know which file belongs to v1.0.0 vs v1.1.0 without the manifest.
  5. No deletion safety: objects/ is governed by git-annex GC rules. Direct consumers risk dangling references when old content is dropped.
  6. Fragile against migration: if NEMAR ever reorganizes the bucket layout, objects/ paths break. data.nemar.org/... is the stable contract.

Therefore: objects/ will have public listing disabled at the bucket policy level. Direct GETs may still work for now (git-annex needs them) but are unsupported as a stable API.

Why aws s3 cp s3://... is intentionally not supported for dataset/

The natural follow-up question is "can I aws s3 cp s3://nemar/nm000103/dataset/latest/ ./?" — no, by design. Reasons:

  • It would require materializing every published version into S3, ~doubling storage for the latest snapshot of all ~300 datasets (50 TB → +50 TB). NEMAR's S3 is hosted under the AWS Open Data Program; AWS sponsors storage and egress, but our quota/bookkeeping footprint is still ours to manage, and doubling it just to support s3:// muscle memory is the wrong tradeoff.
  • Materialization is fragile: partial states during publish, orphans on failed delete, drift between manifest and bytes. Single source of truth (the manifest + objects/) is dramatically simpler.
  • Two equivalent UX paths already exist that don't require materialization: archives/vX.Y.Z.zip (single aws s3 cp of a zip) and HTTPS access via data.nemar.org (works with every parallel downloader; see table below).

If a real audience surfaces later for s3://nemar/.../dataset/latest/ access specifically, materialization can be added without changing the public HTTPS contract.

Parallel download support (-j 16 style)

Parallelism is a client concern, not a protocol concern. All major parallel-download tools work natively over HTTPS because the Worker 302s to presigned URLs and parallel clients hit S3 directly:

Tool Command
aria2c -j 16 curl -sL https://data.nemar.org/nm000103/latest/manifest.json | jq -r '.[].url' | aria2c -j 16 -i -
rclone --transfers 16 rclone copy --transfers 16 --multi-thread-streams 4 :http:data.nemar.org/nm000103/latest/ ./
wget --mirror wget -r -np https://data.nemar.org/nm000103/latest/
curl + xargs -P 16 xargs -P 16 -n 1 curl -O < urls.txt
Whole-dataset zip aws s3 cp s3://nemar/nm000103/archives/v1.0.0.zip ./ (unchanged, still works)

The manifest endpoint makes any parallel downloader work in one line, and clients can verify SHA-256 checksums for free (content-addressing inherited from git-annex).

This epic does NOT introduce or modify any nemar-cli commands. The existing git-annex contract (nemar dataset clone / get / save / push / drop) is unchanged and untouched. A NEMAR-specific CLI wrapper for the HTTPS path can be considered as a separate issue later if users actually ask for one; until then, the existing third-party tools above cover the use case.

Storage backing

NEMAR's S3 bucket is hosted under the AWS Open Data Program; AWS sponsors storage and egress. This epic does NOT migrate the backing store and does NOT introduce a CDN layer. The Workers route reads existing S3 objects via presigned URLs; the only new infrastructure is the Worker route, the manifest endpoint, and the HTML index renderer. Keeping S3 as-is is required to maintain Open Data Program eligibility.

The cost rationale for not materializing dataset/latest/ is therefore operational, not financial: avoid drift, partial states, and doubling our Open Data quota footprint.

Phasing

Phase 1: Workers route + manifest + HTML index

  • New Workers route: GET /data/:datasetId/:version/*path resolves via manifest, 302s to presigned S3 URL
  • :version accepts latest, vX.Y.Z
  • HTML index style: Apache-style directory listing with minimal CSS (similar to AWS S3 static website listings; functional, no theme branding)
  • Manifest JSON endpoint: GET /data/:datasetId/:version/manifest.json with presigned URLs and SHA-256s
  • Public subdomain: data.nemar.org (CNAME to Workers)
  • Docs: docs/guides/downloading.md section on HTTPS access + parallel-download tooling
  • Public datasets only; private/unpublished datasets return 404

Phase 2: Versioned access UX + tombstone indication

  • Version picker in HTML index; sitemap-style listing of available versions at /data/:datasetId/
  • /latest resolves to canonical version
  • Tombstone / file-removed-in-newer-version indication: when a path exists in an older version but not in the requested version (typically latest), return a structured 404 that points the user to the last version where the file existed. Example:
    • Request: GET https://data.nemar.org/nm000103/latest/sub-99/eeg/...edf
    • Response: HTML 404 / JSON 404 with { status: 404, reason: "removed", last_seen_version: "v1.0.0", last_seen_url: "https://data.nemar.org/nm000103/v1.0.0/sub-99/eeg/...edf" }
    • HTML index pages also show a "Files removed since v1.0.0" footer when relevant.
  • Optional: emit 0-byte placeholder objects with x-amz-website-redirect-location for s3://nemar/.../dataset/vX.Y.Z/ only if a user explicitly asks for it (defer)

Phase 3: Sync compatibility (optional)

  • List endpoint compatible enough that rclone sync against https://data.nemar.org/nm.../latest/ works correctly with delta detection (ETags, Last-Modified, size)

Out of scope

  • Replacing git-annex / DataLad workflow for uploaders. This is download-side only.
  • Any changes to nemar-cli commands. The existing nemar dataset get (and the surrounding clone / save / push / drop contract) is part of an interconnected git-annex workflow and is intentionally left untouched. A separate CLI wrapper for HTTPS access can be considered later as its own issue if there is real demand.
  • Private/unpublished dataset access via data.nemar.org. Use nemar-cli + git-annex for those (auth, SSH, collaborator model already in place).
  • Making objects/ public-browseable.
  • Materialization of dataset/latest/ as real S3 objects (explicitly rejected; see "Why aws s3 cp s3://... is intentionally not supported" above).
  • Backing-store migration (R2, CF cache, etc.). NEMAR stays on AWS Open Data sponsored S3.
  • Migration of archives/ or version/ off S3 (also stays on Open Data S3).

Open questions

  • Unpublish/delete flow specifics: how Workers handles a 404 on a previously-published version where the manifest itself is gone (full version unpublished) vs the tombstone-style case in Phase 2 (manifest still there, file just isn't in this version). Likely just: missing manifest → plain 404 with "this version is not published"; missing file in present manifest → structured tombstone response per Phase 2.

References

  • Conversation context: Arno + Bruno feedback on s3://nemar/nm00xxxx/ structure; AWS Open Data Program sponsorship
  • Existing prefixes documented in nemar-cli/CLAUDE.md (S3 Bucket Structure section)
  • Related: hallu-sync uses the existing archives/ prefix; this proposal does not change that

Activity

  1. added
    P2Normal priority - standard features and improvements
    featureNew feature or enhancement
    on May 12, 2026
  2. arnodelorme commented on May 12, 2026

    @arnodelorme
  3. bruAristimunha commented on May 12, 2026

    @bruAristimunha
    Contributor

    I liked the solution, Yahya! Something that came to mind was the CDN Cloudflare solution for indexing, and I was wondering if we could somehow combine it with the metadata; it would be perfect!

    I was playing with eegdash to overcome the s3://object current limitation and implement the eeg/emg/meg/ieeg viewer (https://electrodes.eegdash.org/ and https://eegdash.github.io/eegdash-viewer/). I deployed an experimental index here, and it was very easy with CodeFlare!

    https://cdn.eegdash.org/ds002893/dataset_description.json
    https://cdn.eegdash.org/ds002893/sub-001/eeg/sub-001_task-rest_eeg.edf

    or even:

    https://cdn.eegdash.org/nm000148/objects/SHA256E-s15851104--74283925ddb8087a9561e2be85ddadd0f44518c120287318c05662b0072b6b1b.bdf

    and this unlocks me to do this type of deployment with something only running at the terminal.

    https://eegdash.github.io/eegdash-viewer/?dataset=nm000166&sub=021&ses=02&task=ssvepSA&ext=vhdr

  4. neuromechanist commented on May 14, 2026

    @neuromechanist
    ContributorAuthor

    Thanks @bruAristimunha — that's the right pattern, and data.nemar.org is literally the same trick: Cloudflare in front of S3. The only reason it can't be pure path-passthrough like your cdn.eegdash.org/ds002893/sub-001/... is that our bucket only has objects/SHA... on disk (no BIDS shape — we don't materialize, see the proposal). The Worker does the manifest → object hop, then 302s to the bytes, so to your viewer it looks identical to a passthrough CDN.

    On the "combine with metadata" point: yes — I'll add a sibling https://data.nemar.org/<id>/metadata.json next to the per-version manifest.json. It'll expose the existing enrichment catalog (authors, modalities, MeSH, DOI, available versions) in one fetch so your viewer can resolve dataset → subjects/tasks → files without crawling BIDS by trial-and-error. That's now Phase 2 in the epic.

    Question to confirm the contract: does https://data.nemar.org/<id>/<version>/<bids-path> (Worker-translated, stable, 302 under the hood) work for eegdash-viewer, or do you specifically need a literal passthrough URL with no redirect?

  5. bruAristimunha commented on May 14, 2026

    @bruAristimunha
    Contributor

    About the contract, it would be better if mirrored the act filesystem under the .

    In other words, can we have the sidecars too?

  6. neuromechanist commented on May 15, 2026

    @neuromechanist
    ContributorAuthor

    Quick follow-up on the sidecars ask, @bruAristimunha (and now that Phase 3 just landed on the epic branch, the relevant context is concrete enough to point at):

    Sidecars already work today via the per-path route. JSON sidecars are git-backed entries in every version manifest, so a URL like:

    https://data.nemar.org/nm000148/latest/sub-001/eeg/sub-001_task-rest_eeg.json
    

    302s to the actual JSON file (raw.githubusercontent.com under the hood, pinned to the version tag). Same one-request cost as fetching the EDF/BDF.

    manifest.json enumerates every sidecar. The per-version manifest at /<id>/<version>/manifest.json returns every file in the dataset — including all *.json, *.tsv, participants.tsv, etc. — with its URL. So your viewer can list every sidecar for a dataset in one fetch and resolve them client-side without crawling BIDS by trial-and-error.

    metadata.json stays dataset-level — and that's deliberate. It follows neuroschema's dataset.schema.json, which models dataset-level metadata (authors, modalities, MeSH keywords, DOI, BIDS subjects/sessions/tasks/runs index). Inlining sidecar contents would:

    • balloon the payload past tens of MB for large datasets (thousands of recordings × kilobytes each);
    • step outside the schema (recording-level fields don't belong in a dataset document);
    • and recompute the same information the viewer can resolve from the manifest in one URL away.

    The existing extensions.nemar.bids_index already gives you { subject -> sessions, modalities, tasks, runs }, which is enough to construct any sidecar URL deterministically. So the flow is:

    1. GET /<id>/metadata.json once → catalog + version list + BIDS index.
    2. For any (sub, ses, task, run, modality), build the canonical BIDS path → data.nemar.org/<id>/<version>/<bids-path> (302 to bytes for the JSON sidecar, EDF, etc.).
    3. Or GET /<id>/<version>/manifest.json once → flat enumeration of every URL with size + SHA-256.

    If there's a real use case for a ?type=sidecar filter on the manifest endpoint (or anything else that meaningfully cuts roundtrips for eegdash-viewer), happy to add it — but I'd want to see the access pattern first so the filter matches what you'd actually use.

    Phase 3 also added a couple of things you might find useful:

    • GET /<id>/ (default JSON for machine clients) now returns the full version list inline, so you can drop the metadata.json round-trip if all you need is "what versions exist + where's the manifest".
    • 404s on a path that was removed between versions now return { reason: "removed", last_seen_version, last_seen_url } instead of a bare { error: "File not found" }. The viewer can detect path drops and follow up automatically.

    Once the epic merges and deploys, data.nemar.org starts serving. Will ping the issue when it's live.

  7. bruAristimunha commented on May 15, 2026

    @bruAristimunha
    Contributor

    And what about the events and the participants.tsv?

  8. neuromechanist commented on May 15, 2026

    @neuromechanist
    ContributorAuthor

    Epic complete -- shipped via PR #507 (regular-merged at e101294 on dev). All four phases live:

    Plus a cross-phase polish commit (ae95791) addressing a version-format drift across the metadata.json fields, and Cache-Control on the 308 redirect.

    Next: deploy to dev Worker, smoke-test data.nemar.org// end-to-end, then ride along on the next dev -> main release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal priority - standard features and improvementsfeatureNew feature or enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions