You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Extensible S3 bucket structure with subdirectories for objects, versions, archives, and analysis #71
Builds on the content-addressed storage decision in #56. The current layout puts SHA256E hash files flat at the dataset root with a manifests/ directory. This proposal reorganizes into logical subdirectories that support additional computed artifacts.
Proposed Structure
s3://nemar/
<dataset>/
objects/
SHA256E-s12345--abc123.edf
SHA256E-s67890--def456.set
version/
v100.json # manifest (file→hash mappings, DOI, metadata)
archives/
v100.zip # full dataset snapshot for direct download
derivatives/ # only current version
low-res/
processed/
signalJourney/
v100.json # Signal Journey analysis output
dataquality/
v100.json # data quality metrics
Design Notes
objects/ - Content-addressed files (git-annex hashes). Same data as before, just namespaced under a subdirectory.
version/ - Version manifests (renamed from manifests/). Each JSON maps file paths to object hashes, plus DOI and creation timestamp.
archives/ - Versioned zip files for users who want a simple download without git-annex. Generated at publication time.
derivatives/ - Processed/derived data. Only stores current version to avoid duplication. Could include downsampled data, preprocessed outputs, etc.
signalJourney/ - Per-version Signal Journey analysis metadata.
dataquality/ - Per-version data quality metrics.
Extensibility
The pattern is simple: any new analysis tool gets its own top-level subdirectory under <dataset>/, with versioned JSON outputs. This keeps the bucket organized and allows independent tooling to write results without conflicting.
Open Questions
Should derivatives/ also be versioned (e.g., derivatives/v100/low-res/)? Current proposal says "only current version" to save space.
Should objects/ use a flat layout or nested (e.g., objects/ab/cd/SHA256E-...) for large datasets with many files?
How does this interact with git-annex's S3 special remote key layout? May need a custom fileprefix configuration.
Derivatives only store the current version. Historical processing state is captured by Signal Journey's documentation approach, so no need for versioned derivatives.
The objects/ directory will use git-annex's native content-addressed hash keys (flat). BIDS structure doesn't need to be replicated in S3 because:
The git repo preserves BIDS directory layout (symlinks to annex keys)
Version manifests map BIDS paths → hash keys
objects/ is a blob store, not meant for human browsing
When files are added/removed between versions:
Added: new hash appears in objects/, new entry in manifest
Removed: hash stays in objects/ (old versions reference it), absent from new manifest
Modified: new hash (different content = different key), old hash preserved for old versions
3. git-annex fileprefix - just another layer
Current config uses fileprefix=<dataset>/. Changing to fileprefix=<dataset>/objects/ adds one path segment. Confirmed this should work since it's just a prefix string. Will validate in testing.
Resolved Structure
s3://nemar/
<dataset>/
objects/ # git-annex content-addressed blobs
SHA256E-s12345--abc123.edf
SHA256E-s67890--def456.set
version/ # manifest per published version
v100.json
archives/ # downloadable snapshots
v100.zip
derivatives/ # current version only
low-res/
processed/
signalJourney/ # versioned analysis
v100.json
dataquality/ # versioned quality metrics
v100.json
Blocked by #96 (onboard existing datasets into managed workflow first). Once nm000103-nm000107 are onboarded with CI/protection, the S3 restructuring can proceed as a migration across all datasets.
Activity
neuromechanist commentedon Jan 24, 2026
Clarifications from Discussion
1. Derivatives - confirmed current-version-only
Derivatives only store the current version. Historical processing state is captured by Signal Journey's documentation approach, so no need for versioned derivatives.
2. Objects layout - flat hash keys (git-annex native)
The
objects/directory will use git-annex's native content-addressed hash keys (flat). BIDS structure doesn't need to be replicated in S3 because:objects/is a blob store, not meant for human browsingWhen files are added/removed between versions:
3. git-annex fileprefix - just another layer
Current config uses
fileprefix=<dataset>/. Changing tofileprefix=<dataset>/objects/adds one path segment. Confirmed this should work since it's just a prefix string. Will validate in testing.Resolved Structure
neuromechanist commentedon Feb 2, 2026
Blocked by #96 (onboard existing datasets into managed workflow first). Once nm000103-nm000107 are onboarded with CI/protection, the S3 restructuring can proceed as a migration across all datasets.
Reorganize S3 bucket to subdirectory layout
Reorganize S3 bucket to subdirectory layout (#100)