Skip to content

Extensible S3 bucket structure with subdirectories for objects, versions, archives, and analysis #71

Closed

Description

@neuromechanist

Context

Builds on the content-addressed storage decision in #56. The current layout puts SHA256E hash files flat at the dataset root with a manifests/ directory. This proposal reorganizes into logical subdirectories that support additional computed artifacts.

Proposed Structure

s3://nemar/
    <dataset>/
        objects/
            SHA256E-s12345--abc123.edf
            SHA256E-s67890--def456.set

        version/
            v100.json          # manifest (file→hash mappings, DOI, metadata)

        archives/
            v100.zip           # full dataset snapshot for direct download

        derivatives/           # only current version
            low-res/
            processed/

        signalJourney/
            v100.json          # Signal Journey analysis output

        dataquality/
            v100.json          # data quality metrics

Design Notes

  • objects/ - Content-addressed files (git-annex hashes). Same data as before, just namespaced under a subdirectory.
  • version/ - Version manifests (renamed from manifests/). Each JSON maps file paths to object hashes, plus DOI and creation timestamp.
  • archives/ - Versioned zip files for users who want a simple download without git-annex. Generated at publication time.
  • derivatives/ - Processed/derived data. Only stores current version to avoid duplication. Could include downsampled data, preprocessed outputs, etc.
  • signalJourney/ - Per-version Signal Journey analysis metadata.
  • dataquality/ - Per-version data quality metrics.

Extensibility

The pattern is simple: any new analysis tool gets its own top-level subdirectory under <dataset>/, with versioned JSON outputs. This keeps the bucket organized and allows independent tooling to write results without conflicting.

Open Questions

  • Should derivatives/ also be versioned (e.g., derivatives/v100/low-res/)? Current proposal says "only current version" to save space.
  • Should objects/ use a flat layout or nested (e.g., objects/ab/cd/SHA256E-...) for large datasets with many files?
  • How does this interact with git-annex's S3 special remote key layout? May need a custom fileprefix configuration.

References

Activity

  1. neuromechanist commented on Jan 24, 2026

    @neuromechanist
    ContributorAuthor

    Clarifications from Discussion

    1. Derivatives - confirmed current-version-only

    Derivatives only store the current version. Historical processing state is captured by Signal Journey's documentation approach, so no need for versioned derivatives.

    2. Objects layout - flat hash keys (git-annex native)

    The objects/ directory will use git-annex's native content-addressed hash keys (flat). BIDS structure doesn't need to be replicated in S3 because:

    • The git repo preserves BIDS directory layout (symlinks to annex keys)
    • Version manifests map BIDS paths → hash keys
    • objects/ is a blob store, not meant for human browsing

    When files are added/removed between versions:

    • Added: new hash appears in objects/, new entry in manifest
    • Removed: hash stays in objects/ (old versions reference it), absent from new manifest
    • Modified: new hash (different content = different key), old hash preserved for old versions

    3. git-annex fileprefix - just another layer

    Current config uses fileprefix=<dataset>/. Changing to fileprefix=<dataset>/objects/ adds one path segment. Confirmed this should work since it's just a prefix string. Will validate in testing.

    Resolved Structure

    s3://nemar/
        <dataset>/
            objects/                    # git-annex content-addressed blobs
                SHA256E-s12345--abc123.edf
                SHA256E-s67890--def456.set
    
            version/                   # manifest per published version
                v100.json
    
            archives/                  # downloadable snapshots
                v100.zip
    
            derivatives/               # current version only
                low-res/
                processed/
    
            signalJourney/             # versioned analysis
                v100.json
    
            dataquality/               # versioned quality metrics
                v100.json
    
  2. neuromechanist commented on Feb 2, 2026

    @neuromechanist
    ContributorAuthor

    Blocked by #96 (onboard existing datasets into managed workflow first). Once nm000103-nm000107 are onboarded with CI/protection, the S3 restructuring can proceed as a migration across all datasets.

  3. added 2 commits that reference this issue on Feb 6, 2026
    ad6ac27
    e4b525f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions