Repository navigation
Description
Problem Statement
Once a DOI is assigned to a dataset version, that exact data must remain accessible forever. Currently, our S3 storage structure doesn't protect against overwrites, which could invalidate DOIs.
Research Findings
Based on best practices from Zenodo, OpenNeuro, and RDA Data Versioning Guidelines:
Key Principles
- Two-DOI System: Zenodo uses a "Concept DOI" (all versions) + "Version DOI" (specific version)
- Files trigger versions, not metadata: Zenodo only creates new versions when files change; metadata edits don't trigger new DOIs
- DOIs should NOT contain version numbers: Version info in DOIs is bad practice since DOIs must be permanent
- OpenNeuro uses git tags: Snapshots are implemented as git tags, allowing specific versions to be easily retrieved
- Efficient storage: Zenodo only stores changed files, not full copies
Proposed Architecture: Git-Annex + Tags + Immutability
Core Insight
Git-annex already provides content-addressed storage. Files are stored in S3 by their hash keys (e.g., SHA256E-s12345--abc123.edf). Combined with git tags, this gives us everything we need for versioning:
- Content-addressed storage: git-annex keys in S3 (deduplication is automatic)
- Version snapshots: git tags (
v1.0.0,v1.1.0) - File-to-hash mapping: annex pointer files in the git tree at each tag
- Deduplication: unchanged files share the same annex key across versions (no copies needed)
No S3 version folders required. No duplication of data between versions.
What We Add on Top
- Immutability enforcement - prevent deletion/overwrite of S3 objects referenced by tagged versions
- Version manifests (JSON) - optional file-to-S3-key mappings for direct access without git (web frontend, on-the-fly plotting)
- DOI registration - Zenodo concept + version DOI creation on tag
Dataset Lifecycle
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ PRIVATE │ │ PUBLISHED │ │ VERSIONED │
│ (Development) │───▶│ (First DOI) │───▶│ (Updates) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
No tag Tag: v1.0.0 Tags: v1.1.0, v2.0.0
No DOI Concept + Version New Version DOIs
Mutable S3 objects S3 Object Lock on Lock new objects
referenced keys
S3 Storage (Unchanged from Current)
s3://nemar/nm000151/
├── SHA256E-s12345--abc123.edf # git-annex managed
├── SHA256E-s67890--def456.edf # shared across versions if unchanged
└── ...
No version folders. Git-annex handles the content-addressing. The git tree at each tag defines which files belong to that version.
Version Manifests (JSON)
For non-git access (web frontend, on-the-fly data plotting), we generate a manifest at publication time:
// Stored at s3://nemar/nm000151/manifests/v1.0.0.json
{
"dataset": "nm000151",
"version": "1.0.0",
"doi": "10.5281/zenodo.VERSION1",
"concept_doi": "10.5281/zenodo.CONCEPT",
"created": "2026-01-20T00:00:00Z",
"files": {
"sub-01/eeg/sub-01_task-rest_eeg.edf": {
"key": "SHA256E-s12345--abc123.edf",
"size": 12345,
"checksum": "sha256:abc123..."
},
"sub-01/eeg/sub-01_task-rest_events.tsv": {
"key": "SHA256E-s67890--def456.tsv",
"size": 890,
"checksum": "sha256:def456..."
}
}
}This allows the web frontend to resolve file paths to S3 keys directly, enabling on-the-fly plotting or downloads without cloning the git repo.
Immutability Enforcement
S3 Object Lock
When a version is published (tag + DOI assigned):
- Identify all annex keys referenced by the tagged commit
- Apply S3 Object Lock (Governance or Compliance mode) to those keys
- Locked objects cannot be deleted or overwritten
Git Tag Protection
- GitHub tag protection rules prevent tag deletion/modification
- Only admins can create version tags
- Tags are created as part of the DOI registration workflow
Version Bump Semantics
| Change Type | Version Bump | New DOI? |
|---|---|---|
| Metadata only (README, description) | Patch (1.0.0 → 1.0.1) | Yes |
| New subjects/sessions added | Minor (1.0.0 → 1.1.0) | Yes |
| Reprocessing, breaking changes | Major (1.0.0 → 2.0.0) | Yes |
| Bug fixes in data | Patch (1.0.0 → 1.0.1) | Yes |
Workflow
1. Initial Upload (Private, Mutable)
nemar dataset upload ./my-dataset
# → Creates nm000151 repo
# → Uploads files via git-annex to S3
# → No tag, no DOI, S3 objects are mutable
# → User can overwrite/update freely via PRs2. First Publication (Tag + Lock + DOI)
nemar admin doi create nm000151
# → Validates dataset (BIDS, completeness)
# → Creates git tag v1.0.0
# → Generates manifest JSON, uploads to S3
# → Applies S3 Object Lock to all referenced annex keys
# → Registers with Zenodo:
# - Concept DOI: 10.5281/zenodo.CONCEPT
# - Version DOI: 10.5281/zenodo.VERSION1
# → Updates dataset_description.json with DOI3. Updates (PR + New Tag)
# User creates PR with changes (new/modified files)
# CI validates:
# - Version bumped in dataset_description.json
# - New version > current version
# - Change type matches bump level
# On PR merge:
nemar admin doi version nm000151
# → Creates git tag for new version
# → Generates new manifest JSON
# → Locks newly-added annex keys
# → Registers new Version DOI with ZenodoCI Validation for PRs
# .github/workflows/version-check.yml
- name: Check version bump
run: |
MAIN_VERSION=$(git show origin/main:dataset_description.json | jq -r .Version)
PR_VERSION=$(jq -r .Version dataset_description.json)
# Ensure PR version > main version
# If data files changed, ensure minor/major bumpOpen Questions
-
S3 Object Lock mode: Governance (admin can override) vs Compliance (nobody can override, not even root)?
- Governance is safer for mistakes but less strict
- Compliance is truly immutable but risks permanent storage of bad data
-
Manifest generation timing: Generate on tag creation, or lazily on first web access?
- Eager: always available, small upfront cost
- Lazy: no cost for datasets nobody accesses via web
-
Backwards compatibility: Existing datasets without tags need a migration path
- Option: grandfather them as v1.0.0 with current state
- Option: leave unversioned until next update
-
Shared annex keys across datasets: If two datasets reference the same file (unlikely but possible), locking in one affects the other. Acceptable?
Implementation Phases
Phase 1: Tagging and Manifests
- Add version tagging to
nemar admin doi create - Implement manifest JSON generation from git-annex state
- Upload manifests to S3 at
manifests/v*.json - Add GitHub tag protection rules to dataset repos
Phase 2: Immutability
- Implement S3 Object Lock for annex keys on publication
- Add pre-delete checks (reject deletion of locked objects)
- Test lock behavior with git-annex drop/move commands
Phase 3: DOI Integration
- Zenodo API integration for concept + version DOI creation
- Update dataset_description.json with DOIs
- Link DOI metadata to manifest
Phase 4: Update Flow
- CI workflow for version bump validation in PRs
-
nemar admin doi versioncommand for subsequent versions - Manifest diff tool (compare versions)
Related Issues
- Improve upload workflow: auto-accept invitations, commit authorship, resume support #41 (Improve upload workflow) - Parent issue for upload improvements
Activity
neuromechanist commentedon Jan 21, 2026
Decision: Content-Addressed Storage (Option B)
After discussion, we've decided on content-addressed storage as the only viable approach that:
Final Architecture
How It Works
Manifest Format
{ "version": "1.0.0", "doi": "10.5281/zenodo.123456", "created": "2026-01-21T12:00:00Z", "files": { "sub-01/eeg/sub-01_task-rest_eeg.edf": "SHA256E-s12345--abc123.edf", "sub-01/eeg/sub-01_task-rest_eeg.json": "SHA256E-s500--xyz789.json" } }Version Lifecycle
Key Insight
This requires minimal changes since git-annex already handles content-addressed storage. The main additions are:
neuromechanist commentedon Jan 21, 2026
@arnodelorme, any thoughts?
neuromechanist commentedon Jan 23, 2026
Publication workflow (request/approve pipeline) now tracked in #68. Issue #56 covers the post-publication layer: versioning, DOI management, and S3 Object Lock. The
nemar admin s3 lockandnemar admin doi createcommands from this issue are called by the orchestrator in #68.neuromechanist commentedon Jan 24, 2026
Implemented in PR #75: version manifests (JSON), tag protection rules, S3 Object Lock, manifest CLI command, webhook-based manifest generation.