Skip to content

Design: Versioning architecture for DOI protection and data immutability #56

Closed
1 of 1 issue completed

Description

@neuromechanist

Problem Statement

Once a DOI is assigned to a dataset version, that exact data must remain accessible forever. Currently, our S3 storage structure doesn't protect against overwrites, which could invalidate DOIs.

Research Findings

Based on best practices from Zenodo, OpenNeuro, and RDA Data Versioning Guidelines:

Key Principles

  1. Two-DOI System: Zenodo uses a "Concept DOI" (all versions) + "Version DOI" (specific version)
  2. Files trigger versions, not metadata: Zenodo only creates new versions when files change; metadata edits don't trigger new DOIs
  3. DOIs should NOT contain version numbers: Version info in DOIs is bad practice since DOIs must be permanent
  4. OpenNeuro uses git tags: Snapshots are implemented as git tags, allowing specific versions to be easily retrieved
  5. Efficient storage: Zenodo only stores changed files, not full copies

Proposed Architecture: Git-Annex + Tags + Immutability

Core Insight

Git-annex already provides content-addressed storage. Files are stored in S3 by their hash keys (e.g., SHA256E-s12345--abc123.edf). Combined with git tags, this gives us everything we need for versioning:

  • Content-addressed storage: git-annex keys in S3 (deduplication is automatic)
  • Version snapshots: git tags (v1.0.0, v1.1.0)
  • File-to-hash mapping: annex pointer files in the git tree at each tag
  • Deduplication: unchanged files share the same annex key across versions (no copies needed)

No S3 version folders required. No duplication of data between versions.

What We Add on Top

  1. Immutability enforcement - prevent deletion/overwrite of S3 objects referenced by tagged versions
  2. Version manifests (JSON) - optional file-to-S3-key mappings for direct access without git (web frontend, on-the-fly plotting)
  3. DOI registration - Zenodo concept + version DOI creation on tag

Dataset Lifecycle

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   PRIVATE       │    │   PUBLISHED     │    │   VERSIONED     │
│   (Development) │───▶│   (First DOI)   │───▶│   (Updates)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
   No tag               Tag: v1.0.0           Tags: v1.1.0, v2.0.0
   No DOI               Concept + Version     New Version DOIs
   Mutable S3 objects   S3 Object Lock on     Lock new objects
                        referenced keys

S3 Storage (Unchanged from Current)

s3://nemar/nm000151/
├── SHA256E-s12345--abc123.edf    # git-annex managed
├── SHA256E-s67890--def456.edf    # shared across versions if unchanged
└── ...

No version folders. Git-annex handles the content-addressing. The git tree at each tag defines which files belong to that version.

Version Manifests (JSON)

For non-git access (web frontend, on-the-fly data plotting), we generate a manifest at publication time:

// Stored at s3://nemar/nm000151/manifests/v1.0.0.json
{
  "dataset": "nm000151",
  "version": "1.0.0",
  "doi": "10.5281/zenodo.VERSION1",
  "concept_doi": "10.5281/zenodo.CONCEPT",
  "created": "2026-01-20T00:00:00Z",
  "files": {
    "sub-01/eeg/sub-01_task-rest_eeg.edf": {
      "key": "SHA256E-s12345--abc123.edf",
      "size": 12345,
      "checksum": "sha256:abc123..."
    },
    "sub-01/eeg/sub-01_task-rest_events.tsv": {
      "key": "SHA256E-s67890--def456.tsv",
      "size": 890,
      "checksum": "sha256:def456..."
    }
  }
}

This allows the web frontend to resolve file paths to S3 keys directly, enabling on-the-fly plotting or downloads without cloning the git repo.

Immutability Enforcement

S3 Object Lock

When a version is published (tag + DOI assigned):

  1. Identify all annex keys referenced by the tagged commit
  2. Apply S3 Object Lock (Governance or Compliance mode) to those keys
  3. Locked objects cannot be deleted or overwritten

Git Tag Protection

  • GitHub tag protection rules prevent tag deletion/modification
  • Only admins can create version tags
  • Tags are created as part of the DOI registration workflow

Version Bump Semantics

Change Type Version Bump New DOI?
Metadata only (README, description) Patch (1.0.0 → 1.0.1) Yes
New subjects/sessions added Minor (1.0.0 → 1.1.0) Yes
Reprocessing, breaking changes Major (1.0.0 → 2.0.0) Yes
Bug fixes in data Patch (1.0.0 → 1.0.1) Yes

Workflow

1. Initial Upload (Private, Mutable)

nemar dataset upload ./my-dataset
# → Creates nm000151 repo
# → Uploads files via git-annex to S3
# → No tag, no DOI, S3 objects are mutable
# → User can overwrite/update freely via PRs

2. First Publication (Tag + Lock + DOI)

nemar admin doi create nm000151
# → Validates dataset (BIDS, completeness)
# → Creates git tag v1.0.0
# → Generates manifest JSON, uploads to S3
# → Applies S3 Object Lock to all referenced annex keys
# → Registers with Zenodo:
#   - Concept DOI: 10.5281/zenodo.CONCEPT
#   - Version DOI: 10.5281/zenodo.VERSION1
# → Updates dataset_description.json with DOI

3. Updates (PR + New Tag)

# User creates PR with changes (new/modified files)
# CI validates:
#   - Version bumped in dataset_description.json
#   - New version > current version
#   - Change type matches bump level

# On PR merge:
nemar admin doi version nm000151
# → Creates git tag for new version
# → Generates new manifest JSON
# → Locks newly-added annex keys
# → Registers new Version DOI with Zenodo

CI Validation for PRs

# .github/workflows/version-check.yml
- name: Check version bump
  run: |
    MAIN_VERSION=$(git show origin/main:dataset_description.json | jq -r .Version)
    PR_VERSION=$(jq -r .Version dataset_description.json)
    # Ensure PR version > main version
    # If data files changed, ensure minor/major bump

Open Questions

  1. S3 Object Lock mode: Governance (admin can override) vs Compliance (nobody can override, not even root)?

    • Governance is safer for mistakes but less strict
    • Compliance is truly immutable but risks permanent storage of bad data
  2. Manifest generation timing: Generate on tag creation, or lazily on first web access?

    • Eager: always available, small upfront cost
    • Lazy: no cost for datasets nobody accesses via web
  3. Backwards compatibility: Existing datasets without tags need a migration path

    • Option: grandfather them as v1.0.0 with current state
    • Option: leave unversioned until next update
  4. Shared annex keys across datasets: If two datasets reference the same file (unlikely but possible), locking in one affects the other. Acceptable?

Implementation Phases

Phase 1: Tagging and Manifests

  • Add version tagging to nemar admin doi create
  • Implement manifest JSON generation from git-annex state
  • Upload manifests to S3 at manifests/v*.json
  • Add GitHub tag protection rules to dataset repos

Phase 2: Immutability

  • Implement S3 Object Lock for annex keys on publication
  • Add pre-delete checks (reject deletion of locked objects)
  • Test lock behavior with git-annex drop/move commands

Phase 3: DOI Integration

  • Zenodo API integration for concept + version DOI creation
  • Update dataset_description.json with DOIs
  • Link DOI metadata to manifest

Phase 4: Update Flow

  • CI workflow for version bump validation in PRs
  • nemar admin doi version command for subsequent versions
  • Manifest diff tool (compare versions)

Related Issues

References

Activity

  1. neuromechanist commented on Jan 21, 2026

    @neuromechanist
    ContributorAuthor

    Decision: Content-Addressed Storage (Option B)

    After discussion, we've decided on content-addressed storage as the only viable approach that:

    1. Preserves version access (each version retrievable by DOI)
    2. Doesn't duplicate unchanged files between versions
    3. Works naturally with git-annex (which already uses this approach)

    Final Architecture

    s3://nemar/nm000151/
    ├── SHA256E-s12345--abc123.edf      # Actual data (content-addressed by git-annex)
    ├── SHA256E-s67890--def456.edf
    ├── ...
    └── manifests/
        ├── v1.0.0.json                  # Backup manifest for disaster recovery
        └── v1.1.0.json
    

    How It Works

    1. Git-annex already stores files by hash - this is its default behavior
    2. Git tags serve as version markers - each tag points to a commit that knows the file→hash mappings
    3. JSON manifests are a safety net - human-readable backup that can reconstruct the dataset if needed

    Manifest Format

    {
      "version": "1.0.0",
      "doi": "10.5281/zenodo.123456",
      "created": "2026-01-21T12:00:00Z",
      "files": {
        "sub-01/eeg/sub-01_task-rest_eeg.edf": "SHA256E-s12345--abc123.edf",
        "sub-01/eeg/sub-01_task-rest_eeg.json": "SHA256E-s500--xyz789.json"
      }
    }

    Version Lifecycle

    1. Private (v1.0.0-dev): Files uploaded, mutable, no DOI
    2. Published (v1.0.0):
      • Git tag created
      • Manifest JSON generated and uploaded to S3
      • DOI assigned (concept + version)
      • Version becomes immutable
    3. Updates (v1.1.0):
      • PR requires version bump
      • New/changed files get new hashes
      • Unchanged files share existing hashes (no duplication)
      • New manifest JSON created
      • New version DOI assigned

    Key Insight

    This requires minimal changes since git-annex already handles content-addressed storage. The main additions are:

    • Generate and upload manifest JSON at publication time
    • Enforce version bumps in PRs (CI check)
    • Track DOI → version → manifest mapping
  2. neuromechanist commented on Jan 21, 2026

    @neuromechanist
    ContributorAuthor

    @arnodelorme, any thoughts?

  3. neuromechanist commented on Jan 23, 2026

    @neuromechanist
    ContributorAuthor

    Publication workflow (request/approve pipeline) now tracked in #68. Issue #56 covers the post-publication layer: versioning, DOI management, and S3 Object Lock. The nemar admin s3 lock and nemar admin doi create commands from this issue are called by the orchestrator in #68.

  4. neuromechanist commented on Jan 24, 2026

    @neuromechanist
    ContributorAuthor

    Implemented in PR #75: version manifests (JSON), tag protection rules, S3 Object Lock, manifest CLI command, webhook-based manifest generation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions