Skip to content

Onboard existing datasets nm000103-nm000107 into managed workflow #96

Closed

Description

@neuromechanist

Summary

Datasets nm000103-nm000107 were set up manually before the full CLI workflow existed. They need to be brought into the standard managed workflow so they can use PR-based versioning, CI validation, and the complete publication pipeline.

Current State

All 5 datasets already have:

  • GitHub repos under nemarDatasets (private)
  • D1 backend registration (with concept DOIs from Zenodo)
  • Git-annex branch pushed

What's missing:

  • GitHub Actions workflows deployed (only nm000104 has partial: bids-validation + version-check, no pr-merge)
  • Branch protection rules (none on any dataset)
  • Finalization status (may not have been finalized through the backend)
  • Largefiles pattern verification (may not match current standard)
  • S3 remote configuration audit (verify signature v4, correct bucket paths)

Datasets

ID Name Concept DOI Owner
nm000103 HBN-EEG NC 10.5281/zenodo.17306881 yahya
nm000104 emg2qwerty 10.5281/zenodo.17613953 yahya
nm000105 discrete_gestures 10.5281/zenodo.17613958 yahya
nm000106 handwriting 10.5281/zenodo.17613961 yahya
nm000107 wrist 10.5281/zenodo.17613963 yahya

Tasks

For each dataset:

  1. Deploy GitHub Actions workflows - bids-validation.yml, version-check.yml, pr-merge.yml (use the finalize endpoint or deploy manually)
  2. Apply branch protection - PR-mandatory with status checks (bids-validation, version-check), no force pushes, no deletions
  3. Verify git-annex configuration - largefiles pattern, S3 remote with signature v4
  4. Verify finalization state - Ensure backend marks them as finalized so all endpoints work correctly
  5. Test the PR workflow - Clone, create branch, make a trivial change, open PR, verify CI runs, merge

Approach

Consider writing a nemar admin onboard <dataset-id> command or script that automates steps 1-4 for any existing dataset. This would also be useful for future manual imports.

Alternatively, the existing POST /datasets/:id/finalize endpoint may already handle workflow deployment and branch protection; just needs to be called for each dataset.

Important

These are LIVE datasets with real data and real Zenodo concept DOIs. Do not modify their S3 data, DOI records, or repository content. Only add CI/protection infrastructure.

Activity

  1. neuromechanist commented on Feb 2, 2026

    @neuromechanist
    ContributorAuthor

    Onboarding Complete

    All 5 datasets have been onboarded into the managed workflow:

    What was done

    For each dataset (nm000103-nm000107):

    1. GitHub Actions workflows deployed (via nemar admin ci add):

      • bids-validation.yml
      • version-check.yml
      • pr-merge.yml
    2. Branch protection applied (via GitHub API):

      • PR required for main (0 approvals needed, owner self-merge)
      • Status checks required: bids-validation, version-check
      • Force pushes disabled
      • Branch deletion disabled
      • Stale review dismissal enabled
    3. Auto-merge enabled on all repos

    Pre-existing state (unchanged)

    • D1 backend registration with concept DOIs
    • Git-annex branches pushed
    • S3 data in place

    Remaining verification

    • Open a test PR on one dataset to confirm CI workflows run correctly
    • Verify git-annex largefiles pattern matches current standard
    • Verify S3 remote uses signature v4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions