Repository navigation
Closed
Labels
P2Normal priority - standard features and improvementsNormal priority - standard features and improvementsbackendBackend/API relatedBackend/API relatedcliCLI relatedCLI relatedfeatureNew feature or enhancementNew feature or enhancement
Description
Summary
Design and implement a PR workflow for dataset changes that distinguishes between metadata-only changes (easy) and data changes (requires contributor status).
Background
Currently, users can clone a dataset and potentially push data changes without going through proper contributor verification. We need to:
- Make metadata PRs easy for anyone
- Create friction for data changes that requires contributor status first
Current Workflow
Need to explore and document:
- How users currently submit changes
- What DataLad/git-annex operations are involved
- How data vs metadata is distinguished
- Where staging → final S3 copy occurs
Proposed Workflow
For Metadata-Only Changes
- User forks or creates branch
- Makes changes to JSON/TSV files (stored in git proper)
- Opens PR
- Review and merge (straightforward)
For Data Changes
- User must first request contributor access (
nemar dataset request-access <id>) - Wait for approval (automatic for public datasets, manual for private)
- Once approved, can create branch and push data changes
- Data goes to staging S3 during PR
- On merge, staging → final S3 copy (via GitHub Actions)
Key Technical Considerations
Detection Logic
Metadata files: *.json, *.tsv, *.txt, README*, CHANGES*, LICENSE*
Data files: *.edf, *.bdf, *.set, *.fdt, *.vhdr, *.vmrk, *.eeg, (and largerthan threshold)
git-annex largefiles config
git annex config --set annex.largefiles 'include=*.edf or include=*.bdf or largerthan=100kb'Validated Workflow Reference
See .context/validated_workflows.md for tested patterns:
- Git-annex staging → final (use
git annex copy --from --to) - GitHub Actions S3 copy on PR merge
- Branch protection with
enforce_admins=true
Implementation Tasks
- Document current workflow in detail
- Design change detection (metadata vs data)
- Implement pre-push hook or CLI check
- Add contributor verification step
- Update upload workflow to check contributor status
- Add helpful error messages guiding users
UX Goals
- Clear error messages explaining why data changes require contributor status
- Easy path for metadata contributors (no extra friction)
- Transparent process for becoming a contributor
- Prevent accidental large data uploads without proper access
Related
- Issue Make CLI independent of DataLad, bundle git-annex binaries #22 (DataLad independence)
.context/validated_workflows.md(tested patterns)- PR feat: streamline setup with platform-specific guidance and SSH helper #21 (SSH setup, enables contribution workflow)
Metadata
Metadata
Assignees
Labels
P2Normal priority - standard features and improvementsNormal priority - standard features and improvementsbackendBackend/API relatedBackend/API relatedcliCLI relatedCLI relatedfeatureNew feature or enhancementNew feature or enhancement
Activity
feat(zarr): drive hallu cron off the SQLite queue engine
feat(zarr): queue-driven Hallu cron (SQLite engine) (#692)
neuromechanist commentedon Sep 3, 2026
Closing as decided. ADR 0001 (dataset changes go through pull requests) settled the governance model: review is gated on publish state rather than uniform contributor friction. The supporting pieces exist on dev: the collaborator request and approve flow in backend/src/routes/datasets/collaborators.ts, the single-source annex policy module (ADR 0031), and the staged-S3 then copy-on-merge PR workflow documented in .context/validated_workflows.md.