Skip to content

Fix dataset upload resume: server-side dedup, ID cap, bulk delete - #335

Merged
neuromechanist merged 3 commits into
mainfrom
334-fix-dataset-upload-resume-phantom-datasets-and-nm100000-generation
Mar 30, 2026
Merged

neuromechanist merged 3 commits into
mainfrom
334-fix-dataset-upload-resume-phantom-datasets-and-nm100000-generation

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Fixes #334. User Bruno created ~80+ phantom datasets (nm000136-nm000337, nm100000-nm100001) because every failed upload retry created a new dataset instead of resuming.

  • Server-side dedup: POST /datasets now checks for existing incomplete datasets (same owner, name, private, no DOI, no versions) before creating new ones. Returns the existing dataset with resumed: true.
  • ID cap at 99999: Gap-filling algorithm now enforces MAX_NUMBER = 99999 and excludes nm099999 (test dataset) from candidates. Prevents overflow IDs like nm100000.
  • isValidDatasetId() hardened: Rejects IDs with numeric part > 99999.
  • Bulk delete endpoint: POST /admin/datasets/bulk-delete + nemar admin bulk-delete CLI command for cleaning up phantom datasets (owner-only, rejects public/DOI datasets).
  • Client-side resume handling: CLI detects resumed flag, shows appropriate message, skips IAM propagation wait.

Changed files

  • backend/src/services/datasetId.ts - MAX_NUMBER cap, EXCLUDED_IDS, validation fix
  • backend/src/routes/datasets.ts - Server-side dedup query in POST /datasets
  • backend/src/routes/admin.ts - Bulk delete endpoint
  • src/commands/dataset.ts - Handle resumed flag in CLI
  • src/commands/admin.ts - bulk-delete CLI command
  • src/lib/api.ts - CreateDatasetResponse.resumed, bulkDeleteDatasets()
  • test/dataset-id.test.ts - Unit tests for ID validation

Test plan

  • Unit tests: isValidDatasetId rejects nm100000+, accepts valid IDs (5/5 pass)
  • TypeScript type check clean
  • Pre-commit hooks pass (biome + tsc)
  • Deploy and run TEST_DEDUP_DEPLOYED=1 bun test test/dataset-id.test.ts for API dedup test
  • Run nemar admin bulk-delete <phantom-ids> to clean up 91 phantom datasets

- Add server-side dedup in POST /datasets: return existing incomplete
  dataset (same owner, name, private, no DOI/versions) instead of
  creating a new one. Prevents phantom datasets when .nemar/config.json
  is lost between retries.
- Cap dataset ID generation at 99999 and exclude nm099999 from the
  gap-filling candidate pool to prevent nm100000+ overflow IDs.
- Update isValidDatasetId() to reject IDs with numeric part > 99999.
- Add POST /admin/datasets/bulk-delete endpoint and CLI command for
  cleaning up phantom datasets (owner-only, safety-checked).
- CLI handles resumed=true flag from backend, skips IAM wait on resume.
- Add unit tests for ID validation including the new upper bound.

Fixes #334
- Fix silent failure: presigned URL errors in dedup path now return 500
  instead of silently continuing with empty URLs
- Fix dedup SQL: require github_repo IS NOT NULL AND != '' to skip
  partially-created datasets with no repo
- Fix audit log: bulk-delete uses standard 5-column schema with
  resource_type/resource_id, log errors instead of empty catch
- Fix bulkDeleteSchema: add regex format validation, deduplicate IDs
  via transform, use zValidator for consistency
- Fix type: CreateDatasetResponse.id is string (not number), resumed
  is always boolean (not optional)
- Extract generateUploadUrlsForFiles() helper to eliminate presigned
  URL generation duplication between dedup and creation paths

Fixes #334
@github-actions

github-actions Bot commented Mar 30, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR335

Version: 0.7.24-PR335.42568b3
Commit: 42568b3


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 33a353d into main Mar 30, 2026
8 checks passed
neuromechanist added a commit that referenced this pull request Apr 17, 2026
Brings to dev from main:
- Maintenance mode middleware (#342)
- MAINTENANCE_MODE=read-only flip (prod only; dev [env.dev.vars] stays off)
- Upload resume fix (#335)
- S3 initremote fix on resume (#337)
- PR-merge workflow, HED detection, bulk CI deploy (#338)
- npm publish fixes (Node 20, npm@11, OIDC)
- GitHub Release auto-tag
@neuromechanist
neuromechanist deleted the 334-fix-dataset-upload-resume-phantom-datasets-and-nm100000-generation branch April 28, 2026 01:23

This branch was previously deployed

1 inactive deployment
npm — 42568b3e Deployed Mar 30, 2026 by neuromechanist via publish #575
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix dataset upload resume: phantom datasets and nm100000 generation

1 participant