Repository navigation
Fix dataset upload resume: server-side dedup, ID cap, bulk delete - #335
Merged
neuromechanist merged 3 commits intoMar 30, 2026
Conversation
- Add server-side dedup in POST /datasets: return existing incomplete dataset (same owner, name, private, no DOI/versions) instead of creating a new one. Prevents phantom datasets when .nemar/config.json is lost between retries. - Cap dataset ID generation at 99999 and exclude nm099999 from the gap-filling candidate pool to prevent nm100000+ overflow IDs. - Update isValidDatasetId() to reject IDs with numeric part > 99999. - Add POST /admin/datasets/bulk-delete endpoint and CLI command for cleaning up phantom datasets (owner-only, safety-checked). - CLI handles resumed=true flag from backend, skips IAM wait on resume. - Add unit tests for ID validation including the new upper bound. Fixes #334
- Fix silent failure: presigned URL errors in dedup path now return 500 instead of silently continuing with empty URLs - Fix dedup SQL: require github_repo IS NOT NULL AND != '' to skip partially-created datasets with no repo - Fix audit log: bulk-delete uses standard 5-column schema with resource_type/resource_id, log errors instead of empty catch - Fix bulkDeleteSchema: add regex format validation, deduplicate IDs via transform, use zValidator for consistency - Fix type: CreateDatasetResponse.id is string (not number), resumed is always boolean (not optional) - Extract generateUploadUrlsForFiles() helper to eliminate presigned URL generation duplication between dedup and creation paths Fixes #334
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR335Version: 🤖 Updated on each push. |
neuromechanist
added a commit
that referenced
this pull request
Apr 17, 2026
Brings to dev from main: - Maintenance mode middleware (#342) - MAINTENANCE_MODE=read-only flip (prod only; dev [env.dev.vars] stays off) - Upload resume fix (#335) - S3 initremote fix on resume (#337) - PR-merge workflow, HED detection, bulk CI deploy (#338) - npm publish fixes (Node 20, npm@11, OIDC) - GitHub Release auto-tag
1 of 3 tasks
neuromechanist
deleted the
334-fix-dataset-upload-resume-phantom-datasets-and-nm100000-generation
branch
April 28, 2026 01:23
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #334. User Bruno created ~80+ phantom datasets (nm000136-nm000337, nm100000-nm100001) because every failed upload retry created a new dataset instead of resuming.
POST /datasetsnow checks for existing incomplete datasets (same owner, name, private, no DOI, no versions) before creating new ones. Returns the existing dataset withresumed: true.MAX_NUMBER = 99999and excludesnm099999(test dataset) from candidates. Prevents overflow IDs likenm100000.isValidDatasetId()hardened: Rejects IDs with numeric part > 99999.POST /admin/datasets/bulk-delete+nemar admin bulk-deleteCLI command for cleaning up phantom datasets (owner-only, rejects public/DOI datasets).resumedflag, shows appropriate message, skips IAM propagation wait.Changed files
backend/src/services/datasetId.ts- MAX_NUMBER cap, EXCLUDED_IDS, validation fixbackend/src/routes/datasets.ts- Server-side dedup query in POST /datasetsbackend/src/routes/admin.ts- Bulk delete endpointsrc/commands/dataset.ts- Handle resumed flag in CLIsrc/commands/admin.ts- bulk-delete CLI commandsrc/lib/api.ts- CreateDatasetResponse.resumed, bulkDeleteDatasets()test/dataset-id.test.ts- Unit tests for ID validationTest plan
isValidDatasetIdrejects nm100000+, accepts valid IDs (5/5 pass)TEST_DEDUP_DEPLOYED=1 bun test test/dataset-id.test.tsfor API dedup testnemar admin bulk-delete <phantom-ids>to clean up 91 phantom datasets