Repository navigation
Description
Problem
User Bruno made many upload attempts for a MOABB dataset, creating ~80+ phantom datasets (nm000136-nm000337 range plus nm100000, nm100001). The resume functionality failed to detect existing incomplete uploads, causing a new dataset + GitHub repo to be created on every retry.
Root Causes
1. Resume is entirely client-side — no server-side deduplication
The resume mechanism relies on .nemar/config.json in the local dataset directory. When this file is missing, deleted, or corrupted, the CLI calls POST /datasets which always creates a new dataset — no check for existing incomplete uploads by the same user with the same name.
Common scenarios where resume fails:
- Process crashes before
writeLocalConfig()completes (dataset created on backend but config not saved locally) - User deletes
.nemar/directory (as suggested by the 404 error message) - Config file gets corrupted
- User retries from a different terminal/path
Each failed attempt creates an orphaned dataset record in D1 + an empty GitHub repo in nemarDatasets.
Location: src/commands/dataset.ts:912-1077 (resume detection), backend/src/routes/datasets.ts:90-310 (creation with no dedup)
2. nm100000 generation — no upper bound on dataset IDs
The test dataset nm099999 contributes candidate 100000 to the gap-filling algorithm (99999 + 1 = 100000). The validation regex /^(nm|xx|on)\d{6}$/ allows this since 100000 is exactly 6 digits. There is no upper bound cap.
When enough phantom datasets fill all gaps below the highest allocated ID, the algorithm picks 100000 from the nm099999 candidate — producing IDs like nm100000 and nm100001 that break the expected nm000XXX zero-padded convention.
Location: backend/src/services/datasetId.ts:19-57 (gap algorithm), line 92-94 (validation regex)
Proposed Fixes
Fix 1: Server-side incomplete dataset detection
Before creating a new dataset in POST /datasets, check if the user already has an incomplete (non-finalized) dataset with the same name. If found, return the existing dataset info instead of creating a new one — enabling server-side resume.
Fix 2: Cap dataset ID generation at 99999
Add MAX_NUMBER = 99999 to the gap-filling algorithm. If all IDs up to 99999 are exhausted, return an error instead of generating nm100000+.
Fix 3: Exclude test dataset from candidate pool
Filter out nm099999 from the gap-filling query so it doesn't contribute 100000 as a candidate.
Fix 4: Client-side fallback resume via API
When .nemar/config.json is missing, query GET /datasets?mine=true&search=<name> to find an existing incomplete dataset before creating a new one.
Affected Phantom Datasets
nm000153 nm000154 nm000155 nm000238 nm000251 nm000252 nm000253 nm000256
nm000257 nm000258 nm000261 nm000262 nm000263 nm000273 nm000274 nm000275
nm000276 nm000278 nm000279 nm000280 nm000281 nm000285 nm000286 nm000287
nm000288 nm000289 nm000290 nm000291 nm000292 nm000293 nm000294 nm000295
nm000296 nm000297 nm000298 nm000299 nm000300 nm000304 nm000305 nm000306
nm000307 nm000312 nm000314 nm000315 nm000316 nm000317 nm000319 nm000320
nm000324 nm000325 nm000327 nm000328 nm000330 nm000331 nm000332 nm000335
nm100000 nm100001 nm000147 nm000150 nm000157 nm000174 nm000175 nm000177
nm000178 nm000179 nm000180 nm000228 nm000284 nm000181 nm000232 nm000182
nm000233 nm000183 nm000184 nm000185 nm000229 nm000241 nm000254 nm000282
nm000255 nm000283 nm000302 nm000308 nm000159 nm000333 nm000220 nm000226
nm000269 nm000309 nm000136 nm000318 nm000322 nm000337
Activity
Fix dataset upload resume: server-side dedup, ID cap, bulk delete
Address PR review findings
Fix dataset upload resume: server-side dedup, ID cap, bulk delete (#335)