Skip to content

Fix dataset upload resume: phantom datasets and nm100000 generation #334

Closed

Description

@neuromechanist

Problem

User Bruno made many upload attempts for a MOABB dataset, creating ~80+ phantom datasets (nm000136-nm000337 range plus nm100000, nm100001). The resume functionality failed to detect existing incomplete uploads, causing a new dataset + GitHub repo to be created on every retry.

Root Causes

1. Resume is entirely client-side — no server-side deduplication

The resume mechanism relies on .nemar/config.json in the local dataset directory. When this file is missing, deleted, or corrupted, the CLI calls POST /datasets which always creates a new dataset — no check for existing incomplete uploads by the same user with the same name.

Common scenarios where resume fails:

  • Process crashes before writeLocalConfig() completes (dataset created on backend but config not saved locally)
  • User deletes .nemar/ directory (as suggested by the 404 error message)
  • Config file gets corrupted
  • User retries from a different terminal/path

Each failed attempt creates an orphaned dataset record in D1 + an empty GitHub repo in nemarDatasets.

Location: src/commands/dataset.ts:912-1077 (resume detection), backend/src/routes/datasets.ts:90-310 (creation with no dedup)

2. nm100000 generation — no upper bound on dataset IDs

The test dataset nm099999 contributes candidate 100000 to the gap-filling algorithm (99999 + 1 = 100000). The validation regex /^(nm|xx|on)\d{6}$/ allows this since 100000 is exactly 6 digits. There is no upper bound cap.

When enough phantom datasets fill all gaps below the highest allocated ID, the algorithm picks 100000 from the nm099999 candidate — producing IDs like nm100000 and nm100001 that break the expected nm000XXX zero-padded convention.

Location: backend/src/services/datasetId.ts:19-57 (gap algorithm), line 92-94 (validation regex)

Proposed Fixes

Fix 1: Server-side incomplete dataset detection

Before creating a new dataset in POST /datasets, check if the user already has an incomplete (non-finalized) dataset with the same name. If found, return the existing dataset info instead of creating a new one — enabling server-side resume.

Fix 2: Cap dataset ID generation at 99999

Add MAX_NUMBER = 99999 to the gap-filling algorithm. If all IDs up to 99999 are exhausted, return an error instead of generating nm100000+.

Fix 3: Exclude test dataset from candidate pool

Filter out nm099999 from the gap-filling query so it doesn't contribute 100000 as a candidate.

Fix 4: Client-side fallback resume via API

When .nemar/config.json is missing, query GET /datasets?mine=true&search=<name> to find an existing incomplete dataset before creating a new one.

Affected Phantom Datasets

nm000153 nm000154 nm000155 nm000238 nm000251 nm000252 nm000253 nm000256
nm000257 nm000258 nm000261 nm000262 nm000263 nm000273 nm000274 nm000275
nm000276 nm000278 nm000279 nm000280 nm000281 nm000285 nm000286 nm000287
nm000288 nm000289 nm000290 nm000291 nm000292 nm000293 nm000294 nm000295
nm000296 nm000297 nm000298 nm000299 nm000300 nm000304 nm000305 nm000306
nm000307 nm000312 nm000314 nm000315 nm000316 nm000317 nm000319 nm000320
nm000324 nm000325 nm000327 nm000328 nm000330 nm000331 nm000332 nm000335
nm100000 nm100001 nm000147 nm000150 nm000157 nm000174 nm000175 nm000177
nm000178 nm000179 nm000180 nm000228 nm000284 nm000181 nm000232 nm000182
nm000233 nm000183 nm000184 nm000185 nm000229 nm000241 nm000254 nm000282
nm000255 nm000283 nm000302 nm000308 nm000159 nm000333 nm000220 nm000226
nm000269 nm000309 nm000136 nm000318 nm000322 nm000337

Activity

  1. added a commit that references this issue on Mar 30, 2026
    45db761
  2. added a commit that references this issue on Mar 30, 2026
    8a54ef8
  3. added a commit that references this issue on Apr 17, 2026
    33a353d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions