Skip to content

Phase 4: Delete legacy ds shadows + purge our records from legacy dataexplorer DB #793

Closed

Description

@neuromechanist

Problem

NEMAR's datasets table holds OpenNeuro datasets indexed two ways:

  • on###### (NEMAR-native): the current/correct scheme. Imported via the
    import pipeline (mapToNemarId ds -> on). 75 today.
  • ds###### (legacy): the old-NEMAR scheme that indexed OpenNeuro datasets
    directly by their OpenNeuro ds id. 461 today.

(536 total source='openneuro' = 75 on + 461 ds, as of 2026-06-18.)

The ds-numbered rows are legacy and should be migrated to on numbers and the
old ds indexing retired. Until then, counts/UX that mean "datasets NEMAR imported"
must distinguish the two (the observability "Imported" tile was just changed to
count only on-numbered rows; see nemar-observability).

Why it matters

  • The true count of properly-imported OpenNeuro datasets is the on set (75), not
    the 536 mixed total.
  • Dedup (getImportedSourceIds) correctly excludes BOTH on + ds by source_id, so
    auto-import won't create duplicate on rows for a dataset already present as a
    legacy ds row -- but those 461 will never get an on record until migrated.

Proposed work

  • Inventory the 461 ds-numbered source='openneuro' rows (S3 layout, repos,
    DOIs, public/private, zarr/archive state).
  • Decide migration strategy: re-key ds###### -> on###### (new NEMAR id +
    source_id=ds######) vs. re-import vs. alias.
  • Migrate, preserving DOIs / S3 objects / GitHub repos / collaborators.
  • Retire the ds-id code paths; assert all source='openneuro' rows are on%.
  • Once migration is complete, drop the dataset_id LIKE 'on%' special-case in
    the observability count (back to plain source='openneuro').

Surfaced while fixing the observability "Imported" tile (it was counting
import_jobs complete, then the mixed 536; the true number is the 75 on rows).

Activity

  1. neuromechanist commented on Jun 18, 2026

    @neuromechanist
    ContributorAuthor

    Owner direction (2026-06-18): the ds-numbered rows will be completely removed and not indexed in any shape once the OpenNeuro re-import completes -- expected in ~10-20 days. So the end state is: every source='openneuro' row is on-numbered, and the dataset_id LIKE 'on%' special-case in the observability count can be dropped.

    Note on the auto-import dedup interaction: getImportedSourceIds currently excludes BOTH on + ds rows (by source_id), so the live auto-import engine imports only net-new datasets and will NOT itself re-key the 461 legacy ds rows. The migration to drive them to on (re-key in D1, or re-import + delete the ds rows) is this issue's work; confirm the mechanism before it runs (re-key preserves DOIs/S3/repos; re-import would create new ones).

  2. added
    P1High priority - important features, significant bugs
    infrastructureInfrastructure and deployment
    backendBackend/API related
    on Jun 18, 2026
  3. neuromechanist commented on Jun 26, 2026

    @neuromechanist
    ContributorAuthor

    Adopted as Phase 4 of epic #837 (legacy nemar.org separation). This issue's ds-retirement (delete the 183 ds shadow rows + de-index) is the Phase 4 data purge; the prerequisite (kill the incoming 4h catalog-sync that regenerates them) is Phase 3. Fixes #748.

  4. changed the title [-]Retire legacy ds-numbered OpenNeuro datasets (migrate to on-numbers)[/-] [+]Phase 4: Delete legacy ds shadows + purge our records from legacy dataexplorer DB[/+] on Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priority - important features, significant bugsbackendBackend/API relatedinfrastructureInfrastructure and deployment

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions