You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Phase 4: Delete legacy ds shadows + purge our records from legacy dataexplorer DB #793
NEMAR's datasets table holds OpenNeuro datasets indexed two ways:
on###### (NEMAR-native): the current/correct scheme. Imported via the
import pipeline (mapToNemarId ds -> on). 75 today.
ds###### (legacy): the old-NEMAR scheme that indexed OpenNeuro datasets
directly by their OpenNeuro ds id. 461 today.
(536 total source='openneuro' = 75 on + 461 ds, as of 2026-06-18.)
The ds-numbered rows are legacy and should be migrated to on numbers and the
old ds indexing retired. Until then, counts/UX that mean "datasets NEMAR imported"
must distinguish the two (the observability "Imported" tile was just changed to
count only on-numbered rows; see nemar-observability).
Why it matters
The true count of properly-imported OpenNeuro datasets is the on set (75), not
the 536 mixed total.
Dedup (getImportedSourceIds) correctly excludes BOTH on + ds by source_id, so
auto-import won't create duplicate on rows for a dataset already present as a
legacy ds row -- but those 461 will never get an on record until migrated.
Owner direction (2026-06-18): the ds-numbered rows will be completely removed and not indexed in any shape once the OpenNeuro re-import completes -- expected in ~10-20 days. So the end state is: every source='openneuro' row is on-numbered, and the dataset_id LIKE 'on%' special-case in the observability count can be dropped.
Note on the auto-import dedup interaction: getImportedSourceIds currently excludes BOTH on + ds rows (by source_id), so the live auto-import engine imports only net-new datasets and will NOT itself re-key the 461 legacy ds rows. The migration to drive them to on (re-key in D1, or re-import + delete the ds rows) is this issue's work; confirm the mechanism before it runs (re-key preserves DOIs/S3/repos; re-import would create new ones).
Adopted as Phase 4 of epic #837 (legacy nemar.org separation). This issue's ds-retirement (delete the 183 ds shadow rows + de-index) is the Phase 4 data purge; the prerequisite (kill the incoming 4h catalog-sync that regenerates them) is Phase 3. Fixes #748.
changed the title [-]Retire legacy ds-numbered OpenNeuro datasets (migrate to on-numbers)[/-][+]Phase 4: Delete legacy ds shadows + purge our records from legacy dataexplorer DB[/+]on Jun 26, 2026
Activity
neuromechanist commentedon Jun 18, 2026
Owner direction (2026-06-18): the
ds-numbered rows will be completely removed and not indexed in any shape once the OpenNeuro re-import completes -- expected in ~10-20 days. So the end state is: everysource='openneuro'row ison-numbered, and thedataset_id LIKE 'on%'special-case in the observability count can be dropped.Note on the auto-import dedup interaction:
getImportedSourceIdscurrently excludes BOTH on + ds rows (bysource_id), so the live auto-import engine imports only net-new datasets and will NOT itself re-key the 461 legacydsrows. The migration to drive them toon(re-key in D1, or re-import + delete thedsrows) is this issue's work; confirm the mechanism before it runs (re-key preserves DOIs/S3/repos; re-import would create new ones).neuromechanist commentedon Jun 26, 2026
Adopted as Phase 4 of epic #837 (legacy nemar.org separation). This issue's ds-retirement (delete the 183
dsshadow rows + de-index) is the Phase 4 data purge; the prerequisite (kill the incoming 4h catalog-sync that regenerates them) is Phase 3. Fixes #748.[-]Retire legacy ds-numbered OpenNeuro datasets (migrate to on-numbers)[/-][+]Phase 4: Delete legacy ds shadows + purge our records from legacy dataexplorer DB[/+]