Skip to content

Epic: Complete separation from legacy nemar.org dataexplorer + DOI re-targeting #837

Closed
6 of 6 issues completed

Description

@neuromechanist

Goal

Two-way separation between the new NEMAR system (api.nemar.org / ww2.nemar.org / D1) and the legacy nemar.org/dataexplorer PHP site, plus re-point all DOIs to the canonical https://nemar.org/dataset/<id> (forwarded to ww2).

End state:

  • New system no longer indexes legacy ds###### datasets (the broken Search returns non existent dataset #748 search hits). Every source='openneuro' row is on######.
  • New system no longer pulls FROM or pushes TO the legacy dataexplorer DB.
  • Legacy dataexplorer DB no longer holds our nm/on records.
  • All DOIs (new + existing ~1,182 EZID) point at https://nemar.org/dataset/<id> (version: ?v=v<version>), forwarded to ww2.

Scope is the nemar.org dataexplorer coupling ONLY. SDSC Hallu sync, S3, Zarr, QA pipelines stay intact.

Full technical plan: .context/plan-legacy-separation-793.md.

Ground truth (prod D1, 2026-06-26)

  • on/openneuro=401, nm=197, ds/openneuro=183 (all ds are owner=-1 system shadows, 0 with an on mirror).
  • 181 rows pushed into legacy (nemar_sync_status='synced').
  • 584 concept + 598 version EZID DOIs to re-target; 0 Zenodo.

Two legacy couplings

  • Incoming (pull): catalog-sync.yml (4h cron) -> /admin/catalog/sync -> catalog-sync.ts. Regenerates ds shadows.
  • Outgoing (push): nemar-sync.ts + callers (publish sync_nemar step, version-DOI webhooks x3, reindex, CLI admin sync, endpoints).

Phases

  1. DOI target builder -> canonical /dataset/<id> URL + Cloudflare forwarder on the nemar.org zone.
  2. Bulk re-target ~1,182 existing EZID DOIs (gated, dry-run, canary nm000132).
  3. Kill both legacy couplings in code (incoming + outgoing), extract slim legacy-purge.ts.
  4. One-time data purge: our nm/on rows out of the 4 legacy tables + delete 183 ds shadows (D1 + Vectorize). Closes Search returns non existent dataset #748.
  5. Retire ds code paths + cleanup (regexes, secrets, nemar_sync_* columns, catalog_sync_log, guard test, docs, cross-repo observability follow-up).

Ordering spine (must hold)

  1. Forwarder live + verified BEFORE any DOI flips.
  2. DOIs re-targeted off dataexplorer/detail BEFORE purging our rows from the legacy DB.
  3. Incoming catalog-sync dead BEFORE deleting ds rows.

Every prod mutation (DOI bulk, legacy purge, ds delete) is gated on explicit owner go-ahead.

Subsumes #793; fixes #748.

Activity

  1. neuromechanist commented on Jun 27, 2026

    @neuromechanist
    ContributorAuthor

    Complete separation from legacy nemar.org dataexplorer shipped in v0.8.70 (PR #848). Forwarder live; ~1,210 DOIs canonical; legacy couplings removed; legacy DB purged (615/615); ds shadows deleted (D1+Vectorize) -> #748 fixed; migrations 0050-0053 applied; one-off tooling removed (PR #850). Cross-repo follow-up: nemar-observability LIKE 'on%'.

  2. added a commit that references this issue on Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions