Skip to content

Epic #837: Complete separation from legacy nemar.org dataexplorer + DOI re-targeting - #847

Merged
neuromechanist merged 6 commits into
devfrom
feature/issue-837-epic-legacy-separation
Jun 27, 2026
Merged

neuromechanist merged 6 commits into
devfrom
feature/issue-837-epic-legacy-separation

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Two-way separation between the new NEMAR system and the legacy nemar.org/dataexplorer PHP site, plus re-pointing all DOIs to the canonical https://nemar.org/dataset/<id>. Closes #793; fixes #748.

Phases (all reviewed + merged to this epic branch)

  1. Phase 1: DOI landing target -> canonical /dataset URL #842 — DOI landing target → canonical /dataset/<id> URL (shared helpers); the Cloudflare forwarder nemar.org/dataset/* → ww2 is live.
  2. Phase 2: one-off DOI re-target script #844 — scripts/retarget-dois.ts (guarded run-once). Already executed on prod: 1,194 EZID DOIs re-targeted, failed=0.
  3. Phase 3: Kill both legacy nemar.org couplings #843 — removed both legacy couplings: incoming catalog-sync.yml/catalog-sync.ts//admin/catalog/*, and outgoing nemar-sync.ts + all callers (publish step → no-op, webhooks → metadata-only refresh, reindex runDatasetSync→refreshDatasetMetadata, nemar admin sync CLI, endpoints, NEMAR_USERNAME/PASSWORD). nemar-sync.ts split into bids-tree.ts (kept helpers) + legacy-purge.ts (Phase 4 client).
  4. Phase 4: legacy DB purge + ds-row deletion (closes #748) #845 — scripts/purge-legacy-datapipeline.ts + scripts/delete-ds-rows.ts (guarded run-once). Runs are post-deploy ops.
  5. Phase 5: drop dead nemar_sync state + retire sync docs #846 — migration 0053 drops nemar_sync_* columns + catalog_sync_log; removed the dead CLI sync indicator; docs.

Post-deploy operational sequence (after this reaches prod via dev→main)

  1. Re-run retarget-dois.ts --execute (flips DOIs minted during the interim window).
  2. Disable the GitHub "Catalog Sync" workflow.
  3. Run purge-legacy-datapipeline.ts --execute (clear our rows from the legacy DB).
  4. Run delete-ds-rows.ts --execute (delete the ds shadows from D1 + Vectorize → closes Search returns non existent dataset #748).
  5. Verify ww2 search returns no ds.

Cross-repo follow-up: drop the LIKE 'on%' special-case in nemar-observability.

Test plan

  • Each phase: backend + CLI typecheck, biome, unit tests, per-phase /review-pr (findings addressed).
  • Migration 0053 validates on the SCCN dev D1 mirror on this merge, before prod.

* doi: canonical /dataset landing target

Replace legacy nemar.org/dataexplorer/detail?dataset_id=<id> DOI _target
and enrichment links with the canonical https://nemar.org/dataset/<id>
(version DOIs: ?v=v<version>), via two shared helpers in
shared/datacite-constants.ts. Part of epic #837 (Phase 1, #838).

* doi: merge enrich-dataset shared import
* chore: remove legacy nemar.org dataexplorer couplings

Retire both directions of the legacy nemar.org/dataexplorer coupling
(epic #837 Phase 3):

- Incoming (catalog pull): delete catalog-sync.yml, catalog-sync.ts, and
  the POST /admin/catalog/sync + GET /admin/catalog/status endpoints. This
  is what folded the ds shadow rows (#748).
- Outgoing (datapipeline push): delete nemar-sync.ts + nemar-sync-reference.md.
  Split out the 4 BIDS-tree helpers into bids-tree.ts (still used by the
  metadata pipeline) and the slim delete-only client into legacy-purge.ts
  (for the Phase 4 one-time purge).
- Callers: publish sync_nemar step -> logged no-op (kept in allSteps);
  webhook post-version-DOI sync -> refreshDatasetMetadata (keeps the
  metadata-columns refresh, drops the nemar.org push); reindex runDatasetSync
  -> refreshDatasetMetadata; remove POST /admin/datasets/:id/sync, GET
  /admin/sync/status, the nemar admin sync CLI + api methods, and
  NEMAR_USERNAME/NEMAR_PASSWORD bindings.
- Delete 5 legacy-sync test files; relocate helper-test imports to bids-tree.

nemar_sync_* columns + catalog_sync_log table + CF secrets are dropped in
Phase 5. Tested: backend + CLI typecheck, biome, affected unit tests green.

* webhooks: record metadata-refresh error to D1 on throw

Review fixes (PR #843): delete the orphaned backend/test/version-doi-sync-trigger.test.ts (imported the removed shouldSyncToNemarAfterVersionDoi); wrap the 3 post-version-DOI refresh sites in refreshMetadataAfterVersionDoi so a pre-refresh throw (transient GitHub failure) records metadata_columns_error to D1 instead of only logging.
* scripts: one-off DOI re-target to canonical URL

Epic #837 Phase 2. Guarded run-once ops script (NOT a CLI/endpoint) that
re-points all NEMAR EZID DOI _targets from the legacy dataexplorer URL to the
canonical https://nemar.org/dataset/<id> (forwarded to ww2). Dry-run by
default; --execute requires EZID creds; idempotent/resumable (skips DOIs
already on the canonical target); --only canaries a single dataset. Reuses the
real ezid.ts client. Dry-run validated against prod D1: 1,194 DOIs
(590 concept + 604 version). The execute run is a gated prod op.

* retarget: use project isRetryable (no dataset-id false-match)
Epic #837 Phase 4 tooling (guarded run-once scripts, not CLI/endpoints):
- purge-legacy-datapipeline.ts: delete our nm/on records from the 4 legacy
  dataexplorer tables via the extracted legacy-purge.ts client (dry-run default,
  --execute needs NEMAR_USERNAME/PASSWORD, idempotent). Dry-run: 608 datasets.
- delete-ds-rows.ts: delete the ds shadow rows from D1 (children-first, datasets
  last -> FTS/FK cascade) + Vectorize, with a pre-delete snapshot. Closes #748.
  Dry-run confirms ds rows live ONLY in datasets (173), 0 in dependent tables.
  PRECONDITION: catalog-sync cron must be OFF first (else 4h re-fold).
The --execute runs are gated prod ops.
* cleanup: drop dead nemar_sync state + retire sync docs

Epic #837 Phase 5. Remove the now-dead legacy-sync remnants:
- migration 0053 drops the nemar_sync_status/at/error columns (0014) and the
  catalog_sync_log table (0018); no index/trigger/FTS references them.
- remove the two dead nemar_sync_status reads from the GET /datasets queries.
- AGENTS.md: retire the nemar admin sync command + datapipeline-sync docs;
  describe the legacy site as being retired and DOIs landing on /dataset.
Keeps the (nm|on|ds) exact-id regex (ds bookmarks resolve to their on mirror via
source_id) and the legacy-purge.ts client + one-off scripts (needed for the
post-deploy purge runs). Cross-repo follow-up: nemar-observability LIKE 'on%'.

* cli: drop nemar_sync_status list indicator (now dead)

Review fix (PR #846): the 0053 column drop + SELECT removal made
dataset.nemar_sync_status always undefined, so the 'nemar dataset list' status
indicator would have shown a yellow * on every managed dataset. Remove the
indicator + the Dataset.nemar_sync_status field + the stale test-fixture column.
Cross-phase epic review (#847): seedFromBids merged existing
related_identifiers (which may carry the old nemar.org/dataexplorer/detail
IsDescribedBy URL) and appended the canonical /dataset/ URL without dropping
the old one, so re-enrichment accumulated BOTH. Filter out the legacy
dataexplorer landing URL so re-enrichment truly self-heals to the canonical
one; update the LLM validator prompt to vouch for nemar.org/dataset/...; add a
regression test.
@github-actions

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR847

Version: 0.8.70-PR847.4610711
Commit: 4610711


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 83f84e3 into dev Jun 27, 2026
8 of 10 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-837-epic-legacy-separation branch June 27, 2026 04:32

This branch was previously deployed

1 inactive deployment
npm — 46107117 Deployed Jun 27, 2026 by neuromechanist via publish #1591
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant