Repository navigation
Epic #837: Complete separation from legacy nemar.org dataexplorer + DOI re-targeting - #847
Merged
Merged
Conversation
* doi: canonical /dataset landing target Replace legacy nemar.org/dataexplorer/detail?dataset_id=<id> DOI _target and enrichment links with the canonical https://nemar.org/dataset/<id> (version DOIs: ?v=v<version>), via two shared helpers in shared/datacite-constants.ts. Part of epic #837 (Phase 1, #838). * doi: merge enrich-dataset shared import
* chore: remove legacy nemar.org dataexplorer couplings Retire both directions of the legacy nemar.org/dataexplorer coupling (epic #837 Phase 3): - Incoming (catalog pull): delete catalog-sync.yml, catalog-sync.ts, and the POST /admin/catalog/sync + GET /admin/catalog/status endpoints. This is what folded the ds shadow rows (#748). - Outgoing (datapipeline push): delete nemar-sync.ts + nemar-sync-reference.md. Split out the 4 BIDS-tree helpers into bids-tree.ts (still used by the metadata pipeline) and the slim delete-only client into legacy-purge.ts (for the Phase 4 one-time purge). - Callers: publish sync_nemar step -> logged no-op (kept in allSteps); webhook post-version-DOI sync -> refreshDatasetMetadata (keeps the metadata-columns refresh, drops the nemar.org push); reindex runDatasetSync -> refreshDatasetMetadata; remove POST /admin/datasets/:id/sync, GET /admin/sync/status, the nemar admin sync CLI + api methods, and NEMAR_USERNAME/NEMAR_PASSWORD bindings. - Delete 5 legacy-sync test files; relocate helper-test imports to bids-tree. nemar_sync_* columns + catalog_sync_log table + CF secrets are dropped in Phase 5. Tested: backend + CLI typecheck, biome, affected unit tests green. * webhooks: record metadata-refresh error to D1 on throw Review fixes (PR #843): delete the orphaned backend/test/version-doi-sync-trigger.test.ts (imported the removed shouldSyncToNemarAfterVersionDoi); wrap the 3 post-version-DOI refresh sites in refreshMetadataAfterVersionDoi so a pre-refresh throw (transient GitHub failure) records metadata_columns_error to D1 instead of only logging.
* scripts: one-off DOI re-target to canonical URL Epic #837 Phase 2. Guarded run-once ops script (NOT a CLI/endpoint) that re-points all NEMAR EZID DOI _targets from the legacy dataexplorer URL to the canonical https://nemar.org/dataset/<id> (forwarded to ww2). Dry-run by default; --execute requires EZID creds; idempotent/resumable (skips DOIs already on the canonical target); --only canaries a single dataset. Reuses the real ezid.ts client. Dry-run validated against prod D1: 1,194 DOIs (590 concept + 604 version). The execute run is a gated prod op. * retarget: use project isRetryable (no dataset-id false-match)
Epic #837 Phase 4 tooling (guarded run-once scripts, not CLI/endpoints): - purge-legacy-datapipeline.ts: delete our nm/on records from the 4 legacy dataexplorer tables via the extracted legacy-purge.ts client (dry-run default, --execute needs NEMAR_USERNAME/PASSWORD, idempotent). Dry-run: 608 datasets. - delete-ds-rows.ts: delete the ds shadow rows from D1 (children-first, datasets last -> FTS/FK cascade) + Vectorize, with a pre-delete snapshot. Closes #748. Dry-run confirms ds rows live ONLY in datasets (173), 0 in dependent tables. PRECONDITION: catalog-sync cron must be OFF first (else 4h re-fold). The --execute runs are gated prod ops.
* cleanup: drop dead nemar_sync state + retire sync docs Epic #837 Phase 5. Remove the now-dead legacy-sync remnants: - migration 0053 drops the nemar_sync_status/at/error columns (0014) and the catalog_sync_log table (0018); no index/trigger/FTS references them. - remove the two dead nemar_sync_status reads from the GET /datasets queries. - AGENTS.md: retire the nemar admin sync command + datapipeline-sync docs; describe the legacy site as being retired and DOIs landing on /dataset. Keeps the (nm|on|ds) exact-id regex (ds bookmarks resolve to their on mirror via source_id) and the legacy-purge.ts client + one-off scripts (needed for the post-deploy purge runs). Cross-repo follow-up: nemar-observability LIKE 'on%'. * cli: drop nemar_sync_status list indicator (now dead) Review fix (PR #846): the 0053 column drop + SELECT removal made dataset.nemar_sync_status always undefined, so the 'nemar dataset list' status indicator would have shown a yellow * on every managed dataset. Remove the indicator + the Dataset.nemar_sync_status field + the stale test-fixture column.
Cross-phase epic review (#847): seedFromBids merged existing related_identifiers (which may carry the old nemar.org/dataexplorer/detail IsDescribedBy URL) and appended the canonical /dataset/ URL without dropping the old one, so re-enrichment accumulated BOTH. Filter out the legacy dataexplorer landing URL so re-enrichment truly self-heals to the canonical one; update the LLM validator prompt to vouch for nemar.org/dataset/...; add a regression test.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR847Version: 🤖 Updated on each push. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two-way separation between the new NEMAR system and the legacy
nemar.org/dataexplorerPHP site, plus re-pointing all DOIs to the canonicalhttps://nemar.org/dataset/<id>. Closes #793; fixes #748.Phases (all reviewed + merged to this epic branch)
/dataset/<id>URL (shared helpers); the Cloudflare forwardernemar.org/dataset/* → ww2is live.scripts/retarget-dois.ts(guarded run-once). Already executed on prod: 1,194 EZID DOIs re-targeted, failed=0.catalog-sync.yml/catalog-sync.ts//admin/catalog/*, and outgoingnemar-sync.ts+ all callers (publish step → no-op, webhooks → metadata-only refresh, reindexrunDatasetSync→refreshDatasetMetadata,nemar admin syncCLI, endpoints,NEMAR_USERNAME/PASSWORD).nemar-sync.tssplit intobids-tree.ts(kept helpers) +legacy-purge.ts(Phase 4 client).scripts/purge-legacy-datapipeline.ts+scripts/delete-ds-rows.ts(guarded run-once). Runs are post-deploy ops.nemar_sync_*columns +catalog_sync_log; removed the dead CLI sync indicator; docs.Post-deploy operational sequence (after this reaches prod via dev→main)
retarget-dois.ts --execute(flips DOIs minted during the interim window).purge-legacy-datapipeline.ts --execute(clear our rows from the legacy DB).delete-ds-rows.ts --execute(delete thedsshadows from D1 + Vectorize → closes Search returns non existent dataset #748).ds.Cross-repo follow-up: drop the
LIKE 'on%'special-case innemar-observability.Test plan
/review-pr(findings addressed).