Skip to content

Release 0.8.70: legacy nemar.org separation + DOI re-targeting, ORCID SSO, prescreen fix - #848

Merged
neuromechanist merged 34 commits into
mainfrom
dev
Jun 27, 2026
Merged

neuromechanist merged 34 commits into
mainfrom
dev

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Release 0.8.70 (0.8.69 → 0.8.70)

Three workstreams on dev:

1. Legacy nemar.org dataexplorer separation + DOI re-targeting (epic #837, closes #793, fixes #748)

  • DOI landing target → canonical https://nemar.org/dataset/<id> (forwarder to ww2 live; 1,194 DOIs already re-targeted on prod).
  • Removed both legacy couplings: incoming catalog-sync and outgoing nemar-sync + all callers.
  • One-off guarded scripts for the post-deploy purge: legacy-DB purge + ds-shadow deletion (run after this deploys).
  • Migration 0053 drops the dead nemar_sync_* columns + catalog_sync_log (validated on the dev D1 mirror).

2. ORCID SSO (#835)

  • ORCID OAuth routes + service; web signups require ORCID + city/country (export-control screening). Migrations 0050–0052.

3. #825 — prescreen callback accepts null issue_url

Post-deploy operational sequence (epic #837)

After this deploys to prod: re-run retarget-dois.ts --execute (interim DOIs) → disable the "Catalog Sync" workflow → purge-legacy-datapipeline.ts --execute → delete-ds-rows.ts --execute → verify ww2 search has no ds.

CI / test notes

  • Required checks (lint, unit-pure) green.
  • api-test red is pre-existing + unrelated: the ORCID feature made signup require city/country/ORCID, so api.test.ts's field-less signup tests get 400 before the duplicate-409. Soft live-backend tier; not introduced by the epic. (Worth a follow-up to update api.test.ts for the new signup contract.)

Merge

Regular merge (NOT squash) per repo convention. Triggers auto-tag v0.8.70 → npm publish → prod deploy + migrations → sync-dev.

nemar-bot and others added 30 commits June 21, 2026 20:09
feat: ORCID SSO backend (migration 0050 + /auth/orcid/* routes)
feat: ORCID-canonical name + profile fields (city/country, affiliation) — #835
* doi: canonical /dataset landing target

Replace legacy nemar.org/dataexplorer/detail?dataset_id=<id> DOI _target
and enrichment links with the canonical https://nemar.org/dataset/<id>
(version DOIs: ?v=v<version>), via two shared helpers in
shared/datacite-constants.ts. Part of epic #837 (Phase 1, #838).

* doi: merge enrich-dataset shared import
* chore: remove legacy nemar.org dataexplorer couplings

Retire both directions of the legacy nemar.org/dataexplorer coupling
(epic #837 Phase 3):

- Incoming (catalog pull): delete catalog-sync.yml, catalog-sync.ts, and
  the POST /admin/catalog/sync + GET /admin/catalog/status endpoints. This
  is what folded the ds shadow rows (#748).
- Outgoing (datapipeline push): delete nemar-sync.ts + nemar-sync-reference.md.
  Split out the 4 BIDS-tree helpers into bids-tree.ts (still used by the
  metadata pipeline) and the slim delete-only client into legacy-purge.ts
  (for the Phase 4 one-time purge).
- Callers: publish sync_nemar step -> logged no-op (kept in allSteps);
  webhook post-version-DOI sync -> refreshDatasetMetadata (keeps the
  metadata-columns refresh, drops the nemar.org push); reindex runDatasetSync
  -> refreshDatasetMetadata; remove POST /admin/datasets/:id/sync, GET
  /admin/sync/status, the nemar admin sync CLI + api methods, and
  NEMAR_USERNAME/NEMAR_PASSWORD bindings.
- Delete 5 legacy-sync test files; relocate helper-test imports to bids-tree.

nemar_sync_* columns + catalog_sync_log table + CF secrets are dropped in
Phase 5. Tested: backend + CLI typecheck, biome, affected unit tests green.

* webhooks: record metadata-refresh error to D1 on throw

Review fixes (PR #843): delete the orphaned backend/test/version-doi-sync-trigger.test.ts (imported the removed shouldSyncToNemarAfterVersionDoi); wrap the 3 post-version-DOI refresh sites in refreshMetadataAfterVersionDoi so a pre-refresh throw (transient GitHub failure) records metadata_columns_error to D1 instead of only logging.
* scripts: one-off DOI re-target to canonical URL

Epic #837 Phase 2. Guarded run-once ops script (NOT a CLI/endpoint) that
re-points all NEMAR EZID DOI _targets from the legacy dataexplorer URL to the
canonical https://nemar.org/dataset/<id> (forwarded to ww2). Dry-run by
default; --execute requires EZID creds; idempotent/resumable (skips DOIs
already on the canonical target); --only canaries a single dataset. Reuses the
real ezid.ts client. Dry-run validated against prod D1: 1,194 DOIs
(590 concept + 604 version). The execute run is a gated prod op.

* retarget: use project isRetryable (no dataset-id false-match)
Epic #837 Phase 4 tooling (guarded run-once scripts, not CLI/endpoints):
- purge-legacy-datapipeline.ts: delete our nm/on records from the 4 legacy
  dataexplorer tables via the extracted legacy-purge.ts client (dry-run default,
  --execute needs NEMAR_USERNAME/PASSWORD, idempotent). Dry-run: 608 datasets.
- delete-ds-rows.ts: delete the ds shadow rows from D1 (children-first, datasets
  last -> FTS/FK cascade) + Vectorize, with a pre-delete snapshot. Closes #748.
  Dry-run confirms ds rows live ONLY in datasets (173), 0 in dependent tables.
  PRECONDITION: catalog-sync cron must be OFF first (else 4h re-fold).
The --execute runs are gated prod ops.
* cleanup: drop dead nemar_sync state + retire sync docs

Epic #837 Phase 5. Remove the now-dead legacy-sync remnants:
- migration 0053 drops the nemar_sync_status/at/error columns (0014) and the
  catalog_sync_log table (0018); no index/trigger/FTS references them.
- remove the two dead nemar_sync_status reads from the GET /datasets queries.
- AGENTS.md: retire the nemar admin sync command + datapipeline-sync docs;
  describe the legacy site as being retired and DOIs landing on /dataset.
Keeps the (nm|on|ds) exact-id regex (ds bookmarks resolve to their on mirror via
source_id) and the legacy-purge.ts client + one-off scripts (needed for the
post-deploy purge runs). Cross-repo follow-up: nemar-observability LIKE 'on%'.

* cli: drop nemar_sync_status list indicator (now dead)

Review fix (PR #846): the 0053 column drop + SELECT removal made
dataset.nemar_sync_status always undefined, so the 'nemar dataset list' status
indicator would have shown a yellow * on every managed dataset. Remove the
indicator + the Dataset.nemar_sync_status field + the stale test-fixture column.
Cross-phase epic review (#847): seedFromBids merged existing
related_identifiers (which may carry the old nemar.org/dataexplorer/detail
IsDescribedBy URL) and appended the canonical /dataset/ URL without dropping
the old one, so re-enrichment accumulated BOTH. Filter out the legacy
dataexplorer landing URL so re-enrichment truly self-heals to the canonical
one; update the LLM validator prompt to vouch for nemar.org/dataset/...; add a
regression test.
…eparation

Epic #837: Complete separation from legacy nemar.org dataexplorer + DOI re-targeting
bruAristimunha and others added 2 commits June 26, 2026 22:15
The run-prescreen workflow sends issue_url: null when ISSUE_URL="" via jq
(if $iu=="" then null else $iu end). The server validation checked
`b.issue_url !== undefined` but not `b.issue_url !== null`, so null triggered
'issue_url must be a string' and every prescreen callback returned 400.

The DB handler already handles null correctly (`body.issue_url || null`), so
accepting null here is safe and consistent with the field being truly optional.

Fixes: all prescreen callbacks failing with HTTP 400 since #756 advisory change.
The ORCID feature (#835) made signupSchema require orcid/city/country, but
test/api.test.ts's signup bodies were not updated, so the 3 'existing X
returns 409' tests got 400 (validation) before the duplicate check -- leaving
integration-dev (required on main) red on dev since the ORCID merge, and
blocking the 0.8.70 release. Add valid orcid/city/country to all 7 signup
bodies so each isolates its intended validation; 409s reach the conflict path.
Verified: full api.test.ts is 70/70 green against the dev backend.

This branch was previously deployed

2 inactive deployments
development — b731b8bf Deployed Jun 27, 2026 by neuromechanist via deploy-dev #634
npm — b731b8bf Deployed Jun 27, 2026 by neuromechanist via publish #1598
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 4: Delete legacy ds shadows + purge our records from legacy dataexplorer DB Search returns non existent dataset

2 participants