Skip to content

Phase 4: legacy DB purge + ds-row deletion (closes #748) - #845

Merged
neuromechanist merged 1 commit into
feature/issue-837-epic-legacy-separationfrom
feature/issue-793-phase4-purge-data
Jun 27, 2026
Merged

neuromechanist merged 1 commit into
feature/issue-837-epic-legacy-separationfrom
feature/issue-793-phase4-purge-data

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Phase 4 of epic #837 — the data-purge half of "complete separation". Two guarded run-once scripts (same pattern as the Phase 2 re-target; not CLI/endpoints):

  1. scripts/purge-legacy-datapipeline.ts — deletes NEMAR's own nm/on records from the 4 legacy dataexplorer datapipeline tables so the legacy site stops showing our datasets. Reuses the slim legacy-purge.ts client extracted in Phase 3. Dry-run default; --execute needs NEMAR_USERNAME/NEMAR_PASSWORD; idempotent. Dry-run: 608 nm+on datasets.
  2. scripts/delete-ds-rows.ts — deletes the legacy ds###### shadow rows (folded OpenNeuro catalog, owner=SYSTEM) from D1 + Vectorize, closing Search returns non existent dataset #748 (search returning backing-less shadows). Children-first then datasets (FTS trigger + FK cascade); pre-delete snapshot; Vectorize delete-vectors. Dry-run confirms the shadows live only in datasets (173 rows), 0 in every dependent table.

Ordering precondition: the incoming catalog-sync (GitHub "Catalog Sync" workflow + /admin/catalog/sync) must be stopped before deleting ds rows, or the 4h cron re-folds them. The runs happen post-deploy / after the cron is disabled, gated on owner go.

Closes #840? no — Closes #793; fixes #748. Part of epic #837.

Test plan

  • bun test test/delete-ds-rows.test.ts (FK-safe delete order, ds-only targeting, toCountSql).
  • Biome clean; live read-only dry-runs of both scripts against prod (608 / 173).
  • --execute runs performed out-of-band (cron OFF first), gated on owner go.

Epic #837 Phase 4 tooling (guarded run-once scripts, not CLI/endpoints):
- purge-legacy-datapipeline.ts: delete our nm/on records from the 4 legacy
  dataexplorer tables via the extracted legacy-purge.ts client (dry-run default,
  --execute needs NEMAR_USERNAME/PASSWORD, idempotent). Dry-run: 608 datasets.
- delete-ds-rows.ts: delete the ds shadow rows from D1 (children-first, datasets
  last -> FTS/FK cascade) + Vectorize, with a pre-delete snapshot. Closes #748.
  Dry-run confirms ds rows live ONLY in datasets (173), 0 in dependent tables.
  PRECONDITION: catalog-sync cron must be OFF first (else 4h re-fold).
The --execute runs are gated prod ops.
@github-actions

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR845

Version: 0.8.70-PR845.bd2d5d2
Commit: bd2d5d2


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 65ffc52 into feature/issue-837-epic-legacy-separation Jun 27, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-793-phase4-purge-data branch June 27, 2026 03:42

This branch was previously deployed

1 inactive deployment
npm — bd2d5d20 Deployed Jun 27, 2026 by neuromechanist via publish #1586
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant