Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 5 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,9 @@ The backend code (`ORG_NAME = "nemarDatasets"` in `backend/src/services/github.t
### Public Browser Sites
Two user-facing dataset browsers exist during the cutover; **prefer ww2 for new references**.

- **`https://ww2.nemar.org`** — **current beta browser** (Astro SSR, lives in `nemarOrg/website`). Dataset pages at `ww2.nemar.org/dataset/<id>`. Reads `api.nemar.org` and `data.nemar.org` directly; does not depend on the legacy datapipeline.
- **`https://nemar.org/dataexplorer/...`** — **legacy PHP site**. Backed by the `dataexplorer_*` tables on `nemar.org/api/dataexplorer/datapipeline`, which `nemar admin sync run` populates. Still public, slated for replacement by ww2.
- **Implication for on\* datasets:** `nemar admin sync` intentionally skips on\*-prefix datasets (alternate_id mapping pending, issue #339), so they render on ww2.nemar.org but the legacy `nemar.org/dataexplorer` detail page is empty until that ships. The archive zip download link in the legacy UI is driven by `zip_file_size` on the datapipeline row, so even an existing S3 archive won't surface there for on\* datasets.
- **Default in docs/comments:** when something says "the website," "the browser," or "the UI" without further qualification, mean ww2.nemar.org. Name `nemar.org` explicitly only when referring to the legacy dataexplorer, the datapipeline tables, or the `nemar.org` DOI landing target.
- **`https://ww2.nemar.org`** — **current dataset browser** (Astro SSR, lives in `nemarOrg/website`). Dataset pages at `ww2.nemar.org/dataset/<id>`. Reads `api.nemar.org` and `data.nemar.org` directly. DOIs land on the canonical `https://nemar.org/dataset/<id>` (versions `?v=v<version>`), which a Cloudflare redirect on the `nemar.org` zone forwards to ww2.
- **`https://nemar.org/dataexplorer/...`** — **legacy PHP site, being retired.** As of epic #837 NEMAR no longer syncs to or from it: the outgoing datapipeline push and the incoming 4h catalog pull are removed, our `nm`/`on` records are purged from its `dataexplorer_*` tables, and the legacy `ds######` shadow rows are dropped from our D1. It may still serve its own OpenNeuro `ds` catalog independently.
- **Default in docs/comments:** when something says "the website," "the browser," or "the UI" without further qualification, mean ww2.nemar.org. Name `nemar.org` explicitly only when referring to the legacy dataexplorer or the `nemar.org/dataset/<id>` DOI landing target.

### S3 Bucket Structure
```
Expand Down Expand Up @@ -115,7 +114,7 @@ Datasets are synced to SDSC Hallu (`ssh hallu`) for processing pipelines and dow
- **Zip dir:** `/data/qumulo/openneuro/zip_files/` (downloadable archives from S3)
- **Discovery:** Queries `GET /datasets` API, filters for `nm`-prefix only
- **Manual run:** `ssh hallu /data/qumulo/openneuro/nemar-cli/scripts/hallu-sync.sh --dataset nm000132 --verbose`
- **Note:** This syncs data files. Legacy `nemar.org/dataexplorer` metadata is synced separately via `nemar admin sync run` or automatically on version DOI publish.
- **Note:** This syncs data files only. (The legacy `nemar.org/dataexplorer` metadata sync was retired in epic #837.)

## Environment Setup
```bash
Expand Down Expand Up @@ -412,9 +411,7 @@ nemar admin revert # Revert dataset to previous version
nemar admin make-public # Publish dataset (permanent)
nemar admin delete-dataset # Delete dataset and all resources
nemar admin bulk-delete # Delete multiple phantom/orphaned datasets (owner only)
nemar admin sync run # Sync dataset metadata to nemar.org (legacy dataexplorer)
nemar admin sync status # Show nemar.org sync status
nemar admin reindex <id> # Refresh enrichment + nemar.org sync + D1 metadata columns
nemar admin reindex <id> # Refresh enrichment + D1 metadata columns
nemar admin reindex --all # Bulk; also --missing-metadata, --stale [--older-than N]
nemar admin summary # summary.json coverage across published versions
nemar admin email-preferences show # Show email notification preferences
Expand Down
17 changes: 17 additions & 0 deletions backend/src/db/migrations/0053_drop_legacy_sync_state.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
-- Drop the legacy nemar.org datapipeline sync state (epic #837 Phase 5).
--
-- The outgoing datapipeline push (nemar-sync.ts) and the incoming catalog pull
-- (catalog-sync.ts + /admin/catalog/sync) were removed in Phase 3, so the
-- nemar_sync_* columns (migration 0014) and the catalog_sync_log table
-- (migration 0018) are now dead and have no remaining readers/writers.
--
-- DROP COLUMN is safe here: no index, trigger, FK, or the datasets_fts external
-- -content virtual table references these columns (datasets_fts indexes only
-- name/description/authors/tasks/modalities/readme). Validated on the SCCN dev
-- D1 mirror before prod.

ALTER TABLE datasets DROP COLUMN nemar_sync_status;
ALTER TABLE datasets DROP COLUMN nemar_sync_at;
ALTER TABLE datasets DROP COLUMN nemar_sync_error;

DROP TABLE IF EXISTS catalog_sync_log;
4 changes: 2 additions & 2 deletions backend/src/routes/datasets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -707,7 +707,7 @@ datasetRoutes.get("/", optionalAuthMiddleware, async (c) => {
let query = `
SELECT d.dataset_id, d.name, d.description, d.status, d.visibility,
d.github_repo, d.concept_doi, d.created_at, d.updated_at,
u.username AS owner_username, d.nemar_sync_status,
u.username AS owner_username,
d.source, d.source_id,
COALESCE(d.modalities, '') AS modalities,
COALESCE(d.subject_count, 0) AS participants,
Expand Down Expand Up @@ -754,7 +754,7 @@ datasetRoutes.get("/", optionalAuthMiddleware, async (c) => {
let query = `
SELECT d.dataset_id, d.dataset_id AS id, d.name, d.description, d.status, d.visibility,
d.github_repo, d.concept_doi, d.concept_doi AS doi, d.created_at, d.updated_at,
COALESCE(d.uploader, u.username) AS owner_username, d.nemar_sync_status,
COALESCE(d.uploader, u.username) AS owner_username,
d.source, d.source_id,
COALESCE(d.modalities, '') AS modalities,
COALESCE(d.subject_count, 0) AS participants,
Expand Down
12 changes: 1 addition & 11 deletions src/commands/dataset.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2331,16 +2331,6 @@ function renderDatasetTable(
const subjects = dataset.participants ? String(dataset.participants) : "-";
const owner = dataset.owner_username || "-";

// Discrepancy indicators:
// * = managed (on GitHub) but not synced to nemar.org
// italic/dim = catalog-only (on nemar.org but not on GitHub)
let statusIndicator = "";
if (dataset.source_type === "managed" && !dataset.nemar_sync_status) {
statusIndicator = chalk.yellow("*"); // On GitHub, not synced
} else if (dataset.source_type === "managed" && dataset.nemar_sync_status === "failed") {
statusIndicator = chalk.red("!"); // Sync failed
}

const visLabel = dataset.visibility === "public" ? chalk.green("pub") : chalk.yellow("prv");

let idDisplay: string;
Expand All @@ -2361,7 +2351,7 @@ function renderDatasetTable(
modality.padEnd(modWidth),
subjects.padEnd(subjWidth),
owner.padEnd(ownerWidth),
`${visLabel} ${statusIndicator}`.trim(),
visLabel,
].join(" ");
console.log(row);
}
Expand Down
2 changes: 0 additions & 2 deletions src/lib/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -647,8 +647,6 @@ export interface Dataset {
file_size_formatted?: string;
/** 'managed' = in D1 datasets table, 'catalog' = nemar.org only */
source_type?: "managed" | "catalog";
/** nemar.org sync status for managed datasets */
nemar_sync_status?: string | null;
/** DOI field from catalog (for catalog-only datasets) */
doi?: string | null;
/** Import source (e.g. "openneuro") */
Expand Down
2 changes: 1 addition & 1 deletion test/license-tier.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ CREATE TABLE datasets (
created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')),
is_sandbox INTEGER NOT NULL DEFAULT 0,
visibility TEXT NOT NULL DEFAULT 'private' CHECK (visibility IN ('private','public')),
enrichment_json TEXT, source TEXT, source_id TEXT, nemar_sync_status TEXT,
enrichment_json TEXT, source TEXT, source_id TEXT,
subject_count INTEGER, modalities TEXT, age_min REAL, age_max REAL, file_size INTEGER,
total_files INTEGER, tasks TEXT,
FOREIGN KEY (owner_user_id) REFERENCES users(id)
Expand Down
Loading