Repository navigation
Closed
8 / 108 of 10 issues completed
Copy link
Labels
Description
Goal
Rebuild nemar.org as a Cloudflare Pages site (Astro) that reuses the existing api.nemar.org (datasets catalog, D1) and data.nemar.org (BIDS HTTPS view, metadata.json, manifest.json from issue nemarOrg/nemar-cli#449) as its backend. Replicate every feature the legacy Joomla nemar.org provides; defer the live EEG viewer and NSG REST connector to follow-up epics.
Why
- Legacy site is Joomla +
datapipeline01/02servers; hard to evolve and decoupled from the new Cloudflare/D1/S3 stack. nemar-cliissue S3 dataset/ prefix: tool-agnostic cloud-like access to NEMAR datasets nemar-cli#449 already gives us a stable HTTPS contract for dataset bytes and metadata. The website becomes the front door for that contract.- Data quality artifacts live on
hallu(/data/qumulo/openneuro/processed/<id>/) and must move to S3 so the new site can render them without an SSH hop.
Architecture
User -> nemar.org (Astro on Cloudflare Pages)
|
+-- api.nemar.org/datasets (D1 catalog list/search)
+-- data.nemar.org/<id>/metadata.json (neuroschema doc)
+-- data.nemar.org/<id>/<ver>/manifest.json (BIDS file index)
+-- data.nemar.org/<id>/<ver>/<path> (302 -> presigned S3)
+-- data.nemar.org/<id>/qa/* (NEW: hallu-mirrored QA artifacts)
+-- s3://nemar/<id>/archives/<ver>.zip (whole-dataset zip)
+-- github.com/nemarDatasets/<id>/issues (per-dataset forum)
Phases
- Phase 1: Foundation + Discover - Astro scaffold, design system, nav, landing page, Discover page with cards, filters (modality, format, participant/session/channel counts, HED, data-quality), sort, pagination wired to
api.nemar.org/datasets. - Phase 2: Dataset detail (catalog metadata) -
/dataset/<id>page fromdata.nemar.org/<id>/metadata.json+ landing JSON. Action buttons (Download zip, Compute, Issues, OpenNeuro), README, panel, version picker, collapsible BIDS file tree frommanifest.json. - Phase 3: Data quality + HED visualization - Cross-repo: in
nemar-cliaddhallu-qa-sync.sh+ cron +GET data.nemar.org/<id>/qa/*route (sibling issue onnemarOrg/nemar-cli); inwebsiterender dataset-level charts, HED wordcloud, per-file Vis modal (5 SVGs + dataqual summary). - Phase 4: Citation Dashboard + Community + Docs -
/citation-dashboardpage (S3-synced JSON), per-dataset citation block, Community page wired to per-dataset GitHub Issues, Documentation via Astro Content Collections. - Phase 5: Live-features architecture + cutover - Design doc + follow-up issues for viewer (100Hz preview) and NSG REST connector, placeholder Vis/Compute states, a11y + theming polish, DNS cutover plan.
Out of scope
- Full EEG viewer implementation (100Hz preview generation, multi-channel canvas viewer). Tracked as a follow-up epic in Phase 5.
- NSG REST API connector backend implementation. Tracked as a follow-up epic in Phase 5.
- Backing-store migration off the AWS Open Data S3 bucket.
- Changes to
nemar-cliupload/git-annex flow.
Open questions
- How is the citation dashboard
complete_analysis_data_clean.jsonregenerated today (manualprocess_citations.pyor pipeline)? Reflect on Phase 4. - Should
legacy.nemar.orgbe a hard cutover or a parallel-host period? Decide in Phase 5.
Activity
Phase 1: Foundation + Discover page (#7)
fix(datasets): hide ds* catalog shadows when on* mirror exists (#534)
Release v0.8.21: enrichment description fix + ds*/on* catalog dedup (#…
fix(datasets): include dataset_id + source_id in search clause (#539)
fix(catalog): sync nemar_catalog from enrichment + backfill historica…
fix(catalog): sync nemar_catalog from enrichment + backfill historica…