Skip to content

Epic: Redesign nemar.org on Cloudflare Pages + Astro, reusing data.nemar.org #1

Closed
8 of 10 issues completed

Description

@neuromechanist

Goal

Rebuild nemar.org as a Cloudflare Pages site (Astro) that reuses the existing api.nemar.org (datasets catalog, D1) and data.nemar.org (BIDS HTTPS view, metadata.json, manifest.json from issue nemarOrg/nemar-cli#449) as its backend. Replicate every feature the legacy Joomla nemar.org provides; defer the live EEG viewer and NSG REST connector to follow-up epics.

Why

  • Legacy site is Joomla + datapipeline01/02 servers; hard to evolve and decoupled from the new Cloudflare/D1/S3 stack.
  • nemar-cli issue S3 dataset/ prefix: tool-agnostic cloud-like access to NEMAR datasets nemar-cli#449 already gives us a stable HTTPS contract for dataset bytes and metadata. The website becomes the front door for that contract.
  • Data quality artifacts live on hallu (/data/qumulo/openneuro/processed/<id>/) and must move to S3 so the new site can render them without an SSH hop.

Architecture

User -> nemar.org (Astro on Cloudflare Pages)
            |
            +-- api.nemar.org/datasets               (D1 catalog list/search)
            +-- data.nemar.org/<id>/metadata.json    (neuroschema doc)
            +-- data.nemar.org/<id>/<ver>/manifest.json  (BIDS file index)
            +-- data.nemar.org/<id>/<ver>/<path>     (302 -> presigned S3)
            +-- data.nemar.org/<id>/qa/*             (NEW: hallu-mirrored QA artifacts)
            +-- s3://nemar/<id>/archives/<ver>.zip   (whole-dataset zip)
            +-- github.com/nemarDatasets/<id>/issues (per-dataset forum)

Phases

  • Phase 1: Foundation + Discover - Astro scaffold, design system, nav, landing page, Discover page with cards, filters (modality, format, participant/session/channel counts, HED, data-quality), sort, pagination wired to api.nemar.org/datasets.
  • Phase 2: Dataset detail (catalog metadata) - /dataset/<id> page from data.nemar.org/<id>/metadata.json + landing JSON. Action buttons (Download zip, Compute, Issues, OpenNeuro), README, panel, version picker, collapsible BIDS file tree from manifest.json.
  • Phase 3: Data quality + HED visualization - Cross-repo: in nemar-cli add hallu-qa-sync.sh + cron + GET data.nemar.org/<id>/qa/* route (sibling issue on nemarOrg/nemar-cli); in website render dataset-level charts, HED wordcloud, per-file Vis modal (5 SVGs + dataqual summary).
  • Phase 4: Citation Dashboard + Community + Docs - /citation-dashboard page (S3-synced JSON), per-dataset citation block, Community page wired to per-dataset GitHub Issues, Documentation via Astro Content Collections.
  • Phase 5: Live-features architecture + cutover - Design doc + follow-up issues for viewer (100Hz preview) and NSG REST connector, placeholder Vis/Compute states, a11y + theming polish, DNS cutover plan.

Out of scope

  • Full EEG viewer implementation (100Hz preview generation, multi-channel canvas viewer). Tracked as a follow-up epic in Phase 5.
  • NSG REST API connector backend implementation. Tracked as a follow-up epic in Phase 5.
  • Backing-store migration off the AWS Open Data S3 bucket.
  • Changes to nemar-cli upload/git-annex flow.

Open questions

  • How is the citation dashboard complete_analysis_data_clean.json regenerated today (manual process_citations.py or pipeline)? Reflect on Phase 4.
  • Should legacy.nemar.org be a hard cutover or a parallel-host period? Decide in Phase 5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicMulti-phase epicfeatureNew feature

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions