Repository navigation
Phase 3: Data quality charts + HED wordcloud + file modals - #11
Closed
neuromechanist wants to merge 4 commits into
Closed
neuromechanist wants to merge 4 commits into
neuromechanist wants to merge 4 commits into
Conversation
src/lib/qa.ts defines the contract the website expects from data.nemar.org/<id>/qa/* (sibling backend issue: nemarOrg/nemar-cli#511): - QaAggregates: per-file arrays for the 4 dataset-level charts (pipeline status, frames retained %, channels retained %, ICA %, line-noise dB) + optional demographics from participants.tsv - QaSummary: min/max ranges (existing hallu dataqual.json shape, kept for completeness) - FileQa: per-file summary JSON (matches the shape we sampled from hallu's processed/<id>/sub-XXX/eeg/*_dataqual.json) - HedSummary: token frequency map for the wordcloud - FILE_PLOT_KINDS + filePlotUrl(): the 5 SVGs (eegplot_mid-sample, icaact, icahist, icamaps, spectopo) the modal renders per .set file Pure helpers tested in qa.test.ts (18 cases): - parseLinenoiseDb('14.40dB') -> 14.4 - buildHistogram with N bins + optional domain - bucketAgesBySex with configurable bucket width - filePlotUrl with percent-encoded path segments jsonOrNull() reuses the 429/5xx-as-null pattern from data-api.ts so QA fetches degrade to empty states instead of breaking the dataset page when artifacts haven't been synced yet. Part of #4.
Hand-rolled SVG charts (no chart-lib dependency). Each is ~80-150 lines: - PipelineSuccessChart: stacked horizontal bar (green/amber/red/grey for finished / cleaning / failed / other) with legend - HistogramChart: 10-bin SVG histogram, reused 3x for frames retained %, channels retained %, line-noise dB. Median + sample-count badges. - AgeGenderChart: paired bars per 10-year age bucket, split M/F/O using modality accent colors (semantic: EEG blue for M, iEEG pink for F). - HedWordcloud: log-scaled font sizing, semantic category-colored tags (Sensory-event / Agent-action / Item / Property / Spatial / ...), rectangular flow layout. No JS cloud-packing algorithm — that's overkill for the 24-tag overview. QualityPanel composes the four charts + the wordcloud, renders only when aggregates or hedSummary is present, empty state when both are null. Wired into /dataset/[id].astro between the version switcher and the README. Behind a ?qa=fixture toggle for dev work today; the toggle is a no-op once nemarOrg/nemar-cli#511 ships the live endpoint. Fixtures (test/fixtures/qa-*.json) synthesized from the actual hallu sample (ds002718 sub-002 task-FaceRecognition_eeg_dataqual.json that we read this session) — 30 files, realistic distributions, demographic ages 21-38 spread across M/F. 87/87 unit tests still passing. Typecheck clean across 53 files. Part of #4.
… real buttons
FileVisModal.astro — opens on click of a Vis button on any .set file.
- Header with dataset id, version, file path
- 4-stat strip (data frames %, channels %, ICA %, line noise) from
per-file qa/<path>_dataqual.json
- 5-plot grid (eegplot_mid-sample, icaact, icahist, icamaps, spectopo) as
lazy-loaded <img> elements pointing at the qa SVG URLs
- HEAD-probes each SVG so missing ones render a 'Not available' tile
rather than broken-image icons
- In-memory cache per (id, path) so reopens are instant
- 'QA artifacts not synced yet' empty state when nothing comes back
- Lazy-mounted ONCE at the BidsTree root; opened via [data-open-vis]
attributes on the buttons in BidsDirChildren
FileViewModal.astro — TSV preview, same mounting pattern.
- Fetches the file via the existing data.nemar.org/<id>/<ver>/<path>
- Parses first 100 rows into an HTML table with sticky header
- 'Showing N of M rows' counter + 'Download full file' link in footer
- Renders error state when the fetch fails
BidsDirChildren.astro: takes new datasetId + version props, threads
through recursion, strips them onto buttons via data-* attributes. The
'Vis · soon' and 'View · soon' decorative chips are gone — those tags
were the Phase 5 placeholder. Real buttons take their place.
BidsTree.astro: mounts both modals once after the file tree so the
single global event listener in each modal handles every click. Threads
datasetId + version down to the recursive children.
Detail page now passes datasetId={id} version={selectedVersion} into
<BidsTree>. No-op when those are missing — buttons emit empty data-*
attributes and the modals early-return.
Verified live with ?qa=fixture on nm000104 — Vis modal opens with the
expected stats + plot frames (plots themselves 404 against the live
backend until #511 ships, which is exactly the empty-state we want).
Part of #4.
The 'compile' setting still bundled sharp into the SSR worker, which the
Workers runtime can't execute — deploys failed with:
Uncaught Error: [unenv] process.report.getReport is not implemented yet!
We don't use Astro's <Image> component or astro:assets anywhere — every
visual asset is a plain <img> referencing /public/*.{svg,jpg,png}.
Passthrough lets the original assets ship as-is, keeps sharp out of the
worker bundle, and shaved the deploy from failing to ~3s.
This was referenced May 18, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3 of #1. Adds the data-quality visualizations from the legacy nemar.org back onto the new detail page, plus the per-file Vis/View modals that replace the Phase 5 "soon" placeholders.
Closes #4.
Cross-repo: the backend half (hallu sync +
data.nemar.org/<id>/qa/*route) is tracked at nemarOrg/nemar-cli#511. Until that ships, this PR renders against fixtures via a?qa=fixturedev toggle; the toggle becomes a no-op once #511 deploys.What landed
Library —
src/lib/qa.tsQaAggregates/QaSummary/FileQa/HedSummarytypes — define the contract the website expects fromdata.nemar.org/<id>/qa/*so Epic: Website compute surfaces (#493) #511 has a targetgetQaAggregates/getFileQa/getHedSummarytyped clients with abortable timeouts; 404/429/5xx → null (never break the page)parseLinenoiseDb,buildHistogram,bucketAgesBySex,filePlotUrlpure helpers with 18 unit tests inqa.test.tsCharts (hand-rolled SVG, no chart-lib dep)
PipelineSuccessChart.astro— stacked horizontal bar (finished / cleaning / failed / other), semantic green/amber/red paletteHistogramChart.astro— 10-bin SVG histogram, reused 3x for frames retained %, channels retained %, line-noise dB. Median + sample-count badges.AgeGenderChart.astro— paired bars per 10-year age bucket, split M/F/O using modality accent colorsHedWordcloud.astro— log-scaled font sizing, semantic category-colored tags, rectangular flow layout. No JS cloud-packing — that's overkill for a 24-tag overview.QualityPanel —
src/components/QualityPanel.astroComposes the 4 charts in a 2-col grid + the HED wordcloud in a band below. Renders nothing if both aggregates and HED summary are null (graceful empty state).
Wired into
/dataset/[id].astrobetween the version switcher and the README.File modals
FileVisModal.astro— lazy-mounted once at the BidsTree root, opens on[data-open-vis]click. Header + 4-stat strip + 5-SVG grid (eegplot_mid-sample, icaact, icahist, icamaps, spectopo). HEAD-probes each SVG so missing ones render a 'Not available' tile, not a broken icon. In-memory cache per (id, path).FileViewModal.astro— same mounting pattern, opens on[data-open-view]click. Fetches the .tsv via the existing version-locked URL, renders first 100 rows as a sticky-header table, footer shows row count + 'Download full file'.BidsTree integration
BidsDirChildren.astrotakes newdatasetId+versionprops, threads them through recursion. TheVis · soonandView · soondecorative chips are gone — replaced with real<button>elements emittingdata-open-vis/data-open-view+ the per-file metadata the modals need.BidsTree.astromounts both modals once after the file tree.Verification
bun run typecheck— 0 errors across 55 filesbun run test— 87/87 unit tests passing (69 from Phase 1+2 + 18 new for qa.test.ts)bun run dev+/browsecapture in both themes — QualityPanel renders 4 charts + HED cloud from fixture data on/dataset/nm000104?qa=fixture.setfile → modal opens, stats populated, plot tiles render (or show 'Not available' until Epic: Website compute surfaces (#493) #511 ships the SVGs).tsvfile → modal opens, fetches the tsv via the existing manifest URL, renders the table/dataset/on005262(no QA exists, no fixture) — empty state, page renders cleanlyOut of scope
data.nemar.org/<id>/qa/*endpoint — tracked at QA artifacts sync from hallu + data.nemar.org/<id>/qa/* route nemar-cli#511Test plan
/dataset/nm000104?qa=fixture— all 4 charts + wordcloud render in both themes.setfile → modal opens with stats strip + 5 plot tilesparticipants.tsv→ table renders first 100 rows/dataset/on005262(no QA, no fixture) → no QualityPanel renders, page is otherwise normal?qa=fixturetoggle in a follow-up commit, verify live QA renders on a synced datasetFollow-ups
/design-reviewon the live Cloudflare Pages preview once the project is set up. Visual fixes will land as a separate polish PR if needed.