Skip to content

phase 3: QualityPanel with runtime aggregator from per-file dataqual - #24

Merged
neuromechanist merged 2 commits into
feature/issue-1-epic-nemar-redesignfrom
feature/issue-1-phase3-rework
May 18, 2026
Merged

neuromechanist merged 2 commits into
feature/issue-1-epic-nemar-redesignfrom
feature/issue-1-phase3-rework

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Reworks Phase 3 (the data-quality half) against the current epic architecture. Supersedes the scope of #11 for QualityPanel; closes the QualityPanel half of #21.

Summary

Ships the QualityPanel + 4 charts + HED wordcloud component for /dataset/<id> pages. Renders when QA artifacts exist in S3 (today: 31 mirrored datasets synced 2026-05-18); hides cleanly when not.

Why a rework instead of a rebase

Phase 3 PR #11 forked before the epic moved README + BIDS tree fetching off the page (c15a199). A direct rebase produces unresolvable conflicts in [id].astro because Phase 3's commit relies on synchronous imports (fetchGithubReadme, findReadmeEntry, getManifest, buildTree) that the epic moved into /api/dataset/[id]/* endpoints.

What's new

  • src/lib/qa.ts — types + client (from Phase 3: Data quality charts + HED wordcloud + file modals #11, unchanged), plus getQaAggregates now delegates to the runtime aggregator instead of fetching a static aggregates.json that hallu doesn't produce
  • src/lib/qa-aggregate.ts — new. Walks data.nemar.org/<id>/qa/ recursively, fetches every *_dataqual.json in parallel, assembles QaAggregates (file count, pipeline status, per-file percent arrays for goodData/goodChans/goodICA, line-noise dB). Handles both shallow (sub-XXX/eeg/) and session-aware (sub-XXX/ses-YY/eeg/) BIDS layouts.
  • src/lib/qa-aggregate.test.ts — 4 tests (root 404, no per-file files, full aggregation, icaFail handling)
  • src/components/QualityPanel.astro + PipelineSuccessChart, HistogramChart, AgeGenderChart, HedWordcloud — from Phase 3: Data quality charts + HED wordcloud + file modals #11
  • src/pages/dataset/[id].astro — adds getQaAggregates(id) to the existing Promise.all, renders <QualityPanel aggregates={qaAggregates} hedSummary={null} /> between Readme and BidsTree

Verified on live data

/dataset/on002718 renders QualityPanel with:

  • Pipeline success: 18/18 finished
  • Data frames retained: median 82% across 18 files, distributed across the 30-100% range
  • Channels retained, IC retention, line-noise — all populated

Cold SSR adds ~2-2.5s (the recursive listing fan-out). The page's existing s-maxage=300 Cache-Control absorbs repeat visits.

Deliberately deferred

  • HED wordcloud — depends on hed-summary.json which hallu doesn't produce today; QualityPanel passes hedSummary={null} for now
  • Demographics chart — depends on participants.tsv ingestion; aggregator omits the field
  • FileVis/FileView modals — file-level click integration with the BIDS tree (the tree is now rendered server-side via render-tree.ts string templates rather than the old BidsDirChildren component, so the modal trigger needs different wiring). Will file as a separate sub-issue.
  • Deep-path listing on session-structured datasets — data.nemar.org/<id>/qa/sub-XX/ses-YY/<modality>/ returns children: 0 even though S3 has the files. Worker-side bug, separate from this PR; the aggregator handles it gracefully by hiding the panel. Will file against nemar-cli.

Tests + lint

  • 102/102 tests pass (4 new aggregator tests + 18 existing qa.ts tests pre-existing in Phase 3)
  • typecheck clean
  • biome clean

Reworked Phase 3 against the current epic architecture (deferred README +
BIDS tree fetching). Ships the data-quality half: pipeline success, three
percent-retained histograms, line-noise distribution.

Phase 3 originally expected the hallu sync to produce a pre-aggregated
aggregates.json + hed-summary.json. Hallu instead emits per-file
dataqual.json files at every leaf path under <id>/qa/. New
lib/qa-aggregate.ts walks the data.nemar.org listing tree recursively,
fetches per-file dataquals in parallel, and assembles QaAggregates at
request time. [id].astro's Cache-Control absorbs the fan-out cost for
repeat visits.

QualityPanel renders for datasets with QA in S3 today (31 datasets
synced 2026-05-18; on002718 renders 18 files end-to-end). For
datasets without QA, the panel hides itself gracefully.

Out of scope:
- HED wordcloud (depends on hed-summary.json which hallu doesn't produce)
- Demographics (depends on participants.tsv ingestion)
- FileVis/FileView modals (file-level click integration; will close
  #21 separately)

Closes #21 in part — file modals tracked as a follow-up.
Refs #11 (the original Phase 3 PR; supersedes its scope
half-A).
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented May 18, 2026 •

Copy link
Copy Markdown

Deploying nemar-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 5891c4a
Status: ✅  Deploy successful!
Preview URL: https://09f4bfb8.nemar-website.pages.dev
Branch Preview URL: https://feature-issue-1-phase3-rewor.nemar-website.pages.dev

View logs

Fixes from code-reviewer, silent-failure-hunter, and pr-test-analyzer:

- Cap per-file fan-out at 25 parallel fetches (was unbounded). Large
  datasets with hundreds of files could saturate data.nemar.org or
  graze Cloudflare's worker subrequest ceiling.
- Surface truncated listings: console.warn when the listing endpoint
  reports it served a partial subtree, instead of silently producing
  an under-counted aggregate.
- Distinguish non-404 listing/file failures: console.warn on 5xx or
  parse errors so a worker-side regression is visible across all
  datasets instead of silently hiding QualityPanel.
- Warn when more than zero per-file fetches fail; an aggregate over
  20 of 100 files looks plausible but is misleading.
- Stricter test router: exact-URL match, so a regression that drops
  the trailing-slash append is caught by the mocked tests (the bare
  form 308-redirects to a broken /data/ prefix in production).
- New tests: session-structured (sub/ses/eeg) recursion, partial
  mid-walk 404, 'cleaning' branch (no ICA percent), 'other' branch
  (line-noise only, no percent fields).

107/107 tests pass, typecheck clean.
@neuromechanist

Copy link
Copy Markdown
Contributor Author

pr-review-toolkit review (sonnet)

Three reviewers, seven findings, all addressed in 5891c4a.

code-reviewer (REQUEST CHANGES)

Finding Resolution
truncated flag declared but never checked Added console.warn when listing returns truncated: true so a partial subtree doesn't silently produce an under-counted aggregate
Unbounded per-file fan-out (could saturate data.nemar.org or graze Worker subrequest ceiling on large datasets) Capped at 25 parallel fetches via fetchBatched(). ~2.4s end-to-end on on002718 (unchanged), bounded for >25-file datasets
other bucket fragility comment Added to docstring noting the pipelineStatus heuristic is field-presence-based and would need a re-think when hallu exposes explicit state

silent-failure-hunter (2 SILENT FAILURES)

Finding Resolution
fetchListing/fetchPerFileDataqual swallowed non-404 errors + JSON parse errors with no log Distinguished status codes: 404 returns null quietly; 5xx, parse errors, and other failures emit console.warn with the URL and reason before returning null
Per-file discard rate never surfaced when N of M fetches fail Added a final console.warn when valid.length < dataqualUrls.length so under-counted aggregates are visible in logs

pr-test-analyzer (NEEDS MORE COVERAGE)

Gap Resolution
cleaning branch (file with goodData + goodChans but no goodICA) untested New test covering the cleaning bucket
other branch (line-noise only) untested New test covering the other bucket
Session-structured recursion (sub/ses/eeg) untested New test exercising the 3-level descent in walkForDataqual
Partial mid-walk 404 untested New test where one of two subject listings 404s, asserting the other still aggregates
Mock router's slash-stripping masked the trailing-slash append in production Tightened router to require exact URL match; tests now break if the code drops the trailing-slash append (which production servers 308-redirect to a broken path)

107/107 tests pass (was 102, +5 new), typecheck clean, biome clean. Live on002718 still renders 18 files at ~2.4s cold call.

@neuromechanist
neuromechanist merged commit f738a55 into feature/issue-1-epic-nemar-redesign May 18, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-1-phase3-rework branch May 18, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant