Skip to content

Per-version HED detection + filtering (epic #869) - #878

Merged
neuromechanist merged 6 commits into
devfrom
feature/issue-869-epic-hed-detection
Jun 30, 2026
Merged

neuromechanist merged 6 commits into
devfrom
feature/issue-869-epic-hed-detection

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Makes datasets filterable by HED (Hierarchical Event Descriptors) presence, per dataset version. Fixes the original bug: the website's "Has HED annotation" filter was a dead control (it filtered nothing, so non-HED datasets like on007058 appeared). There was no HED field anywhere in D1 or the API. This epic detects HED, stores it first-class per version, backfills existing datasets, and exposes a real ?has_hed=1 filter.

Detection rule: has_hed = 1 only when dataset_description.json declares a HEDVersion AND at least one real HED annotation exists (an *_events.json column with a "HED" key, OR an *_events.tsv with a HED column). High-precision AND — eliminates the false positives.

Per-version: HED truth lives on dataset_versions; the latest version's value is denormalized onto datasets for fast single-table list filtering. e.g. nm000132 v1.0.0 has no HED but v1.1.1 does.

Phases (each reviewed + squash-merged into this epic branch)

  1. Schema (Phase 1: HED schema + projection plumbing #874): migration 0056 — has_hed/hed_version/hed_checked_at on datasets, has_hed/hed_version on dataset_versions, CHECK (has_hed IN (0,1)), index; projected in /datasets.
  2. Detection/compute (Phase 2: probeHed detection + per-version compute #875): probeHed() in getBidsTreeStats (reads HEDVersion + scans root/subject events sidecars); threaded through computeDatasetMetadataColumns/writeDatasetMetadataColumns + new writeVersionHed; wired at publish via webhooks.ts -> dataset-reindex.ts.
  3. Backfill sweep (Phase 3: HED backfill sweep + admin command #876): POST /admin/datasets/hed-sweep (modality-agnostic, per-version write, ?reset=1, no updated_at bump) + nemar admin hed-sweep CLI.
  4. Filter + CLI (Phase 4: has_hed filter + dataset list/search --hed + badge #877): ?has_hed=1 on /datasets (browse) + /datasets/search; nemar dataset list/search --hed + a HED badge.

Dependent follow-up (separate repo)

nemarOrg/website#139 — wire the has_hed filter into filterStateToAPIQuery, add the field to the Dataset type, surface the badge. This epic makes the API honor ?has_hed=1; the website change makes the checkbox live.

Test plan

  • Per-phase reviews (3-4 Sonnet agents each: code / silent-failure / tests / comments); all findings addressed; summaries on each phase PR.
  • 130 HED tests pass on this branch after merging latest dev (all four phases together): detection helpers, compute/write, migration, per-version, sweep SQL contract, browse + search filter (real bun:sqlite, no mocks). Biome (src) + backend typecheck + CLI typecheck clean.
  • The 13 cli.test.ts row-output failures are pre-existing live-API integration tests (identical on the dev baseline), unrelated to HED.

Closes #869

* feat(db): HED columns + /datasets projection (#870)

* test/db: address review (has_hed CHECK, 0-vs-null, counts)
* feat(hed): per-version HED detection + compute (#871)

* review: HED parse-throw, version threading, tests/docs (#871)
* feat(hed): backfill sweep + nemar admin hed-sweep (#872)

* review: safe write ordering, null-remaining guard, per-version tests (#872)
* feat(hed): ?has_hed filter + dataset list/search --hed + badge (#873)

* test: hasHed+hasDoi stacking + HTTP-parse scope note (#873)
@neuromechanist

Copy link
Copy Markdown
Contributor Author

Cross-phase integration review (Sonnet, full epic diff)

Per-phase reviews passed in isolation; this finalize review checked the phase SEAMS and found two real cross-phase bugs, both fixed in 2e7b129:

1. Spurious console.error from the reindex per-version write on unpublished datasets (conf 82)
nemar admin reindex <id> passes no version; for an unpublished dataset (no dataset_versions row) with a successful probe, writeVersionHed's latest-fallback matched 0 rows and logged an error. P3's sweep already guards this via its latest_version select; P2's reindex didn't. Fix: the reindex path now resolves the target version (passed version else the latest) and skips the per-version write when the dataset has none — mirroring P3.

2. Sweep clobbered a prior classification on a transient probe failure (conf 80)
The reindex/enrich path writes has_hed without stamping hed_checked_at, so such a row is still a sweep candidate. The sweep's direct write set has_hed = NULL on a probe miss and stamped it done, destroying the good classification (recoverable only via --reset). Fix: when the sweep probe can't classify (hasHedInt null), it now ONLY stamps hed_checked_at and leaves has_hed/hed_version untouched. Added a contract test ("a failed probe does NOT clobber a prior classification (reindex seam)").

Everything else verified consistent across phases: has_hed 3-state (1/0/NULL) typing end to end (DB INTEGER -> probe boolean -> compute number|null -> API -> CLI === 1), the AND d.has_hed = 1 filter vs detection semantics, writeVersionHed signatures/guards from both callers, hed_version denormalization, the has_hed=1/true query convention across browse + search, and the HedSweepBatchResponse wire shape.

Gates after the fix: 99 HED/reindex tests green; Biome (src) + backend typecheck clean.

@github-actions

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR878

Version: 0.8.74-PR878.2e7b129
Commit: 2e7b129


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 04cd77c into dev Jun 30, 2026
10 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-869-epic-hed-detection branch June 30, 2026 03:16

This branch was previously deployed

1 inactive deployment
npm — 2e7b129c Deployed Jun 30, 2026 by neuromechanist via publish #1643
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant