Repository navigation
Per-version HED detection + filtering (epic #869) - #878
Conversation
* feat(db): HED columns + /datasets projection (#870) * test/db: address review (has_hed CHECK, 0-vs-null, counts)
Cross-phase integration review (Sonnet, full epic diff)Per-phase reviews passed in isolation; this finalize review checked the phase SEAMS and found two real cross-phase bugs, both fixed in 1. Spurious 2. Sweep clobbered a prior classification on a transient probe failure (conf 80) Everything else verified consistent across phases: Gates after the fix: 99 HED/reindex tests green; Biome (src) + backend |
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR878Version: 🤖 Updated on each push. |
Summary
Makes datasets filterable by HED (Hierarchical Event Descriptors) presence, per dataset version. Fixes the original bug: the website's "Has HED annotation" filter was a dead control (it filtered nothing, so non-HED datasets like
on007058appeared). There was no HED field anywhere in D1 or the API. This epic detects HED, stores it first-class per version, backfills existing datasets, and exposes a real?has_hed=1filter.Detection rule:
has_hed = 1only whendataset_description.jsondeclares aHEDVersionAND at least one real HED annotation exists (an*_events.jsoncolumn with a"HED"key, OR an*_events.tsvwith aHEDcolumn). High-precision AND — eliminates the false positives.Per-version: HED truth lives on
dataset_versions; the latest version's value is denormalized ontodatasetsfor fast single-table list filtering. e.g. nm000132 v1.0.0 has no HED but v1.1.1 does.Phases (each reviewed + squash-merged into this epic branch)
has_hed/hed_version/hed_checked_atondatasets,has_hed/hed_versionondataset_versions,CHECK (has_hed IN (0,1)), index; projected in/datasets.probeHed()ingetBidsTreeStats(reads HEDVersion + scans root/subject events sidecars); threaded throughcomputeDatasetMetadataColumns/writeDatasetMetadataColumns+ newwriteVersionHed; wired at publish viawebhooks.ts->dataset-reindex.ts.POST /admin/datasets/hed-sweep(modality-agnostic, per-version write,?reset=1, noupdated_atbump) +nemar admin hed-sweepCLI.?has_hed=1on/datasets(browse) +/datasets/search;nemar dataset list/search --hed+ a HED badge.Dependent follow-up (separate repo)
nemarOrg/website#139— wire thehas_hedfilter intofilterStateToAPIQuery, add the field to the Dataset type, surface the badge. This epic makes the API honor?has_hed=1; the website change makes the checkbox live.Test plan
dev(all four phases together): detection helpers, compute/write, migration, per-version, sweep SQL contract, browse + search filter (real bun:sqlite, no mocks). Biome (src) + backendtypecheck+ CLItypecheckclean.cli.test.tsrow-output failures are pre-existing live-API integration tests (identical on the dev baseline), unrelated to HED.Closes #869