Skip to content

Epic: Per-version HED detection + filtering #869

Closed
4 of 4 issues completed
Closed

Description

@neuromechanist

Problem

The website "Has HED annotation" filter (FilterSidebar has_hed) is a dead control: filterStateToAPIQuery() never sends it and applyClientFilters() has no predicate, so checking it narrows nothing and non-HED datasets (e.g. on007058) still appear. There is no HED field in D1 or the /datasets API today.

Goal

Detect Hierarchical Event Descriptors (HED) annotations per dataset VERSION, store them first-class, and make datasets filterable by HED presence from both the website and the CLI.

Detection rule (decided)

has_hed = true only when BOTH hold:

  1. dataset_description.json has a HEDVersion key, AND
  2. at least one real HED annotation exists: an *_events.json column with a "HED" key, OR an *_events.tsv with a HED column.

hed_version = the HEDVersion string (e.g. "8.3.0").

This high-precision AND is what eliminates the false positives.

Storage (per-version, denormalized latest)

  • Per-version truth on dataset_versions (has_hed, hed_version).
  • Denormalized latest-version value on datasets (has_hed, hed_version, hed_checked_at) so list filtering stays a single-table query (mirrors modality/license_tier).

Per-version matters: e.g. nm000132 v1.0.0 lacks HED but v1.1.1 has it.

Template

Follow the channel/montage epic #854 end-to-end: migration (0054/0055 style) -> probeHed() in getBidsTreeStats (github.ts) next to probeChannelMontage -> computeDatasetMetadataColumns/writeDatasetMetadataColumns (dataset-metadata-columns.ts, both enrich + version-DOI callers) -> admin backfill sweep -> /datasets projection + filter -> CLI flag.

Phases

  1. Schema: D1 columns on datasets + dataset_versions, indexes, type/projection plumbing (NULL until populated).
  2. Detection/compute: probeHed() + thread through compute/write for both callers; write per-version row + denormalized latest.
  3. Backfill sweep: POST /admin/datasets/hed-sweep with hed_checked_at resumability + nemar admin subcommand.
  4. API filter + CLI: /datasets ?has_hed=1 server filter + projection; nemar dataset list/search --hed + HED badge.

Dependent follow-up (separate repo)

nemarOrg/website: send has_hed in filterStateToAPIQuery, add has_hed to the Dataset type, make the checkbox live. Tracked separately.

Activity

  1. added a commit that references this issue on Jun 30, 2026
  2. neuromechanist commented on Jun 30, 2026

    @neuromechanist
    ContributorAuthor

    Epic complete. All 4 phases (#870-#873) merged into the epic branch; epic merged into dev via PR #878 (regular merge, preserving phase history). Cross-phase finalize review fixed 2 seam bugs. Follow-up: nemarOrg/website#139 wires the has_hed filter on the website.

  3. added a commit that references this issue on Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions