Skip to content

Phase 3: HED backfill sweep + admin command - #876

Merged
neuromechanist merged 2 commits into
feature/issue-869-epic-hed-detectionfrom
feature/issue-872-phase3-hed-sweep
Jun 30, 2026
Merged

neuromechanist merged 2 commits into
feature/issue-869-epic-hed-detectionfrom
feature/issue-872-phase3-hed-sweep

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Adds the one-time backfill that seeds HED for datasets predating phase 2, mirroring the channel/montage sweep (#859) with two deliberate divergences.

Backend — POST /admin/datasets/hed-sweep

  • Bounded per call (?limit= default 15, clamped [1,30]); run until remaining reaches 0; idempotent via hed_checked_at IS NULL.
  • No modality filter (the key divergence): HED lives in *_events.{json,tsv} across eeg/meg/ieeg/beh/func, so every managed dataset is a candidate.
  • Per-version write (second divergence): the candidate query also selects latest_version, and the sweep calls writeVersionHed for the published latest version so dataset_versions stays consistent with the denormalized datasets row. Skipped for unpublished datasets (no version row -> no spurious 0-row error).
  • Direct write (no updated_at/metadata_updated_at bump); marks every processed row checked so remaining converges even on probe miss/error; ?reset=1 clears probed rows.
  • Response: { processed, withHed, withoutHed, unknown, errors, remaining }.

CLI — nemar admin hed-sweep

  • Loops do…while (remaining > 0) accumulating tallies; --limit, --reset, --verbose (per-batch [batch N] lines), --json. requireAuth() gate, ~1s pacing, a progress guard that bails if remaining stops decreasing, non-zero exit on errors. API client hedSweep/hedSweepReset mirror reindexBulk.

Docs

  • nemar admin hed-sweep added to the AGENTS.md admin command list.

Out of scope

  • Historical per-version backfill of non-latest versions (would re-probe each tagged tree). Going-forward publishes already write per-version HED (phase 2). Phase 4 (Phase 4: /datasets has_hed filter + CLI --hed #873) is the ?has_hed=1 filter + CLI --hed.

Closes #872
Part of epic #869

Test plan

  • test/hed-sweep.unit.test.ts (new, real bun:sqlite + migration 0056): asserts the modality-agnostic candidate scoping (meg-only + no-modality datasets ARE candidates; catalog/sandbox excluded), the latest_version subquery, the no-updated_at-bump direct write, unknown-probe convergence, and ?reset=1.
  • writeVersionHed itself covered by backend/test/hed-write.test.ts (phase 2). Handler GitHub I/O not unit-tested (matches the channel-montage sweep test = SQL contract).
  • All HED + channel-montage-sweep suites green; Biome + backend typecheck + CLI typecheck clean.
  • The 13 cli.test.ts failures are pre-existing live-API integration tests (identical on the epic baseline), unrelated to this change.

@github-actions

github-actions Bot commented Jun 30, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR876

Version: 0.8.73-PR876.449f6ec
Commit: 449f6ec


🤖 Updated on each push.

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review summary (3 agents: code, silent-failure, tests — Sonnet)

Code review found no issues at confidence >=80 (candidate/remaining predicate consistency, latest_version subquery, hasHedInt conversion, tally classification, writeVersionHed guard, and CLI loop termination all confirmed correct). Silent-failure + tests surfaced findings, all addressed in 449f6ec:

Addressed

  • Split-state on per-dataset write (silent-failure, Important): reordered so writeVersionHed (dataset_versions) runs BEFORE the datasets hed_checked_at stamp. Now any write failure leaves the row unstamped, so a plain re-run re-probes and retries both (idempotent) — no terminal state where datasets is stamped but the per-version row is still NULL (which would have needed --reset).
  • Null remaining silently exits the CLI loop (silent-failure, Important): the loop printed remaining=0 and exited 0 on a null count (false completion). Now warns, prints remaining=unknown, and exits non-zero so a caller never reads a partial/indeterminate sweep as success.
  • Per-version write contract untested (tests, Important): added applyVersionProbe + two tests — the published latest version row is stamped (older versions stay NULL), and an unpublished dataset gets no per-version write (the latest_version guard skip).
  • Reset asymmetry untested (tests, Suggestion): the reset test now asserts dataset_versions retains its publish-time values after ?reset=1 (datasets-level clear only).
  • remaining predicate untested (tests, Suggestion): added a REMAINING_SQL mirror + an assertion that it shares the candidate scoping and decreases as rows are stamped — catches predicate drift that would break loop termination.

Confirmed sound (no change)

  • writeVersionHed 0-row path is correctly gated (hasHedInt != null && latest_version) — never fires spuriously for unpublished datasets.
  • Candidate-query / token-fetch abort paths log + 500, mirroring channel-montage-sweep.
  • The uncaught remaining COUNT query mirrors the existing channel-montage-sweep precedent (not a new regression).

Local gates: 9 sweep tests + all HED suites green; Biome + backend typecheck + CLI typecheck clean. The 13 cli.test.ts failures are pre-existing live-API integration tests (identical on the epic baseline).

@neuromechanist
neuromechanist merged commit 81e6a58 into feature/issue-869-epic-hed-detection Jun 30, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-872-phase3-hed-sweep branch June 30, 2026 02:11

This branch was previously deployed

1 inactive deployment
npm — 449f6ec0 Deployed Jun 30, 2026 by neuromechanist via publish #1637
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant