You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of epic #896. Corrected after verifying the actual architecture (the original framing of this issue was wrong — see below).
Correction: zarr conversion is NOT a GitHub-Actions/runner-cap problem
Zarr conversion runs on SDSC Hallu, not GitHub Actions — moved there in #689/#690/#692 precisely because Actions couldn't sustain bulk/backfill. The engine (scripts/hallu-zarr.sh, queue-driven via zarr_queue.py, cron 30 * * * *):
Streams each recording's annex blob just-in-time from S3, converts in parallel on NVMe scratch, and frees it — it never holds the whole dataset, so dataset/file size is not a local-disk or time-cap constraint. Multi-TB datasets convert incrementally.
Is crash-safe/resumable (SQLite queue, flock across ticks, inprogress→pending on crash) and already classifies failures: deterministic-data (a recording biosigIO can't read, e.g. MaxShield → marked failed) vs infra (exponential backoff + retry to max-attempts).
Auto-dispatch to Actions is gated OFF (ZARR_AUTODISPATCH default off, webhooks.ts:2593); run-generate-zarr.yml is manual-recovery only.
So fixing #884/#886 does not endanger zarr conversion, and the "6h/120-min runner cap" premise is retracted. Archive already has an explicit scale policy too (archive-policy.ts: >100 GiB or >200k files → skip zip, direct-download fallback with a reason).
What is genuinely still open (narrower)
Zarr: explicit size-based skip. The queue classifies unreadable (deterministic-data) and infra failures, but a readable-but-pathologically-large recording (exhausts worker RAM/NVMe under parallelism) just retries to max-attempts and lands as a generic failure. Add an explicit skipped: too-large classification with a reason, distinct from unreadable/infra.
Surface skip/failure reasons in the served shape. Per-recording zarr skip/failure reasons and the dataset-level archive-skip reason should appear in the catalog/landing response so the website can show "preview not available (too large) — direct download available" instead of a blank or generic-failed panel. This ties into the Epic: Platform hardening + website↔CLI contract (arch-review follow-ups, 2026-07) #896 contract work (the reason is part of the served contract).
Verify manifest / version-DOI scaling. Confirm these run off-Worker (central workflow) and are not file-count-bound at extreme scale (the import forensics noted a >60 s Worker file-count bound — verify whether the central-workflow move already resolved it; if not, chunk/move it). Verify, do not assume broken.
Not in scope
Zarr conversion throughput/scale (already solved by the Hallu engine).
Fix in progress: nemarDatasets/.github#62 (the zarr driver lives there; Hallu clones it at main).
Addresses the two observed production failures on the Hallu node (62 GB, ZARR_JOBS=8):
split-FIF misrouting → OOM cascade (crossmodal MEG sub-023..028 all 'worker crashed' in one batch): fixed by summing the whole split chain so it routes to the bounded-memory streamer.
no memory guard → EDF/BDF/SET OOM + infra retry-burn (on006269 rc=1 across days): fixed by a per-recording projected-memory preflight that SKIPS before the load (deterministic recording_too_large, surfaced in index.json, no retry). Budget = usable_RAM / --jobs, so a manual --jobs 1 re-run converts medium recordings.
The real fix so large EDF/BDF/SET convert (streaming reader) split to 944.
Zarr half of item 2 is now tracked concretely as #1197 under epic #1181 (per-dataset coverage: every raw recording accounted for in index.json, detail off-database). The shared schema for these reports is the later-effort issue #1198.
Closing. Item 1 landed as the recording_too_large classification (#1114, scripts/zarr/generate_zarr.py). Item 2: archive_skip_reason is served (#760, backend/src/routes/data.ts and page-bundle.ts) and the Zarr half is delivered by index v3 coverage under epic #1181 (#1197), with the shared report schema tracked in #1198. Item 3: manifest and version-DOI generation dispatch to the central workflow off-Worker (backend/src/routes/callbacks/version-doi.ts).
Activity
[-]Scale-aware downstream workflows: zarr/manifest must succeed or skip explicitly at large-dataset scale (generalize archive-policy)[/-][+]Surface explicit downstream skip/failure reasons (zarr per-recording, archive) + verify manifest scaling[/+]neuromechanist commentedon Jul 7, 2026
Fix in progress: nemarDatasets/.github#62 (the zarr driver lives there; Hallu clones it at
main).Addresses the two observed production failures on the Hallu node (62 GB, ZARR_JOBS=8):
recording_too_large, surfaced in index.json, no retry). Budget = usable_RAM / --jobs, so a manual --jobs 1 re-run converts medium recordings.The real fix so large EDF/BDF/SET convert (streaming reader) split to 944.
Address Phase 2 review findings
Phase 2: Contain the blast radius of an OOM-killed worker (#1115)
neuromechanist commentedon Sep 2, 2026
Zarr half of item 2 is now tracked concretely as #1197 under epic #1181 (per-dataset coverage: every raw recording accounted for in index.json, detail off-database). The shared schema for these reports is the later-effort issue #1198.
neuromechanist commentedon Sep 3, 2026
Closing. Item 1 landed as the recording_too_large classification (#1114, scripts/zarr/generate_zarr.py). Item 2: archive_skip_reason is served (#760, backend/src/routes/data.ts and page-bundle.ts) and the Zarr half is delivered by index v3 coverage under epic #1181 (#1197), with the shared report schema tracked in #1198. Item 3: manifest and version-DOI generation dispatch to the central workflow off-Worker (backend/src/routes/callbacks/version-doi.ts).