Skip to content

Surface explicit downstream skip/failure reasons (zarr per-recording, archive) + verify manifest scaling #909

Closed

Description

@neuromechanist

Part of epic #896. Corrected after verifying the actual architecture (the original framing of this issue was wrong — see below).

Correction: zarr conversion is NOT a GitHub-Actions/runner-cap problem

Zarr conversion runs on SDSC Hallu, not GitHub Actions — moved there in #689/#690/#692 precisely because Actions couldn't sustain bulk/backfill. The engine (scripts/hallu-zarr.sh, queue-driven via zarr_queue.py, cron 30 * * * *):

  • Streams each recording's annex blob just-in-time from S3, converts in parallel on NVMe scratch, and frees it — it never holds the whole dataset, so dataset/file size is not a local-disk or time-cap constraint. Multi-TB datasets convert incrementally.
  • Is crash-safe/resumable (SQLite queue, flock across ticks, inprogress→pending on crash) and already classifies failures: deterministic-data (a recording biosigIO can't read, e.g. MaxShield → marked failed) vs infra (exponential backoff + retry to max-attempts).
  • Auto-dispatch to Actions is gated OFF (ZARR_AUTODISPATCH default off, webhooks.ts:2593); run-generate-zarr.yml is manual-recovery only.

So fixing #884/#886 does not endanger zarr conversion, and the "6h/120-min runner cap" premise is retracted. Archive already has an explicit scale policy too (archive-policy.ts: >100 GiB or >200k files → skip zip, direct-download fallback with a reason).

What is genuinely still open (narrower)

  1. Zarr: explicit size-based skip. The queue classifies unreadable (deterministic-data) and infra failures, but a readable-but-pathologically-large recording (exhausts worker RAM/NVMe under parallelism) just retries to max-attempts and lands as a generic failure. Add an explicit skipped: too-large classification with a reason, distinct from unreadable/infra.
  2. Surface skip/failure reasons in the served shape. Per-recording zarr skip/failure reasons and the dataset-level archive-skip reason should appear in the catalog/landing response so the website can show "preview not available (too large) — direct download available" instead of a blank or generic-failed panel. This ties into the Epic: Platform hardening + website↔CLI contract (arch-review follow-ups, 2026-07) #896 contract work (the reason is part of the served contract).
  3. Verify manifest / version-DOI scaling. Confirm these run off-Worker (central workflow) and are not file-count-bound at extreme scale (the import forensics noted a >60 s Worker file-count bound — verify whether the central-workflow move already resolved it; if not, chunk/move it). Verify, do not assume broken.

Not in scope

Activity

  1. added
    P1High priority - important features, significant bugs
    backendBackend/API related
    on Jul 2, 2026
  2. changed the title [-]Scale-aware downstream workflows: zarr/manifest must succeed or skip explicitly at large-dataset scale (generalize archive-policy)[/-] [+]Surface explicit downstream skip/failure reasons (zarr per-recording, archive) + verify manifest scaling[/+] on Jul 2, 2026
  3. neuromechanist commented on Jul 7, 2026

    @neuromechanist
    ContributorAuthor

    Fix in progress: nemarDatasets/.github#62 (the zarr driver lives there; Hallu clones it at main).

    Addresses the two observed production failures on the Hallu node (62 GB, ZARR_JOBS=8):

    • split-FIF misrouting → OOM cascade (crossmodal MEG sub-023..028 all 'worker crashed' in one batch): fixed by summing the whole split chain so it routes to the bounded-memory streamer.
    • no memory guard → EDF/BDF/SET OOM + infra retry-burn (on006269 rc=1 across days): fixed by a per-recording projected-memory preflight that SKIPS before the load (deterministic recording_too_large, surfaced in index.json, no retry). Budget = usable_RAM / --jobs, so a manual --jobs 1 re-run converts medium recordings.

    The real fix so large EDF/BDF/SET convert (streaming reader) split to 944.

  4. neuromechanist commented on Sep 2, 2026

    @neuromechanist
    ContributorAuthor

    Zarr half of item 2 is now tracked concretely as #1197 under epic #1181 (per-dataset coverage: every raw recording accounted for in index.json, detail off-database). The shared schema for these reports is the later-effort issue #1198.

  5. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    Closing. Item 1 landed as the recording_too_large classification (#1114, scripts/zarr/generate_zarr.py). Item 2: archive_skip_reason is served (#760, backend/src/routes/data.ts and page-bundle.ts) and the Zarr half is delivered by index v3 coverage under epic #1181 (#1197), with the shared report schema tracked in #1198. Item 3: manifest and version-DOI generation dispatch to the central workflow off-Worker (backend/src/routes/callbacks/version-doi.ts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priority - important features, significant bugsbackendBackend/API related

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions