Skip to content

Phase 3: >100GB archive policy — skip zip, steer to direct download (#752) - #760

Merged
neuromechanist merged 2 commits into
feature/issue-749-epic-import-robustnessfrom
feature/issue-752-phase3-archive-policy
Jun 16, 2026
Merged

neuromechanist merged 2 commits into
feature/issue-749-epic-import-robustnessfrom
feature/issue-752-phase3-archive-policy

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

run-generate-archive.yml has a 60-min cap and no size guard, so large datasets stream into a zip forever and get cancelled (the whole 2026-06-14 batch: on007523 478 GB cancelled at ~24%, on005752 680 GB / on007524 321 GB never produced a zip). This adds a size policy: datasets over 100 GiB (or 200k files) skip the zip build and the website steers users to the range-resumable per-file direct download (data.nemar.org/<id>/<v>/manifest.json bytes_url, verified 206/wget -c).

Backend (this PR)

  • Migration 0043: additive archive_skip_reason TEXT (no CHECK). Adding 'skipped' to the 0036 archive_status CHECK would require recreating the central datasets table (FTS5 + triggers) — high risk; the additive column matches the 0040–0042 pattern. skipped = archive_skip_reason IS NOT NULL (archive_status stays NULL).
  • services/archive-policy.ts: ARCHIVE_MAX_BYTES/ARCHIVE_MAX_FILES + pure shouldSkipArchive (shared with the CI preflight + admin sweep).
  • /webhooks/archive-ready: accept status:"skipped" + reason → record reason, no auto-retry.
  • Landing payload (data-router.ts / data.ts) exposes archive {status,size,skip_reason}.
  • Admin archive-sweep marks oversized-with-no-archive candidates skipped (from cheap datasets.file_size/total_files) instead of absent.
  • Tests (no mocks): shouldSkipArchive boundaries; migration 0043 + skipped-state SQL. 247 backend tests pass, tsc + biome clean.

Cross-repo companions

  • nemarDatasets/.github (run-generate-archive.yml): a preflight job reads the manifest totals and, over threshold, POSTs archive-ready skipped + exits before the build; gated archive job via needs/if; workflow concurrency group; multipart queueSize 4→6.
  • nemarOrg/website: LandingPayload.archive; ActionBar renders a wget/curl recipe (copy button) instead of the zip button when skip_reason is set. Only the skipped state changes behavior (non-regressive).

Closes #752
Part of epic #749

Test plan

  • bun test (247 pass); tsc + biome clean; website astro check clean.
  • E2E (gated): dispatch generate-archive for a >100 GB set → preflight POSTs skipped, no zip; website shows the recipe. A <100 GB set still builds a zip.
  • Backfill (gated): mark on005752/on007523/on007524 skipped; on005385 (79 GB) re-dispatch to get a zip.

Datasets over 100 GiB (or 200k files) skip the zip build that can't finish in
run-generate-archive.yml's 60-min cap, and steer to the range-resumable per-file
direct download.

- Migration 0043: additive archive_skip_reason TEXT (no CHECK; avoids recreating
  the FTS-backed datasets table just to add a 'skipped' enum value). skipped =
  archive_skip_reason IS NOT NULL; archive_status stays NULL.
- services/archive-policy.ts: ARCHIVE_MAX_BYTES/FILES + pure shouldSkipArchive.
- /webhooks/archive-ready: accept status='skipped' + reason -> record reason, no
  auto-retry.
- Landing payload (data-router/data.ts) exposes archive {status,size,skip_reason}
  so the website can render zip vs wget/curl recipe.
- Admin archive-sweep: oversized + no-archive candidates marked skipped (from the
  cheap datasets.file_size/total_files) instead of 'absent'.
- Tests (no mocks): shouldSkipArchive boundaries; migration 0043 + skipped-state SQL.
@github-actions

github-actions Bot commented Jun 16, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR760

Version: 0.8.53-PR760.a9af9f2
Commit: a9af9f2


🤖 Updated on each push.

From the 3-agent /review-pr pass:
- archive-ready 'ready' (+ admin sweep ready) now clears archive_skip_reason so a
  size-reduced re-publish that builds a zip drops the stale direct-download recipe.
- page-bundle.ts threads archive {status,size,skip_reason} into its landing payload
  (was always-null for the bundle endpoint).
- archive-sweep D1 error now names the branch (ready vs skip/absent) so a dropped
  skip_reason write isn't masked as a 'ready' failure.
- archive-ready 404 log includes status.
- Tests: buildLandingPayload archive output (3 cases) + ready-clears-skip_reason.
@neuromechanist

Copy link
Copy Markdown
Contributor Author

/review-pr summary (3 Sonnet agents: code-reviewer, silent-failure-hunter, pr-test-analyzer)

All findings addressed.

CRITICAL (workflow, convergent) — skip-callback dead-end. If the preflight's archive-ready status=skipped POST failed, the build was skipped but archive_skip_reason was never recorded → NULL status + NULL reason = "missing archive" forever, and the green preflight hid it. Now exit 1 (red, re-runnable); the gated archive job is skipped on the failed dep, so no doomed build; re-run is idempotent.

HIGH (workflow) — non-array manifest. jq on a non-array silently yielded 0/0 → under-threshold → it built the very oversized archive we meant to skip. Added a jq type == "array" guard (fail open + ::error::).

HIGH (workflow) — invisible 404 fail-open. Manifest-404 fail-open was ::warning:: (invisible on a green job); now ::error::.

Important (backend) — stale skip_reason. archive-ready 'ready' and the admin sweep's ready branch now clear archive_skip_reason, so a size-reduced re-publish that builds a zip drops the direct-download recipe (was showing both).

Important (backend) — page-bundle. buildPageBundle now threads archive {status,size,skip_reason} into its landing payload (was always-null for the bundle endpoint).

Medium — archive-sweep D1 error names the branch (ready vs skip/absent) so a dropped skip_reason write isn't masked as "ready"; archive-ready 404 log includes status.

Tests — added buildLandingPayload archive output (3 cases, the page-bundle bug class) + ready-clears-skip_reason. 251 backend tests pass, tsc + biome clean.

Confirmed clean by review (no change)

  • archiveRetrySweep filters status='failed', so skipped datasets (status NULL) are never re-dispatched.
  • skipped branch never schedules the failed-retry waitUntil (retry stays null); 404 guard intact.
  • jq -nc payload is injection-safe; bash 64-bit handles 680 GB; concurrency group keyed on (id,version); website recipe URL + non-regressive zip gating.

@neuromechanist
neuromechanist merged commit eaaed23 into feature/issue-749-epic-import-robustness Jun 16, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-752-phase3-archive-policy branch June 16, 2026 15:29

This branch was previously deployed

1 inactive deployment
npm — a9af9f25 Deployed Jun 16, 2026 by neuromechanist via publish #1403
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant