Repository navigation
Phase 3: >100GB archive policy — skip zip, steer to direct download (#752) - #760
Conversation
Datasets over 100 GiB (or 200k files) skip the zip build that can't finish in
run-generate-archive.yml's 60-min cap, and steer to the range-resumable per-file
direct download.
- Migration 0043: additive archive_skip_reason TEXT (no CHECK; avoids recreating
the FTS-backed datasets table just to add a 'skipped' enum value). skipped =
archive_skip_reason IS NOT NULL; archive_status stays NULL.
- services/archive-policy.ts: ARCHIVE_MAX_BYTES/FILES + pure shouldSkipArchive.
- /webhooks/archive-ready: accept status='skipped' + reason -> record reason, no
auto-retry.
- Landing payload (data-router/data.ts) exposes archive {status,size,skip_reason}
so the website can render zip vs wget/curl recipe.
- Admin archive-sweep: oversized + no-archive candidates marked skipped (from the
cheap datasets.file_size/total_files) instead of 'absent'.
- Tests (no mocks): shouldSkipArchive boundaries; migration 0043 + skipped-state SQL.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR760Version: 🤖 Updated on each push. |
From the 3-agent /review-pr pass:
- archive-ready 'ready' (+ admin sweep ready) now clears archive_skip_reason so a
size-reduced re-publish that builds a zip drops the stale direct-download recipe.
- page-bundle.ts threads archive {status,size,skip_reason} into its landing payload
(was always-null for the bundle endpoint).
- archive-sweep D1 error now names the branch (ready vs skip/absent) so a dropped
skip_reason write isn't masked as a 'ready' failure.
- archive-ready 404 log includes status.
- Tests: buildLandingPayload archive output (3 cases) + ready-clears-skip_reason.
/review-pr summary (3 Sonnet agents: code-reviewer, silent-failure-hunter, pr-test-analyzer)All findings addressed. CRITICAL (workflow, convergent) — skip-callback dead-end. If the preflight's HIGH (workflow) — non-array manifest. HIGH (workflow) — invisible 404 fail-open. Manifest-404 fail-open was Important (backend) — stale skip_reason. Important (backend) — page-bundle. Medium — archive-sweep D1 error names the branch (ready vs skip/absent) so a dropped skip_reason write isn't masked as "ready"; archive-ready 404 log includes Tests — added Confirmed clean by review (no change)
|
eaaed23
into
feature/issue-749-epic-import-robustness
Summary
run-generate-archive.ymlhas a 60-min cap and no size guard, so large datasets stream into a zip forever and get cancelled (the whole 2026-06-14 batch: on007523 478 GB cancelled at ~24%, on005752 680 GB / on007524 321 GB never produced a zip). This adds a size policy: datasets over 100 GiB (or 200k files) skip the zip build and the website steers users to the range-resumable per-file direct download (data.nemar.org/<id>/<v>/manifest.jsonbytes_url, verified 206/wget -c).Backend (this PR)
archive_skip_reason TEXT(no CHECK). Adding'skipped'to the 0036archive_statusCHECK would require recreating the centraldatasetstable (FTS5 + triggers) — high risk; the additive column matches the 0040–0042 pattern. skipped =archive_skip_reason IS NOT NULL(archive_status stays NULL).services/archive-policy.ts:ARCHIVE_MAX_BYTES/ARCHIVE_MAX_FILES+ pureshouldSkipArchive(shared with the CI preflight + admin sweep)./webhooks/archive-ready: acceptstatus:"skipped"+reason→ record reason, no auto-retry.data-router.ts/data.ts) exposesarchive {status,size,skip_reason}.skipped(from cheapdatasets.file_size/total_files) instead ofabsent.shouldSkipArchiveboundaries; migration 0043 + skipped-state SQL. 247 backend tests pass,tsc+biomeclean.Cross-repo companions
run-generate-archive.yml): apreflightjob reads the manifest totals and, over threshold, POSTsarchive-readyskipped+ exits before the build; gatedarchivejob vianeeds/if; workflow concurrency group; multipartqueueSize4→6.LandingPayload.archive; ActionBar renders awget/curlrecipe (copy button) instead of the zip button whenskip_reasonis set. Only the skipped state changes behavior (non-regressive).Closes #752
Part of epic #749
Test plan
bun test(247 pass);tsc+biomeclean; websiteastro checkclean.