Repository navigation
Bound audit and catalog payloads to counts and pointers - #1190
Merged
Merged
Conversation
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1190Version: 🤖 Updated on each push. |
This was referenced Sep 2, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1188. Implements #1189.
The production D1 backup could not be restored: 15 single-row INSERT statements exceeded D1's ~100 KB statement limit (
SQLITE_TOOBIG), because two columns inlined an unbounded per-file list. This PR replaces those payloads with flags, counts, and a pointer to the artifact that already owns the detail, and bounds the audit write path so the next payload of this shape cannot recreate the failure.What changed
Write paths
import_verify_forcedaudit rows now storeintegrityAuditSummary()(services/import-integrity.ts):complete, the count/byte scalars,missing_count,zero_byte_count, anddetail_ref: ".nemar/availability-report.json"instead ofJSON.stringify(verified)with its fullmissingKeys/zeroByteKeysarrays (largest production row: 12,397 keys, 1.15 MB). The per-path detail already lives, richer, in the availability report (Phase 1: Generate + write .nemar/availability-report.json (single dataset) #1000).import_reclassified_incompletewrite (services/import-retry.ts) wrote the same unbounded shape; it now stores the same summary. The spec named only the verify route, but this second writer would have recreated the problem.datasets.zarr_data_failuresnow stores{"count": N, "detail_ref": "zarr/index.json"}instead of the sanitized entries array (largest: 877 entries, 178 KB). The per-entry detail is owned by the published Zarr index'sfailureslist. The clear-on-clean-rebuild behavior is unchanged.auditLogParamsbounds everydetailspayload atAUDIT_DETAILS_MAX_BYTES(16 KB, UTF-8), replacing an oversized one with a truncation marker (audit_details_truncated,original_bytes,head). AllauditLogStatementwriters inherit it; the two import services' hand-rolled audit INSERTs were converged ontoauditLogStatementso they do too. Other pre-Decompose Phase 1: Split backend admin.ts by domain #903 hand-rolled audit INSERTs still bypass the helper; they write small fixed-shape payloads today.Migration 0074 rewrites the rows already stored, deriving the counts from the arrays it drops (
json_array_length) rather than zeroing them, and marks rewritten payloads withcompacted_by: "migration_0074". Scoped by payload shape, not action: both integrity arrays must be present for the audit rewrite, andzarr_data_failuresmust be a top-level JSON array. NULL, non-JSON, already-compact, and other-shape rows are left byte-identical. Naturally idempotent. Not applied to any live database.Docs: ADR 0036 (operational rows carry counts and pointers, not per-file lists), plus a note in the disaster-recovery section of
.context/systems-inventory.mdthat a restore is only as good as its largest statement.Disclosure: undeclared passthrough shape change
zarr_data_failuresis not declared inshared/contract/dataset.tsand is read by no production code (verified: it appears only in the three UPDATE statements inroutes/callbacks/zarr-ready.ts). It reaches clients only through the detail route'sSELECT d.*passthrough (routes/datasets/catalog.ts), the same undeclared-passthrough class #1185 already trimmed. Any external consumer scraping that field from the detail response will see a{count, detail_ref}object (withcompacted_byon migrated rows) instead of an array.Acceptance
After 0074, no
audit_logordatasetsrow renders a backup statement over 95,000 bytes. The migration test seeds the production worst cases (12,397 keys / 877 entries), proves the rendered pre-migration statements exceed the bound, and pins the post-migration statements under it.Tests
Real engine throughout (bun:sqlite + the production migrations/SQL, no mocks):
backend/test/compact-oversized-payloads-migration.test.ts- 0074 applied statement-at-a-time; counts survive and are correct, leave-alone classes byte-identical, compacted rows distinguishable, re-run is a no-op, acceptance pin.backend/test/bounded-audit-payloads.test.ts- theauditLogStatementbound (marker, exact-limit passthrough, UTF-8-byte measurement), the verify audit write, and the zarr-ready callback driven through the real Hono handler.test/zarr-failure-tracking.test.tsupdated to the summary shape.The verify route itself still cannot be driven past
verifyDatasetVersionS3without live AWS credentials (its S3 listing hardcodes the AWS host; same constraint documented inimports-verify-route.test.ts), so the audit write is exercised via the extractedimportVerifyAuditStatement; the route-to-function call link and the sweep's summary call site are the uncovered residue.Every test was mutation-proved, one perturbation at a time (mutation applied and confirmed, targeted test failed, reverted): zeroed
missing_countderivation, zeroed zarrcountderivation, loosened rewrite predicate (leave-alone rows), disabled audit rewrite (acceptance pin fails), droppedcompacted_bymarker, restoredJSON.stringify(verified), bypassed the details bound, chars-instead-of-bytes measurement, boundary off-by-one, restored the zarr array storage, dropped the clear-on-clean branch.Gates:
bun test(3898 pass; the 51 failures are the pre-existing live-server-dependent API/CLI suites, byte-identical to theorigin/devbaseline set),bun run lint,bun run typecheckall clean.