Skip to content

Bound audit and catalog payloads to counts and pointers - #1190

Merged
neuromechanist merged 7 commits into
devfrom
feature/issue-1189-bounded-audit-payloads
Sep 2, 2026
Merged

neuromechanist merged 7 commits into
devfrom
feature/issue-1189-bounded-audit-payloads

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Fixes #1188. Implements #1189.

The production D1 backup could not be restored: 15 single-row INSERT statements exceeded D1's ~100 KB statement limit (SQLITE_TOOBIG), because two columns inlined an unbounded per-file list. This PR replaces those payloads with flags, counts, and a pointer to the artifact that already owns the detail, and bounds the audit write path so the next payload of this shape cannot recreate the failure.

What changed

Write paths

  • import_verify_forced audit rows now store integrityAuditSummary() (services/import-integrity.ts): complete, the count/byte scalars, missing_count, zero_byte_count, and detail_ref: ".nemar/availability-report.json" instead of JSON.stringify(verified) with its full missingKeys/zeroByteKeys arrays (largest production row: 12,397 keys, 1.15 MB). The per-path detail already lives, richer, in the availability report (Phase 1: Generate + write .nemar/availability-report.json (single dataset) #1000).
  • The reclassification sweep's import_reclassified_incomplete write (services/import-retry.ts) wrote the same unbounded shape; it now stores the same summary. The spec named only the verify route, but this second writer would have recreated the problem.
  • datasets.zarr_data_failures now stores {"count": N, "detail_ref": "zarr/index.json"} instead of the sanitized entries array (largest: 877 entries, 178 KB). The per-entry detail is owned by the published Zarr index's failures list. The clear-on-clean-rebuild behavior is unchanged.
  • auditLogParams bounds every details payload at AUDIT_DETAILS_MAX_BYTES (16 KB, UTF-8), replacing an oversized one with a truncation marker (audit_details_truncated, original_bytes, head). All auditLogStatement writers inherit it; the two import services' hand-rolled audit INSERTs were converged onto auditLogStatement so they do too. Other pre-Decompose Phase 1: Split backend admin.ts by domain #903 hand-rolled audit INSERTs still bypass the helper; they write small fixed-shape payloads today.

Migration 0074 rewrites the rows already stored, deriving the counts from the arrays it drops (json_array_length) rather than zeroing them, and marks rewritten payloads with compacted_by: "migration_0074". Scoped by payload shape, not action: both integrity arrays must be present for the audit rewrite, and zarr_data_failures must be a top-level JSON array. NULL, non-JSON, already-compact, and other-shape rows are left byte-identical. Naturally idempotent. Not applied to any live database.

Docs: ADR 0036 (operational rows carry counts and pointers, not per-file lists), plus a note in the disaster-recovery section of .context/systems-inventory.md that a restore is only as good as its largest statement.

Disclosure: undeclared passthrough shape change

zarr_data_failures is not declared in shared/contract/dataset.ts and is read by no production code (verified: it appears only in the three UPDATE statements in routes/callbacks/zarr-ready.ts). It reaches clients only through the detail route's SELECT d.* passthrough (routes/datasets/catalog.ts), the same undeclared-passthrough class #1185 already trimmed. Any external consumer scraping that field from the detail response will see a {count, detail_ref} object (with compacted_by on migrated rows) instead of an array.

Acceptance

After 0074, no audit_log or datasets row renders a backup statement over 95,000 bytes. The migration test seeds the production worst cases (12,397 keys / 877 entries), proves the rendered pre-migration statements exceed the bound, and pins the post-migration statements under it.

Tests

Real engine throughout (bun:sqlite + the production migrations/SQL, no mocks):

  • backend/test/compact-oversized-payloads-migration.test.ts - 0074 applied statement-at-a-time; counts survive and are correct, leave-alone classes byte-identical, compacted rows distinguishable, re-run is a no-op, acceptance pin.
  • backend/test/bounded-audit-payloads.test.ts - the auditLogStatement bound (marker, exact-limit passthrough, UTF-8-byte measurement), the verify audit write, and the zarr-ready callback driven through the real Hono handler.
  • test/zarr-failure-tracking.test.ts updated to the summary shape.

The verify route itself still cannot be driven past verifyDatasetVersionS3 without live AWS credentials (its S3 listing hardcodes the AWS host; same constraint documented in imports-verify-route.test.ts), so the audit write is exercised via the extracted importVerifyAuditStatement; the route-to-function call link and the sweep's summary call site are the uncovered residue.

Every test was mutation-proved, one perturbation at a time (mutation applied and confirmed, targeted test failed, reverted): zeroed missing_count derivation, zeroed zarr count derivation, loosened rewrite predicate (leave-alone rows), disabled audit rewrite (acceptance pin fails), dropped compacted_by marker, restored JSON.stringify(verified), bypassed the details bound, chars-instead-of-bytes measurement, boundary off-by-one, restored the zarr array storage, dropped the clear-on-clean branch.

Gates: bun test (3898 pass; the 51 failures are the pre-existing live-server-dependent API/CLI suites, byte-identical to the origin/dev baseline set), bun run lint, bun run typecheck all clean.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1190

Version: 0.9.10-PR1190.079c305
Commit: 079c305


🤖 Updated on each push.

This branch was previously deployed

1 inactive deployment
npm — 079c305e Deployed Sep 2, 2026 by neuromechanist via publish #2147
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant