Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# ADR 0036: Operational rows carry counts and pointers, not per-file lists

**Status:** accepted
**Date:** 2026-09-02
**Owner:** Seyed Yahya Shirazi

## Context

The hourly production D1 backup could not be restored (#1188):
`restore-remote.sh` aborted with `SQLITE_TOOBIG` on 15 single-row INSERT
statements over D1's ~100 KB statement limit.
Every offender was a row column that inlined an unbounded per-file list,
so row size scaled with a dataset's file count:
`audit_log.details` on forced import verifications inlined the full
`missingKeys`/`zeroByteKeys` arrays (largest: 12,397 annex keys, 1.15 MB,
~88% of the entire audit log),
and `datasets.zarr_data_failures` inlined the per-recording conversion
failure array (largest: 877 entries, 178 KB;
~29% of all `datasets` data).
In both cases the inlined detail duplicated a richer artifact that already
existed outside D1:
`.nemar/availability-report.json` on the dataset repo
(path-keyed with a per-entry reason, versioned in git)
and the published Zarr index's `failures` list
(`<dataset>/zarr/index.json` in the serving bucket).

## Decision

A row in an operational table (audit log, catalog bookkeeping) carries
**flags, counts, and a pointer** —
booleans and totals that answer "how many" and "did it complete",
plus a `detail_ref` naming the artifact that owns the per-file detail.
The detail itself is never inlined, and never merely truncated:
a truncated list is still the wrong data in the wrong place,
and silently discards everything past the cut.

Concretely (#1189):
integrity audit writes store `integrityAuditSummary()`
(services/import-integrity.ts) instead of the raw result;
`zarr_data_failures` stores `{count, detail_ref}`;
migration 0074 rewrote the pre-existing rows into the same summaries,
deriving the counts from the arrays it dropped;
and `auditLogParams` (db/audit-log.ts) bounds every `details` payload at
`AUDIT_DETAILS_MAX_BYTES`,
replacing an oversized one with a truncation marker,
so the next unbounded payload shape cannot recreate the failure.

## Consequences

- Row size — and therefore backup-statement size — is independent of how
many files a dataset has, which is exactly the condition that makes a
restore work (a restore is only as good as its largest statement).
- Answering "which files" requires following the pointer to the owning
artifact rather than querying D1.
That artifact is richer than the dropped copy was,
but it is a second hop, and for the availability report it requires repo
access.
- Compacted historical rows carry `compacted_by: "migration_0074"`;
their per-key lists are gone from D1.
The counts survive, and the reports/indexes hold the current detail —
though not the historical per-run snapshots the audit rows once froze.
- Any new column or audit payload that wants to store a per-item list must
instead store counts plus a pointer,
or argue its way past the write-time bound in an ADR superseding this one.
- Hand-rolled `INSERT INTO audit_log` call sites predating #903 bypass the
bound until they converge on `auditLogStatement`.
Three were converged in #1189: the two import services,
and the owner draft-delete route (`routes/datasets/draft-delete.ts`),
which embedded `steps.s3.failed` from `deleteDatasetCascade` --
and `deleteObjects` pushes EVERY key of a failed batch into that array,
so a single S3 outage during an owner delete could reproduce
exactly the unbounded shape this ADR exists to prevent.
It was found in review, after an earlier draft of this ADR asserted
the remaining sites were all small and fixed-shape; that claim was wrong,
which is itself the argument for the write-time bound rather than
a per-call-site audit.
The sites still hand-rolled today (`index.ts` scheduled_cleanup, capped at
`MAX_DELETIONS_PER_RUN`; `auth.ts`; `publication.ts`; `collaborators.ts`;
`upload.ts`; `dataset-search.ts`; `auto-import.ts`)
were each checked and write small fixed-shape payloads,
but that is a property of today's code, not a guarantee --
converging them is the durable fix.

- `zarr_data_failures` changes shape on the wire, from an array to
`{count, detail_ref}`. It is not declared in `shared/contract/dataset.ts`,
and nothing in this repo reads it, but the detail route serves it
incidentally through `SELECT d.*` (`routes/datasets/catalog.ts`),
so it IS wire-visible. The observability dashboard in `nemarOrg/website`
is the known consumer and lives outside this repo;
a reader doing `.length` or `.map()` on it will break.
Tracked as a heads-up against that repo rather than blocked here,
because the count it now carries is the number that dashboard actually shows.

## Alternatives considered

- **Truncate the arrays to N entries:** keeps wrong data in the wrong
place, silently discards entries N+1 onward, and leaves row size coupled
to a cap instead of to the row's actual job. Rejected by design review.
- **Fix it in the backup format** (bound statements, chunked UPDATEs,
parameterized loads): treats the symptom; every D1 reader and the
dashboard still pays for megabyte rows, and the duplicate detail still
drifts from the owning artifact. Kept only as a defense-in-depth idea for
the backup repo.
- **A separate detail table:** normalizes the duplicate instead of removing
it; the artifact outside D1 remains the source of truth, so D1 would hold
a second copy either way.

## Receipts

- #1188 — the restore failure and its statement-size measurements.
- #1189 — the fix: summaries, write-time bound, migration 0074.
- ADR 0004 — the backup this exists to keep restorable.
- ADR 0034 — the same instinct applied to column count rather than row size.
1 change: 1 addition & 0 deletions .context/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ invisible one — which is why it is checked rather than trusted.
- [ADR 0033](0033-the-zarr-queue-stamps-the-engine-that-converted-each-dataset.md) - The Zarr queue stamps the engine that converted each dataset; pre-existing rows are declared current
- [ADR 0034](0034-datasets-stays-one-table-under-a-column-budget.md) - `datasets` stays one table, under an enforced column budget
- [ADR 0035](0035-sweep-stamps-live-in-one-json-column.md) - Sweep bookkeeping stamps live in one JSON column
- [ADR 0036](0036-operational-rows-carry-counts-and-pointers-not-per-file-lists.md) - Operational rows carry counts and pointers, not per-file lists

## Backfill note (2026-07-31)

Expand Down
9 changes: 9 additions & 0 deletions .context/systems-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,3 +236,12 @@ Both recovery tools live in the backup repo, not here:
Verifies sha256 and row counts, and refuses production without `--force-prod`.
- `scripts/run-local.sh --nemar-cli <path>` — loads a real snapshot into a local
miniflare D1 and runs `wrangler dev` against it.

**A restore is only as good as its largest statement.** The backup renders one
INSERT per row and D1 refuses any statement over ~100 KB on restore
(`SQLITE_TOOBIG`), so a single oversized row makes the whole backup
unrestorable — and nothing at backup time detects it (#1188: 15 such
statements, found only by rehearsing a real restore). Row payloads are
therefore bounded at the write path (ADR 0036: counts and pointers, never
inline per-file lists; `AUDIT_DETAILS_MAX_BYTES` in `backend/src/db/audit-log.ts`),
and migration 0074 compacted the rows that had already outgrown the limit.
42 changes: 41 additions & 1 deletion backend/src/db/audit-log.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,51 @@ export interface AuditLogEntry {
export const AUDIT_LOG_INSERT_SQL = `INSERT INTO audit_log (user_id, action, resource_type, resource_id, details)
VALUES (?, ?, ?, ?, ?)`;

/**
* Upper bound, in UTF-8 bytes, on an audit row's `details` payload (#1189).
*
* The D1 backup renders each row as one INSERT statement, and D1 refuses to
* execute a statement over ~100 KB on restore (SQLITE_TOOBIG, #1188) -- so an
* unbounded `details` column is a row that can be written but never restored.
* 16 KB leaves a wide margin under that limit even after the backup's SQL
* quote-escaping (worst case doubles the payload), while being far above any
* legitimate summary payload: an audit row carries flags, counts, and a
* pointer to the artifact that owns the detail, not the detail itself.
*/
export const AUDIT_DETAILS_MAX_BYTES = 16_384;

/** Characters of an oversized payload kept in the truncation marker -- enough
* to expose a JSON payload's leading summary fields for identification. */
const AUDIT_DETAILS_HEAD_CHARS = 2_000;

/**
* Bound a `details` payload to {@link AUDIT_DETAILS_MAX_BYTES}: a payload at
* or under the limit passes through untouched; an oversized one is replaced
* by a small JSON marker recording that it was dropped, its original size,
* and its head. This is deliberately lossy -- by the time a payload is this
* large it is per-file detail that belongs in the artifact that owns it
* (see #1189), and preserving it here is what made the backup unrestorable.
*/
export function boundAuditDetails(details: string | null): string | null {
if (details === null) return null;
const bytes = new TextEncoder().encode(details).byteLength;
if (bytes <= AUDIT_DETAILS_MAX_BYTES) return details;
return JSON.stringify({
audit_details_truncated: true,
original_bytes: bytes,
head: details.slice(0, AUDIT_DETAILS_HEAD_CHARS),
});
}

/**
* Positional params for AUDIT_LOG_INSERT_SQL. Exported separately so the
* behavioral test can run the production SQL + marshaling against bun:sqlite
* (same approach as user-tombstone.ts, whose test uses a different driver API
* than D1).
*
* `details` is bounded here, in the shared marshaling, rather than at call
* sites: every write through auditLogStatement inherits the bound, so the
* next unbounded payload shape cannot silently recreate #1188.
*/
export function auditLogParams(
entry: AuditLogEntry,
Expand All @@ -33,7 +73,7 @@ export function auditLogParams(
entry.action,
entry.resourceType ?? null,
entry.resourceId ?? null,
entry.details ?? null,
boundAuditDetails(entry.details ?? null),
];
}

Expand Down
76 changes: 76 additions & 0 deletions backend/src/db/migrations/0074_compact_oversized_payloads.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
-- Compact the two unbounded per-file payloads already stored in D1 (#1189,
-- fixes #1188): rewrite them into the counts-and-pointer summaries the write
-- paths now produce, so no `audit_log` or `datasets` row renders a backup
-- INSERT statement over D1's ~100 KB statement limit. Restoring the hourly
-- backup aborted with SQLITE_TOOBIG on 15 such statements: 11 audit rows
-- whose `details` inlined a full integrity-check `missingKeys`/`zeroByteKeys`
-- key list (largest: 12,397 keys, 1.15 MB), and 4 datasets rows whose
-- `zarr_data_failures` inlined the per-recording failure array (largest: 877
-- entries, 178 KB).
--
-- Counts are DERIVED from the arrays being dropped (json_array_length), not
-- zeroed: the summary must answer "how many" even though "which" now lives in
-- the artifact that owns the detail -- `.nemar/availability-report.json` on
-- the dataset repo for missing content, the published Zarr index's `failures`
-- list (`<dataset>/zarr/index.json` in the serving bucket) for conversion
-- failures. Both are richer than the copies dropped here.
--
-- Each rewrite carries 'compacted_by' = 'migration_0074' so a reader can tell
-- a migrated record from one the write paths produced natively.
--
-- Scoped by payload SHAPE, not by action/status: BOTH integrity key arrays
-- must be present for the audit rewrite (that pair is the
-- DatasetVersionIntegrityResult signature, written by import_verify_forced
-- AND import_reclassified_incomplete), and `zarr_data_failures` must be a
-- top-level JSON array. Rows that are NULL, non-JSON, or already compact
-- match neither predicate and are left byte-identical.
--
-- The CASE wrapper in each WHERE is load-bearing: json_type() throws on
-- malformed JSON rather than returning NULL, and SQLite does not guarantee
-- left-to-right evaluation of AND operands, so `json_valid(x) AND
-- json_type(x, ...)` could still evaluate json_type first. CASE guarantees
-- the json_valid gate runs before any json_type call.
--
-- json(col -> '$.key') rather than json_extract(col, '$.key') for the copied
-- scalar fields: json_extract collapses JSON true/false to 1/0, while `->`
-- returns the JSON subvalue verbatim and json() re-embeds it, so `complete`
-- stays a boolean and a null `version` stays null -- byte-compatible with
-- what JSON.stringify(integrityAuditSummary(...)) writes.
--
-- Naturally idempotent: a rewritten payload no longer carries the arrays, so
-- a re-run matches zero rows.

UPDATE audit_log
SET details = json_object(
'complete', json(details -> '$.complete'),
'expectedCount', json(details -> '$.expectedCount'),
'presentCount', json(details -> '$.presentCount'),
'bytesPresent', json(details -> '$.bytesPresent'),
'declaredBytes', json(details -> '$.declaredBytes'),
'declaredFiles', json(details -> '$.declaredFiles'),
'version', json(details -> '$.version'),
'missing_count', json_array_length(details, '$.missingKeys'),
'zero_byte_count', json_array_length(details, '$.zeroByteKeys'),
'detail_ref', '.nemar/availability-report.json',
'compacted_by', 'migration_0074'
)
WHERE CASE
WHEN details IS NOT NULL AND json_valid(details)
THEN json_type(details, '$.missingKeys') = 'array'
AND json_type(details, '$.zeroByteKeys') = 'array'
ELSE 0
END;

-- No datasets trigger fires here: datasets_fts_au and datasets_embed_dirty_au
-- are UPDATE OF triggers on the search/embedding columns only.
UPDATE datasets
SET zarr_data_failures = json_object(
'count', json_array_length(zarr_data_failures),
'detail_ref', 'zarr/index.json',
'compacted_by', 'migration_0074'
)
WHERE CASE
WHEN zarr_data_failures IS NOT NULL AND json_valid(zarr_data_failures)
THEN json_type(zarr_data_failures) = 'array'
ELSE 0
END;
39 changes: 32 additions & 7 deletions backend/src/routes/admin/imports.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ import { stampDatasetIntegrity } from "../../services/dataset-metadata-columns";
import { deleteDatasetCascade } from "../../services/deletion";
import { type GitHubRepo, createRepository, deleteRepository } from "../../services/github";
import { getDatasetsToken } from "../../services/github-auth";
import { verifyDatasetVersionS3 } from "../../services/import-integrity";
import type { DatasetVersionIntegrityResult } from "../../services/import-integrity";
import { integrityAuditSummary, verifyDatasetVersionS3 } from "../../services/import-integrity";
import { IMPORT_STATUSES } from "../../services/import-recovery";
import { recoverRow } from "../../services/import-retry";
import { hasRole } from "../../types/bindings";
Expand Down Expand Up @@ -74,6 +75,35 @@ export function isValidSourceIdForDataset(
return rule ? rule(datasetId, sourceId) === true : false;
}

/**
* The `import_verify_forced` audit write, as a bound statement (#1189).
*
* `details` is the bounded {@link integrityAuditSummary} -- counts and a
* pointer to `.nemar/availability-report.json` -- never
* `JSON.stringify(verified)`, whose inlined key arrays are what made the D1
* backup unrestorable (#1188).
*
* Extracted from the verify route body so a real-D1 test can drive the
* write: the route itself cannot be exercised past verifyDatasetVersionS3
* without live AWS credentials (its S3 listing hardcodes a
* `*.s3.*.amazonaws.com` host; see imports-verify-route.test.ts), and the
* no-mocks policy rules out faking that call. Same seam discipline as
* stampDatasetIntegrity.
*/
export function importVerifyAuditStatement(
db: D1Database,
userId: number,
datasetId: string,
verified: DatasetVersionIntegrityResult,
): D1PreparedStatement {
return auditLogStatement(db, {
userId,
action: "import_verify_forced",
resourceId: datasetId,
details: JSON.stringify(integrityAuditSummary(verified)),
});
}

export function registerImportRoutes(admin: AdminRouter): void {
// ─── Import external dataset ────────────────────────────────────────────────

Expand Down Expand Up @@ -513,12 +543,7 @@ export function registerImportRoutes(admin: AdminRouter): void {
);
}

await auditLogStatement(c.env.DB, {
userId: c.get("user").id,
action: "import_verify_forced",
resourceId: datasetId,
details: JSON.stringify(verified),
}).run();
await importVerifyAuditStatement(c.env.DB, c.get("user").id, datasetId, verified).run();

// Explicit pick, not a spread: verifyDatasetVersionS3's runtime result
// carries extra bytesPresent/declaredBytes/declaredFiles/version fields
Expand Down
31 changes: 18 additions & 13 deletions backend/src/routes/callbacks/zarr-ready.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,16 @@ interface ZarrReadyBody {
* `zarr_failed_at`, so a clean run clears the failure detail and the dashboard
* can sort/filter recent failures. Defensive against missing/garbage fields so
* the always-200 callback contract holds.
*
* `dataFailuresJson` is a bounded summary -- an entry count plus a pointer --
* never the entries themselves (#1189). This column used to store the full
* sanitized `data_failures` array (largest production row: 877 entries,
* 178 KB), which pushed `datasets` rows past D1's ~100 KB statement limit in
* the single-INSERT-per-row backup and made it unrestorable (#1188). The
* per-entry detail (path/code/reason) is owned by the published Zarr index's
* `failures` list at `<dataset>/zarr/index.json` in the serving bucket
* (scripts/zarr/generate_zarr.py writes it; purge_non_raw_stores.py rewrites
* it), which is what `detail_ref` names.
*/
export function zarrFailureColumns(body: {
errors?: number;
Expand All @@ -94,23 +104,18 @@ export function zarrFailureColumns(body: {
// Typed data failures are a SUBSET of total errors; clamp so a converter bug
// (or a missing failure_count) can never render "3 data failures of 1 error".
const failureCount = Math.min(rawFailureCount, errors);
// Store only the known fields, length-capped: the values are display-only on
// the dashboard and `datasets` is heavily queried, so a malformed/huge
// data_failures item must not bloat the row's TEXT column.
const sanitized = dataFailures.map((item) => {
if (typeof item !== "object" || item === null) return {};
const i = item as Record<string, unknown>;
return {
...(typeof i.path === "string" ? { path: i.path.slice(0, 512) } : {}),
...(typeof i.code === "string" ? { code: i.code.slice(0, 64) } : {}),
...(typeof i.reason === "string" ? { reason: i.reason.slice(0, 256) } : {}),
};
});
return {
errors,
failureCount,
deterministic: body.deterministic === true ? 1 : 0,
dataFailuresJson: sanitized.length > 0 ? JSON.stringify(sanitized) : null,
// `count` is the reported entry count of the index's failures list,
// deliberately unclamped -- the clamped display count is the separate
// zarr_failure_count column (failureCount above). Row size is now
// independent of how many recordings failed.
dataFailuresJson:
dataFailures.length > 0
? JSON.stringify({ count: dataFailures.length, detail_ref: "zarr/index.json" })
: null,
hadErrors: errors > 0,
};
}
Expand Down
Loading
Loading