Skip to content

Heads-up: zarr_data_failures changes from an array to a {count, detail_ref} summary #1191

Closed

Description

@neuromechanist

Heads-up for the observability dashboard, from PR #1190 (issue #1189, fixing #1188).

datasets.zarr_data_failures changes shape on the API. It used to be a JSON array of per-file Zarr conversion failures; it is now a bounded summary object:

{"count": 877, "detail_ref": "zarr/index.json"}

The field is not declared in shared/contract/dataset.ts, so this is not a contract break, but it is served incidentally by GET /datasets/:id through that route's SELECT d.* passthrough, which means a consumer can be reading it today. Anything doing .length or .map() on it will break.

Why it changed

The array was unbounded: one row held 877 entries at 178 KB, and across 199 rows it accounted for about 29 percent of all datasets data. Together with a similar array in audit_log, it pushed 16 rows past D1's single-statement size limit, which meant the production database backup could not be restored at all (#1188). Verified by attempting a real restore, which aborted with statement too long: SQLITE_TOOBIG.

The full failure list was already published in the Zarr index.json, which is where detail_ref now points, so no information is lost - it simply is not duplicated into the operational database any more.

What a consumer should do

If the dashboard shows a failure count, read count directly; that is the number it was previously deriving from the array's length. If it lists individual failures, read them from the dataset's published Zarr index.json failures list, which is richer than the copy that was in D1.

Existing rows were compacted in place by migration 0074_compact_oversized_payloads.sql, preserving the original counts, so count is accurate for historical rows too.

Activity

  1. neuromechanist commented on Sep 2, 2026

    @neuromechanist
    ContributorAuthor

    Filed under epic #1181 (zarr serving next phase); closest sibling is #1062, which covers exposing Zarr state in the public catalog.

    Context: this shape change is already live on production as of the 0.9.10 cutover (#1182). zarr_data_failures now serves {"count": N, "detail_ref": "zarr/index.json"} instead of the raw array. Production currently has 199 datasets carrying it, and the counts were preserved through the migration, so a consumer reading count gets the same number it was previously deriving from the array's length.

  2. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1201 (zarr_data_failures is now the {count, detail_ref} summary declared in shared/contract/dataset.ts; the website reader in nemarOrg/website#278 consumes it). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions