You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Heads-up: zarr_data_failures changes from an array to a {count, detail_ref} summary #1191
Heads-up for the observability dashboard, from PR #1190 (issue #1189, fixing #1188).
datasets.zarr_data_failures changes shape on the API. It used to be a JSON array of per-file Zarr conversion failures; it is now a bounded summary object:
{"count": 877, "detail_ref": "zarr/index.json"}
The field is not declared in shared/contract/dataset.ts, so this is not a contract break, but it is served incidentally by GET /datasets/:id through that route's SELECT d.* passthrough, which means a consumer can be reading it today. Anything doing .length or .map() on it will break.
Why it changed
The array was unbounded: one row held 877 entries at 178 KB, and across 199 rows it accounted for about 29 percent of all datasets data. Together with a similar array in audit_log, it pushed 16 rows past D1's single-statement size limit, which meant the production database backup could not be restored at all (#1188). Verified by attempting a real restore, which aborted with statement too long: SQLITE_TOOBIG.
The full failure list was already published in the Zarr index.json, which is where detail_ref now points, so no information is lost - it simply is not duplicated into the operational database any more.
What a consumer should do
If the dashboard shows a failure count, read count directly; that is the number it was previously deriving from the array's length. If it lists individual failures, read them from the dataset's published Zarr index.jsonfailures list, which is richer than the copy that was in D1.
Existing rows were compacted in place by migration 0074_compact_oversized_payloads.sql, preserving the original counts, so count is accurate for historical rows too.
Filed under epic #1181 (zarr serving next phase); closest sibling is #1062, which covers exposing Zarr state in the public catalog.
Context: this shape change is already live on production as of the 0.9.10 cutover (#1182). zarr_data_failures now serves {"count": N, "detail_ref": "zarr/index.json"} instead of the raw array. Production currently has 199 datasets carrying it, and the counts were preserved through the migration, so a consumer reading count gets the same number it was previously deriving from the array's length.
Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1201 (zarr_data_failures is now the {count, detail_ref} summary declared in shared/contract/dataset.ts; the website reader in nemarOrg/website#278 consumes it). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.
Activity
neuromechanist commentedon Sep 2, 2026
Filed under epic #1181 (zarr serving next phase); closest sibling is #1062, which covers exposing Zarr state in the public catalog.
Context: this shape change is already live on production as of the 0.9.10 cutover (#1182).
zarr_data_failuresnow serves{"count": N, "detail_ref": "zarr/index.json"}instead of the raw array. Production currently has 199 datasets carrying it, and the counts were preserved through the migration, so a consumer readingcountgets the same number it was previously deriving from the array's length.neuromechanist commentedon Sep 3, 2026
Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1201 (zarr_data_failures is now the {count, detail_ref} summary declared in shared/contract/dataset.ts; the website reader in nemarOrg/website#278 consumes it). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.