Skip to content

Collapse the 12 sweep bookkeeping stamps into one JSON column #1183

Closed

Description

@neuromechanist

Context

Follow-up to #1182, which reclaimed 11 columns on datasets to get under D1's 100-column cap. This is the largest remaining reclamation and was deliberately deferred out of that PR because it is a much wider review surface than a release unblock should carry.

Proposal

Collapse the 12 sweep bookkeeping stamps into a single JSON column, freeing 11.

enrichment_updated_at, metadata_updated_at, archive_checked_at, zarr_checked_at, records_checked_at, citations_updated_at, channel_montage_checked_at, hed_checked_at, data_checked_at, availability_report_at, recording_stats_at, signal_defaults_at.

Why this is not free

Unlike the attestation group in #1182, these columns are filtered on directly. Every sweep selects its candidates with some form of WHERE <name> IS NULL ORDER BY ... LIMIT n, so each of those predicates becomes a json_extract. The rewrite touches every sweep in backend/src/services/.

Query performance is not the concern at current scale (roughly 800 rows, so a full scan is trivial either way). The concern is that a mistake in a rewritten candidate predicate is silent: a sweep that selects the wrong candidates does not error, it just quietly stops making progress or reprocesses the same rows forever.

Requirement

Each rewritten sweep predicate needs a test that fails if the predicate stops selecting the right candidates, proved by mutating the predicate and confirming the test catches it. Per .rules/testing.md, test the sweep entry point rather than a helper.

Activity

  1. neuromechanist commented on Sep 2, 2026

    @neuromechanist
    ContributorAuthor

    Landed in PR #1187, squash-merged to dev as fe994f57. Closing by hand because GitHub only honours closing keywords on merges into the default branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions