perf(ingest): size worker batches by bytes so each insert writes one compact part #779

Area: ingest · performance — gap

Expected: each worker flush is as large as it can be while ClickHouse still writes it as a compact part: a batch target expressed in bytes (for example around 8 MiB uncompressed), below the table's min_bytes_for_wide_part, instead of a fixed row count. Fewer, larger parts per table, each a small, fixed set of files.

Actual: a table's batch flushes at 500 rows or 5 s (internal/ingest/worker.go:132-133) whatever the row width, and each flush is one INSERT, so one new part per partition it touches (unless the server buffers inserts with async_insert, see #557).

Measured with clickhouse local on 26.6.3.62 (the version pinned on main), a 20-column MergeTree table, merges stopped:

• min_bytes_for_wide_part defaults to 10,485,760 (10 MiB) and min_rows_for_wide_part to 0, on both 26.6.3.62 and 26.8.15.10. The comparison is on uncompressed bytes: the 14.2 MiB part went Wide although it compressed to 3.8 MiB.
• A compact part is a fixed ~10 files whatever its row count; a wide part is about two files per column stream. On an object-storage disk each file is written as its own object (inferred), so a compact part costs a roughly fixed number of writes.
• 500 rows of this schema is about 1% of the threshold. Small parts also mean more merge work and count toward parts_to_delay_insert (1,000) and parts_to_throw_insert (3,000) per partition (defaults, measured).

Constraints the change has to account for (measured in code unless marked):

• maxAckPending = 10,000 caps the rows a tenant can have delivered and unacked (worker.go:145), and every queued message is one row (deployment.md:715). At ~210 B/row that is ~2 MiB, below an 8 MiB target. Under mq.backend: nats the cap is per shard durable, shared by the tenants on that shard (internal/mq/nats_topology.go:52-55,70).
• The worker holds every delivered row until its batch is acked (worker.go:140-145), so a larger target raises worker memory per tenant served (ingest: per-table worker goroutines (tableLoops) are never reaped — unbounded under untrusted publishers #263, api: bound total in-flight ingest body bytes now that the handler buffers the whole body #544).
• ackWait (60 s) has to stay above the flush interval plus the insert time (worker.go:136-138), so a time trigger remains.
• ClickHouse compares the block's in-memory uncompressed size, which differs from the queued payload's bytes (inferred), so a byte trigger is an estimate and a target below the threshold leaves headroom. The threshold is per table (it can be set in the table's SETTINGS) and per server (system.merge_tree_settings).
• A larger batch changes the cost of ingest: split oversized batches by halving, and don't split TOO_MANY_PARTS caused by merge backpressure #657's split-by-halving on failure.
• feat(typelayer)!: validate, coerce and filter through chtypes #712 changes the insert path in worker.go, and fix(chconn): release a tuple's HTTP clients with its pool #770 also edits that file.

Impact: a steady ingest stream writes many ~100 KiB parts per table instead of a few multi-MiB ones: more objects written on object storage, more merges, and less headroom before the too-many-parts limits under bursty or many-table load.

Related: #778 (making the target and the time trigger configurable, per tenant where it fits), #557, #657, #263, #544, #94 (batch-size and flush-duration metrics).

Found during a backlog review; measured with clickhouse local 26.6.3.62 and 26.8.15.10 and by code-read against origin/main at afd76695 on 2026-10-08.