Repository navigation
Per-row cost of a batch preview: ~52 µs/row inside the library on a realistic ingest shape (measured) #504
Description
Per-row cost of a batch preview on a realistic ingest shape: most of it is inside the library. Measured end to end by a downstream consumer (measured by the reporter).
Setup:
- Graviton4, 8 cores; go v1.0.2 with the production library build
20261004.052404. - One 100-row NDJSON batch per call, into an 8-column table.
- A row filter,
JSONCompactEachRowexport,WithDocFlags(0).
| where | cost |
|---|---|
| inside the library, per call | about 0.9 ms |
| inside the library, per row | about 52 µs |
| per row, end to end at concurrency 32 | about 75 µs |
| Go-side decode of the result, per row | about 3.5 µs |
| cgo overhead and GC | negligible |
| a hand-written validator for the same shape, per row | about 22 µs (the reporter's own baseline) |
Reading: the binding's own share (decode, cgo, GC) is small. The per-row time is spent inside the library's preview path.
Next: the artifact producer is profiling the preview path, to find where the per-row time goes. That covers the parse, the defaults, the constraint and filter evaluation, the export, and the per-call setup behind the ~0.9 ms. Findings and any fix will be posted here. A fix would be a library change, so all four bindings benefit together.
Related: #456 (Go Rows scaling across goroutines, closed) and #480 (one shared handle's scaling).
🤖 Generated with Claude Code
Activity
EricAndrechek commentedon Oct 6, 2026
Per-row costs on production library build
20261006.170903(measuredby the artifact producer, on public table shapes, Graviton4, 8 cores):20261004.052404; one schema per thread costs 24.2 µs/row. This repository's own scaling re-measure is on One shared handle should scale across threads like N separate handles #480.DEFAULT f(other_column)adds about 98 µs/row, because its expression is analysed once per row. A library fix is in progress, and the cost stays until a build ships it. ADEFAULTthat reads no other column does not pay it. This is now documented indocs/reference/bindings-v1.md§3.input_format_allow_errors_ratiohad no measurable effect on valid bodies.We'll post the build that removes each cost here.