perf(scan): reduce system column pipeline overhead

[jiaoew1991]

Summary

Reduce fixed per-batch work in the system-column scan pipeline after the row-ID and version cursors have removed decoder prefix scans.

This supersedes #8717, which was merged into its temporary integration base while the dependency stack was being refreshed. No version of #8717 was merged into main.

This change:

• reads stable row IDs ahead in batch-aligned chunks and serves uniform adjacent batches as zero-copy Arrow slices;
• validates the complete prefetched selection before exposing or caching it;
• drains the current row-ID chunk and falls back to exact per-task decoding when non-final task sizes vary, avoiding repeated boundary copies or cursor rewinds;
• assembles all requested system columns into one RecordBatch for uniform task streams, while variable task streams retain the prior incremental assembly;
• caches structurally equivalent schemas for uniform task streams, not only pointer-identical Arc<Schema> values, and disables that cache after task sizes vary;
• reuses a full-size zero-column batch template;
• skips no-op projections for system-only reads only when schemas are strictly equal;
• bounds read-ahead by the actual range selection, including empty and tail reads.

For batches up to 64K rows, each cached row-ID chunk is at most 64K u64 values (512 KiB). A larger batch is decoded as one batch without additional read-ahead. Zero-copy slices can keep a chunk alive until adjacent batches are released; an irregular stream can make one boundary copy while draining the existing chunk.

Performance

Benchmarks compare main at a8bec27d5 (including #8716) with this branch, use release-with-debug, pin both revisions to the same CPU, and run each case in a fresh process.

The 100K-row uniform-task matrix improved all 16 combinations of batch size (64/1024), bitmap density (holes every 2/17 values), payload (absent/present), and system columns (_rowid/all):

• batch size 64: _rowid improved 30.3%-33.7%; all system columns improved 51.4%-56.8%;
• batch size 1024: _rowid improved 13.1%-26.5%; all system columns improved 29.5%-36.7%.

A standard Criterion run for the representative batch-size-64, holes-17, zero-payload, all-system-columns case measured 3.7000 ms on main and 1.7455 ms here (-52.8%). CPU profiles attribute the change: RecordBatchExt::try_with_column (7.47% self time on main) and SchemaExt::try_with_column (4.59%) both fell below the 0.1% reporting threshold.

The review reproducer uses 10M rows, holes-17, one payload column, _rowid, and alternating 32,768/32,769-row tasks. With isolated base/head target directories, standard Criterion measured 10.387-10.468 ms on main and 10.449-10.499 ms here (+0.51% by point estimate, within run noise). A second pair under perf record measured 10.699-10.746 ms and 10.652-10.724 ms, respectively. Profiles show the same dominant workload (SegmentCursorState::extend_dense_range, 42.17% / 42.22% self time); RecordBatchExt::try_with_column was only 0.16% / 0.10%, and schema reconstruction and boundary copying were not hotspots.

Validation

• cargo test -p lance-table --lib (384 passed)
• cargo test -p lance --lib dataset::fragment (108 passed)
• cargo check -p lance-table --tests --benches
• cargo clippy --all --tests --benches -- -D warnings
• cargo fmt --all -- --check
• git diff --check
• Python integration: test_to_batches_with_partial_last_batch, test_to_batches, test_scan_no_columns, and test_roundtrip_reader (4 passed)
• read-only production-scale matrix: 32/32 cases passed against an independent metadata oracle, with the dataset version asserted unchanged before and after

Targeted regressions cover mixed payload + all-system projection schema/order, uniform structurally equal schemas with different Arcs, variable-task fallback, stable-row-ID unsorted indices and metadata underfill, read-ahead tail/chunk boundaries, empty tasks, and system-only read_all / read_ranges.

Dependencies #8713, #8715, and #8716 are merged into main. This PR's diff is limited to the system-column benchmark and the two implementation files.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebase preserves all five substantive patches exactly. Current-head stream and fragment regressions remain clean, and the intervening row-ID optimization is isolated to index/cache lookup rather than this stream-selection path, so the prior correctness and base-equivalent variable-task performance evidence remains valid.