perf(scan): reduce system column pipeline overhead - #8717
Merged
jiaoew1991 merged 2 commits intoAug 25, 2026
Merged
Conversation
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
jiaoew1991
force-pushed
the
perf/system-column-scan-pipeline
branch
from
August 24, 2026 12:10
ce92c1d to
23a1f6c
Compare
jiaoew1991
force-pushed
the
jiaoew/stack-system-column-base
branch
from
August 24, 2026 12:10
e004e20 to
90c4cdb
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-approved
jiaoew1991
force-pushed
the
perf/system-column-scan-pipeline
branch
from
August 24, 2026 13:53
23a1f6c to
9cdd0dc
Compare
github-actions
Bot
added
A-python
jiaoew1991
changed the base branch from
jiaoew/stack-system-column-base
to
jiaoew/stack-system-column-base-v2
August 24, 2026 13:54
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
jiaoew1991
force-pushed
the
perf/system-column-scan-pipeline
branch
from
August 24, 2026 14:33
9cdd0dc to
4432222
Compare
jiaoew1991
changed the base branch from
jiaoew/stack-system-column-base-v2
to
jiaoew/stack-system-column-base-v3
August 24, 2026 14:33
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
jiaoew1991
force-pushed
the
perf/system-column-scan-pipeline
branch
from
August 25, 2026 01:09
4432222 to
f47b8b9
Compare
github-actions
Bot
added
A-index
jiaoew1991
changed the base branch from
jiaoew/stack-system-column-base-v3
to
jiaoew/stack-system-column-base-v4
August 25, 2026 01:09
lance-gatekeeper
Bot
removed
the
K-approved
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The base rebase leaves the reviewed implementation patch-equivalent. Bounded read-ahead and consolidated system-column assembly continue to preserve scan and error semantics while delivering the verified performance improvement.
lance-gatekeeper
Bot
added
K-approved
jiaoew1991
merged commit Aug 25, 2026
73df0d9
into
lance-format:jiaoew/stack-system-column-base-v4
11 checks passed
Contributor
Author
Xuanwo
added a commit
that referenced
this pull request
Sep 4, 2026
## Summary Reduce fixed per-batch work in the system-column scan pipeline after the row-ID and version cursors have removed decoder prefix scans. This supersedes #8717, which was merged into its temporary integration base while the dependency stack was being refreshed. No version of #8717 was merged into `main`. This change: - reads stable row IDs ahead in batch-aligned chunks and serves uniform adjacent batches as zero-copy Arrow slices; - validates the complete prefetched selection before exposing or caching it; - drains the current row-ID chunk and falls back to exact per-task decoding when non-final task sizes vary, avoiding repeated boundary copies or cursor rewinds; - assembles all requested system columns into one `RecordBatch` for uniform task streams, while variable task streams retain the prior incremental assembly; - caches structurally equivalent schemas for uniform task streams, not only pointer-identical `Arc<Schema>` values, and disables that cache after task sizes vary; - reuses a full-size zero-column batch template; - skips no-op projections for system-only reads only when schemas are strictly equal; - bounds read-ahead by the actual range selection, including empty and tail reads. For batches up to 64K rows, each cached row-ID chunk is at most 64K `u64` values (512 KiB). A larger batch is decoded as one batch without additional read-ahead. Zero-copy slices can keep a chunk alive until adjacent batches are released; an irregular stream can make one boundary copy while draining the existing chunk. ## Performance Benchmarks compare `main` at `a8bec27d5` (including #8716) with this branch, use `release-with-debug`, pin both revisions to the same CPU, and run each case in a fresh process. The 100K-row uniform-task matrix improved all 16 combinations of batch size (64/1024), bitmap density (holes every 2/17 values), payload (absent/present), and system columns (`_rowid`/all): - batch size 64: `_rowid` improved 30.3%-33.7%; all system columns improved 51.4%-56.8%; - batch size 1024: `_rowid` improved 13.1%-26.5%; all system columns improved 29.5%-36.7%. A standard Criterion run for the representative batch-size-64, holes-17, zero-payload, all-system-columns case measured 3.7000 ms on `main` and 1.7455 ms here (-52.8%). CPU profiles attribute the change: `RecordBatchExt::try_with_column` (7.47% self time on `main`) and `SchemaExt::try_with_column` (4.59%) both fell below the 0.1% reporting threshold. The review reproducer uses 10M rows, holes-17, one payload column, `_rowid`, and alternating 32,768/32,769-row tasks. With isolated base/head target directories, standard Criterion measured 10.387-10.468 ms on `main` and 10.449-10.499 ms here (+0.51% by point estimate, within run noise). A second pair under `perf record` measured 10.699-10.746 ms and 10.652-10.724 ms, respectively. Profiles show the same dominant workload (`SegmentCursorState::extend_dense_range`, 42.17% / 42.22% self time); `RecordBatchExt::try_with_column` was only 0.16% / 0.10%, and schema reconstruction and boundary copying were not hotspots. ## Validation - `cargo test -p lance-table --lib` (384 passed) - `cargo test -p lance --lib dataset::fragment` (108 passed) - `cargo check -p lance-table --tests --benches` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` - Python integration: `test_to_batches_with_partial_last_batch`, `test_to_batches`, `test_scan_no_columns`, and `test_roundtrip_reader` (4 passed) - read-only production-scale matrix: 32/32 cases passed against an independent metadata oracle, with the dataset version asserted unchanged before and after Targeted regressions cover mixed payload + all-system projection schema/order, uniform structurally equal schemas with different `Arc`s, variable-task fallback, stable-row-ID unsorted indices and metadata underfill, read-ahead tail/chunk boundaries, empty tasks, and system-only `read_all` / `read_ranges`. Dependencies #8713, #8715, and #8716 are merged into `main`. This PR's diff is limited to the system-column benchmark and the two implementation files. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded
This PR was merged only into its temporary integration base while the dependency stack was being refreshed; it was not merged into
main.Please review #8747, which carries the focused system-column pipeline diff on the refreshed dependency base.