perf(filtered-read): prune non-candidate fragment metadata

[BubbleCal]

Problem

After a scalar index returns its row-ID mask, FilteredReadExec::get_or_create_plan_impl still loads deletion vectors, row counts, and row-ID metadata for every fragment before applying that mask. An exact index miss therefore performs an O(fragment count) metadata pass even though no fragment can contribute a row.

This is a follow-up to #7792. That PR removed the second all-fragment metadata load during stream construction; the first planning-time pass remained. Related to #4189, which tracks broader filtered-read planning costs.

Change

Use the index result's upper bound to decide whether a fragment can contribute before loading its full metadata:

• skip every covered fragment for an exact empty allow-list;
• use the fragment portion of address-style row IDs for direct pruning;
• for non-empty stable row-ID masks, use each fragment's row-ID sequence to route candidates, then load deletion vectors and row counts only for candidate fragments;
• carry the routed stable row-ID sequence and upper offset ranges into final planning, so exact and refined/at-most results do not map the same IDs twice;
• cap retained stable-ID range payload at 16 MiB per plan and drain retained vectors fragment-by-fragment; over-budget fragments reuse the loaded row-ID sequence but recompute ranges during final planning;
• keep uncovered fragments unless only_indexed_fragments is enabled;
• conservatively keep block-list upper bounds and every fragment needed to calculate a pre-filter scan range.

Candidate fragments still load and apply their deletion vectors, so stale deleted index hits remain excluded. Pruning uses the upper bound of refined index results to avoid false negatives.

Benchmark

Lower is better for every metric below.

Scenario / metric	Baseline (e958adfdf)	This PR (150e000f0)	Benefit
Exact-empty plan latency	1,630.20 ms/query	5.14 ms/query	316.9x speedup
Indexed zero-hit query latency	2,067.66 ms/query	84.73 ms/query	24.4x speedup
Indexed one-hit query latency	2,081.91 ms/query	163.69 ms/query	12.7x speedup
Exact-empty plan S3 reads	2,600 reads/query	0 reads/query	2,600 reads/query eliminated
Indexed zero-hit S3 reads	2,603 reads/query	3 reads/query	867.7x fewer reads
Indexed one-hit S3 reads	2,606 reads/query	7 reads/query	372.3x fewer reads

The review follow-up also measures cache-hot dense stable-row-ID planning, where all 5,200 physical row IDs route to all 2,600 fragments:

Scenario / metric	Before follow-up (824551d44)	This PR (150e000f0)	Benefit
Dense stable-ID plan latency	9.17 ms/plan	7.86 ms/plan	1.17x speedup

The main-branch control was 7.13 ms/plan. The current implementation is 0.72 ms/plan (1.10x) above that control because it performs the stable-ID routing pass needed for pruning, but it no longer repeats the same mapping during final planning. All cache-hot measured plans performed 0 S3 reads.

Environment and methodology:

• AWS m7i.4xlarge in us-east-1a, reading S3 in the same region.
• One dataset with 2,600 stable-row-ID fragments, 2 rows per fragment, one real deletion vector per fragment, 14 payload columns, and BTree indices on org_id and repo_id.
• Separate main, pre-follow-up, and current PR binaries built with Rust 1.97.1, release-with-debug, --no-default-features --features aws; their SHA256 hashes were checked before measurement.
• Three serialized trials per case; base/PR order alternated by trial. Each trial used a new process and fresh Lance Session; the table reports medians.
• Dataset-open time and I/O were excluded. Query/planning I/O uses incremental IOTracker statistics after open.
• The dense case used three processes per revision. Each process ran one excluded S3 warm-up plan followed by 21 measured plans; the table reports the median of the three process medians. Index-result serialization and exec construction were outside the timed region.
• This measures the metadata path against real S3, not the Plan Executor persistent-disk cache.

Measured bytes followed the same pattern: exact-empty planning dropped from 1,814,800 B/query to 0 B/query; the zero-hit query dropped from 1,882,874 B/query to 68,074 B/query; the one-hit query dropped from 1,965,927 B/query to 151,825 B/query.

Correctness and limitations

The regression coverage includes stable and address-style row IDs, exact-empty and sparse non-empty masks, refined upper bounds, partially indexed datasets, and only_indexed_fragments behavior. A deterministic dense stable-ID test counts mask_to_offset_ranges spans: four candidate fragments must produce exactly four mappings; the pre-follow-up implementation produced eight. A separate boundary test verifies that retained range payload cannot exceed the per-plan budget.

For a non-empty stable row-ID mask, this change still visits each covered fragment's row-ID sequence to discover candidate fragments. The benchmark fixture stores those sequences inline, so it removes the S3 deletion-vector/row-count reads but not the O(fragment count) routing walk. Datasets with external row-ID metadata can still perform O(fragment count) row-ID metadata reads. Eliminating that remaining cost requires carrying physical candidate-fragment information from scalar-index execution or persisted row-ID routing metadata.

Validation

• cargo fmt --all
• cargo test -p lance io::exec::filtered_read::tests -- --test-threads=1 (75 passed)
• cargo clippy --all --tests --benches -- -D warnings
• RUSTFLAGS='-D warnings' cargo +nightly-2026-07-13 check -p lance --tests
• AWS S3 benchmark described above

[BubbleCal]

Addressed in 4e4207f. Stable-ID routing now carries the loaded RowIdSequence and computed upper offset ranges into LoadedFragment; Exact and Refined/AtMost planning reuse them, while AtLeast still maps its lower bound once. I added a deterministic regression that counts mask_to_offset_ranges spans (4 candidate fragments produce 4 calls; the previous head produced 8). On the 2,600-fragment cache-hot dense workload, median plan latency improved from 8.26 ms at 824551d to 7.15 ms at the current head (1.15x); the main control was 6.40 ms and the measured phase performed 0 S3 reads. The PR body now includes the methodology and exact-current-head S3 results.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The re-requested revision preserves the bounded stable-ID routing budget with a standard compare-exchange loop. It removes the nightly atomic API warning without changing candidate pruning, the 16 MiB retention bound, or fallback recomputation.