test(index): cover the IVF partition transformer and partition filter - #8765
Merged
Xuanwo merged 2 commits intoSep 2, 2026
Merged
Conversation
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
`ivf/transform.rs` had no tests, unlike its siblings `sq/transform.rs` and `flat/transform.rs`. Both types in it fail silently: a vector assigned to the wrong partition is never searched in the right one, and a `PartitionFilter` that keeps the wrong rows drops vectors from a sharded build without an error. The subtle part is the guard at the top of `PartitionTransformer::transform`. Partitions already present means skip — except when `with_distance` is set and the distance column is missing, where it has to drop both and recompute, or it returns a batch without the column it promised. Adds 11 tests: nearest-centroid assignment, the loss metadata `v3/shuffler.rs` reads back, opt-in distances, both skip and recompute branches, the two column errors, and range filtering including the keep-nothing case. Verified non-vacuous: zeroing the assignment, zeroing the loss, ignoring `with_distance`, and making the filter keep everything fail 6 of the 11.
jackylee-ch
force-pushed
the
test/ivf-partition-transform-coverage
branch
from
August 26, 2026 10:09
79d3c7f to
d74bd76
Compare
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
lance-gatekeeper
Bot
removed
the
K-approved
lance-gatekeeper
Bot
added
the
K-approved
Contributor
Author
|
Thanks @Xuanwo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ivf/transform.rshad no tests, unlike its siblingssq/transform.rsandflat/transform.rs. Both types in it fail silently: a vector assigned to thewrong partition is never searched in the right one, and a
PartitionFilterthatkeeps the wrong rows drops vectors from a sharded build without an error.
The subtle part is the guard at the top of
PartitionTransformer::transform.Partitions already present means skip — except when
with_distanceis set andthe distance column is missing, where it has to drop both and recompute, or it
returns a batch without the column it promised.
Adds 11 tests: nearest-centroid assignment, the loss metadata
v3/shuffler.rsreads back, opt-in distances, both skip and recompute branches, the two column
errors, and range filtering including the keep-nothing case.
Verified non-vacuous: zeroing the assignment, zeroing the loss, ignoring
with_distance, and making the filter keep everything fail 6 of the 11.