Repository navigation
perf(zarr-indexing): skip the sort in _last_occurrences for sorted indices - #4603
Merged
d-v-b merged 4 commits intoNov 6, 2026
Conversation
…dices When the flat in-hull positions are non-decreasing (the usual C-order case), repeats are adjacent and the last occurrence of each can be found in O(n) without np.unique. When there are no repeats at all, return a full slice so callers index without a gather copy. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b
marked this pull request as ready for review
November 6, 2026 14:29
…s#4603 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
performance win for pre-sorted coordinate indices.
🤖 AI text below 🤖
Writes whose indices land on a chunk in sorted order no longer sort them to drop repeated indices, so deduplication is linear in the number of points; writes without repeats also skip a gather copy. Unsorted indices behave as before.
_last_occurrencesnow checks in O(n) whether the flat positions are non-decreasing, the usual case since a chunk's points arrive in C order. If they are, repeats are adjacent and the last of each run is kept vianp.flatnonzero(np.append(flat[1:] != flat[:-1], True)). If there are no repeats, or fewer than two positions, it returnsslice(None), so_scatter_numpyand_scatter_plannedindex without copying. Unsorted input still goes throughnp.unique. Callers only use the result to pick matching rows of coordinates and values, so the order of the kept positions doesn't matter. The sorted-input check is borrowed from thelast_occurrenceshelper that Bhaskar Gurram (bhaskargurram-ai) wrote in #4601; credit for the idea goes to them.A new parametrized test,
test_last_occurrences_matches_sequential_assignment, checks empty, single, sorted, sorted-with-repeats, unsorted-with-repeats and reversed input against sequential NumPy assignment.Timing for
_last_occurrenceson 5Mintppositions (NumPy 2.5.1, Python 3.13, macOS arm64, best of 5):The zarr-indexing suite passes on Python 3.13 (1745 passed), and
just typecheckandjust lintare clean. On Python 3.15, four tests fail onmaintoo, because they match the__index__error message that 3.15 changed.🤖 Generated with Claude Code
Author attestation
TODO
docs/user-guide/*.mdchanges/