[opt](lance) Support distributed execution for vector_search - #66841
Merged
yiguolei merged 8 commits intoAug 20, 2026
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Requesting changes for five correctness and rolling-upgrade issues: the deployed Lance Thrift fields are renumbered; a failed one-shot row-id task is retained after its caller wakes; V3 row-location bytes are silently misdecoded by an old materializer; the removed negative-fragment guard can turn an old-FE range into a successful empty scan; and new fragment-scoped searches have no old-BE capability fallback.
Critical checkpoints:
- Distributed search: the same-version fragment-local
top_k + offsetbound, global OFFSET/LIMIT, outer-WHERE placement, flat/indexed modes, filters, ties, deletes, and empty/tail fragments were traced and are consistent. Mixed-version wire and physical-execution compatibility is not. - Row identity and lazy materialization: same-version uint64 propagation, snapshot/storage identity, grouping, deduplication, input-order restoration, hidden-row-ID injection, aliases/nested columns, and direct take were checked. The V3 consumer compatibility issue remains blocking.
- Reader/FFI lifecycle: dataset/scanner/Arrow ownership, virtual-only projection, returned-row validation, and error cleanup were checked. The deserialized fragment boundary regression remains blocking.
- Concurrency: the new worker defer avoids the waiter's hang, but an error result violates the scheduler callback's completion contract and retains the task/closure.
- Coverage: the changed tests cover same-version distributed search and row fetch paths, but not the mixed-version Thrift/layout/capability cases, the old-FE high-bit fragment path, or failed-task cleanup.
No additional user focus was supplied. The review converged after three full FE, BE/third-party, and cross-layer rounds. Static review only; builds and tests were not run because the review environment explicitly forbids them.
11 tasks
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 20th, 2026 3:35 AM. The selected account is excluded until 2026-08-20T03:35:00Z. Please trigger /review again; another configured account may be available. |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-20T03:35:00Z. Please trigger /review again after that time. |
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
yiguolei
approved these changes
Aug 20, 2026
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
reviewed
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
FANNG1
added a commit
to FANNG1/doris
that referenced
this pull request
Aug 20, 2026
Conflicts came from apache#66841 (distributed vector search), which touched the same Lance files. Resolved as follows. **Thrift.** apache#66841 renumbered `lance_substrait_filter` to 37 and `external_search_request` to 38, which is the id this branch had taken for `lance_storage_options`. Upstream's numbering wins and the new field moves to 39. This is the id divergence raised on this PR earlier, now realised inside branch-4.1 rather than only against master. **One option map, not two.** Upstream threads `javaStorageOptions` and `backendStorageOptions` separately, and still calls `LanceStorageOptions .forBackend` - the re-encoding this PR removes, and the cause of the bug it fixes. Its index-segment work is kept in full; both parameters collapse onto the single Lance-native map, so `LanceMetadataLoader.loadLatest`, `loadLatestWithIndexSegments`, `loadVersion` and `loadInternal` each take one option map, and `LanceTableMetadata.withIndexSegments` / `withoutIndexSegments` carry it. `getBackendStorageOptions` becomes `getLanceStorageOptions`. **BE.** Upstream moved dataset opening into `_ensure_dataset_open` and added the row-id take path. That structure is kept, with this branch's `Status`-returning `_storage_options` and `_dataset_key` applied on top, so a NUL still fails rather than being dropped. `LanceSnapshotTest` keeps upstream's `LanceFragmentInfo` extraction and the `withoutIndexSegments` factory; its fixture key becomes `aws_endpoint`, which is what this branch actually emits. Verified: FE builds, 12 unit tests pass. The BE translation unit compiles clean once apache#66841's `lance-c-0.1.6-doris.patch` is applied to the local thirdparty - this checkout predates that patch, so the BE binary here cannot be rebuilt without redoing thirdparty. The regression suite is therefore not re-run in this commit: the running BE is older than both the new thrift numbering and upstream's BE changes, so it would not be testing this code.
FANNG1
added a commit
to FANNG1/doris
that referenced
this pull request
Aug 21, 2026
… sites
Review follow-up on the merge.
- lance_reader_test had no case for lance_storage_options at all, so the one
backend behaviour this series adds - failing on an option a C string cannot
carry - was untested. A key with an embedded NUL now has to be rejected
rather than dropped.
- ExternalFileTableValuedFunction set the field unconditionally, so an empty
map still flipped __isset, while the other TVF path guards on non-empty.
Harmless either way, but the two paths should not disagree.
- LanceExternalCatalog's lanceStorageOptions is written once and read once,
now that nothing else needs the catalog-level map; it is a local.
The new test is not compile-verified here: this checkout has no backend test
translation unit in its compile database, and the backend cannot be built at all
until the thirdparty lance-c carries apache#66841's patch. CI covers it.
Claude-Session: https://claude.ai/code/session_01PfVSEUJsfWwKy7WEuBaFCG
1 task
12 tasks
yiguolei
pushed a commit
that referenced
this pull request
Aug 27, 2026
…ement types (#67039) ### What problem does this PR solve? Issue Number: Part of #66495 Problem Summary: Companion documentation PR: [apache/doris-website#4082](apache/doris-website#4082). #66512 gave `vector_search()` end-to-end coverage for all six Lance ANN algorithms, but only with **Float32 + L2**. Reviewer feedback there was to merge it and follow up with the other element types, which is what this PR does. The gap is not only a dependency-compatibility one. #66841 made the FE metric-aware: Doris plans an indexed split only when the requested metric equals the metric the index was built with (`LanceScanNode.metricMatches`), and otherwise plans flat splits rather than letting Lance fall back to brute force silently. Until now the **cosine and dot branches of that comparison had never run against a real index**, because no non-L2 index existed anywhere in the fixture. This PR covers them, in both directions. ### What is changed? There is no FE or BE production-code change. #### The matrix is now covered in full, in two tiers The support matrix has 96 cells (4 element types x 4 metrics x 6 algorithms). All 96 were built against the embedded Lance generation, one per subprocess: **56 build and answer a search, 40 do not**, and the 40 fall into three groups — a float type with `hamming` (IVF training rejects it), `uint8` with a non-hamming metric (it is read as a binary vector), and `uint8` under a quantizing builder. `buildable_combos()` in the generator yields exactly those 56; the measurement and the three failure groups are recorded next to it. All 56 are covered: - **12 tables in depth.** All six algorithms on Float32 + L2, plus cosine and dot, plus Float64, Float16 and UInt8. Committed goldens, a closed-form distance ladder where the data shape has one, and a discriminator per table. Data shape is per metric, because one shape cannot serve them all: the collinear ladder is degenerate under cosine (directions converge and the top distances collapse to zero) and under dot (the answer stops depending on the query), so those tables use a directional shape, and UInt8 uses a thermometer code that makes hamming an exact ladder. - **44 cells in one table**, `doris.vs_index_matrix`, at 64 rows with one vector column per cell. One column per cell rather than several indexes on one column, because only the first index built on a column is reachable — Lance answers the others with a brute-force scan, and Doris arrives at the same place by another route, since `selectIndexSegments` keeps only the segments of the first index it finds for a field id. `vs_index_matrix` asserts three things per cell, and no goldens, so a fixture rebuild leaves it untouched: 1. Doris plans an indexed split, with `lanceSearchUnindexedFragments=0` — the only check that would catch an index reaching one of the two fragments. 2. `nprobes=1` answers differently from `nprobes=4`. A flat scan has no partitions and cannot, so this is what separates a real indexed search from a silent fallback. Four query rows are tried and the first that discriminates is enough. 3. Reranked with `refine_factor`, the indexed search returns exactly the rows an exhaustive scan returns. Recall is not measured, and (3) compares two paths inside the same backend, so it does not pin absolute distances — that is what the 12 depth tables' closed-form ladders do. #### Fixture self-check The self-check is the whole contract for a fixture whose bytes are not reproducible, so it now also pins a digest of every profile's vectors and reads rows back from each table; asserts the metric each index was **actually built with**, read from `stats["indices"][*]["metric_type"]` rather than searched for, because a cosine IVF_PQ reports `l2` on its sub-index; keeps the indexed-vs-flat comparison bit-exact for the integer ladders; and enforces the `pylance` and `lance-namespace` pins before building or verifying. Discriminator margins are reported and warned on when thin. `--repin` downgrades a stale discriminator row to a warning so a rebuild can complete and the row can then be re-measured against the fixture that was promoted. #### Fixture publishing `iceberg.yaml.tpl` now uses `mc mirror --overwrite --remove` instead of `mc cp -r`. The `__manifest` version is a commit count that a rebuild can lower, and Lance names version files `u64::MAX - version` so a listing resolves newest first; a stale higher-versioned manifest left in the bucket therefore wins over the one just published. A live test environment's MinIO had accumulated several. Also removes two `entrypoint.sh` files that a `git add -A` had picked up — they are rendered from the `.tpl` beside them by `run-thirdparties-docker.sh` — and adds the `.gitignore` rule that was missing for them. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [x] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> apache/doris-website#4082 ### Check List (For Reviewer who merge this PR) - [x] Confirm the release note - [x] Confirm test cases - [x] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
issue: #66340
support two phase in distributed execution for lance vector_search.
will return _row_id, _distance and some postfilter column from lance scan result, and execute the where postfilter.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)