[test](lance) Add indexed coverage for cosine, dot and non-Float32 element types by FANNG1 · Pull Request #67039 · apache/doris

FANNG1 · 2026-08-21T23:25:26Z

What problem does this PR solve?

Issue Number: Part of #66495

Problem Summary:

Companion documentation PR: apache/doris-website#4082.

#66512 gave vector_search() end-to-end coverage for all six Lance ANN algorithms, but only
with Float32 + L2. Reviewer feedback there was to merge it and follow up with the other
element types, which is what this PR does.

The gap is not only a dependency-compatibility one. #66841 made the FE metric-aware: Doris
plans an indexed split only when the requested metric equals the metric the index was built
with (LanceScanNode.metricMatches), and otherwise plans flat splits rather than letting
Lance fall back to brute force silently. Until now the cosine and dot branches of that
comparison had never run against a real index, because no non-L2 index existed anywhere in
the fixture. This PR covers them, in both directions.

What is changed?

There is no FE or BE production-code change.

The matrix is now covered in full, in two tiers

The support matrix has 96 cells (4 element types x 4 metrics x 6 algorithms). All 96 were
built against the embedded Lance generation, one per subprocess: 56 build and answer a
search, 40 do not, and the 40 fall into three groups — a float type with hamming (IVF
training rejects it), uint8 with a non-hamming metric (it is read as a binary vector), and
uint8 under a quantizing builder. buildable_combos() in the generator yields exactly
those 56; the measurement and the three failure groups are recorded next to it.

All 56 are covered:

12 tables in depth. All six algorithms on Float32 + L2, plus cosine and dot, plus
Float64, Float16 and UInt8. Committed goldens, a closed-form distance ladder where the
data shape has one, and a discriminator per table. Data shape is per metric, because one
shape cannot serve them all: the collinear ladder is degenerate under cosine (directions
converge and the top distances collapse to zero) and under dot (the answer stops depending
on the query), so those tables use a directional shape, and UInt8 uses a thermometer code
that makes hamming an exact ladder.
44 cells in one table, doris.vs_index_matrix, at 64 rows with one vector column per
cell. One column per cell rather than several indexes on one column, because only the
first index built on a column is reachable — Lance answers the others with a brute-force
scan, and Doris arrives at the same place by another route, since selectIndexSegments
keeps only the segments of the first index it finds for a field id.

vs_index_matrix asserts three things per cell, and no goldens, so a fixture rebuild leaves
it untouched:

Doris plans an indexed split, with lanceSearchUnindexedFragments=0 — the only check that
would catch an index reaching one of the two fragments.
nprobes=1 answers differently from nprobes=4. A flat scan has no partitions and cannot,
so this is what separates a real indexed search from a silent fallback. Four query rows are
tried and the first that discriminates is enough.
Reranked with refine_factor, the indexed search returns exactly the rows an exhaustive
scan returns.

Recall is not measured, and (3) compares two paths inside the same backend, so it does not pin
absolute distances — that is what the 12 depth tables' closed-form ladders do.

Fixture self-check

The self-check is the whole contract for a fixture whose bytes are not reproducible, so it now
also pins a digest of every profile's vectors and reads rows back from each table; asserts the
metric each index was actually built with, read from stats["indices"][*]["metric_type"]
rather than searched for, because a cosine IVF_PQ reports l2 on its sub-index; keeps the
indexed-vs-flat comparison bit-exact for the integer ladders; and enforces the pylance and
lance-namespace pins before building or verifying. Discriminator margins are reported and
warned on when thin. --repin downgrades a stale discriminator row to a warning so a rebuild
can complete and the row can then be re-measured against the fixture that was promoted.

Fixture publishing

iceberg.yaml.tpl now uses mc mirror --overwrite --remove instead of mc cp -r. The
__manifest version is a commit count that a rebuild can lower, and Lance names version files
u64::MAX - version so a listing resolves newest first; a stale higher-versioned manifest left
in the bucket therefore wins over the one just published. A live test environment's MinIO had
accumulated several.

Also removes two entrypoint.sh files that a git add -A had picked up — they are rendered
from the .tpl beside them by run-thirdparties-docker.sh — and adds the .gitignore rule
that was missing for them.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
  [doc](lance) Document the vector element type and metric compatibility matrix doris-website#4082

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-08-21T23:25:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

FANNG1 · 2026-08-25T09:25:03Z

@zhangstar333 @Gabriel39 PTAL

zhangstar333 · 2026-08-25T15:03:39Z

run buildall

zhangstar333 · 2026-08-26T01:42:36Z

@FANNG1 failed one test in test_lance_vector_search_indexed_element_types

FANNG1 · 2026-08-26T03:32:38Z

@zhangstar333 Thanks for running it. I can't read the failure myself — the TeamCity instance asks for a login — so could you paste the assertion block from the log? The part I need is:

Check tag '...' failed, line N, ... mismatch.
Expect cell is: ...
But real is   : ...

The tag name alone would probably be enough to pin it down.

For context on where I'd expect trouble: that suite covers three tables, and only the Float16 one records values that are not exactly representable.

f64_l2 / f64_l2_flat record the closed-form L2 ladder 0, 16, 64, 144, 256
u8_hamming / u8_hamming_flat record the hamming ladder 0, 1, 2, 3, 4

Both are exact integers on any platform. The Float16 blocks are different — they record float32 cosine distances printed to 7-8 significant digits (1.1920929E-7, 0.2166698, 0.29892504, ...), computed from Float16 inputs.

I regenerated the goldens on macOS/arm64 after rebuilding the fixture. If the failing tag is f16_cosine or f16_cosine_flat, the likely cause is simply that those last digits come out differently on the CI's x86_64 build. In that case pinning them is the bug, not the values: that table uses directional data with no closed-form expected distance, so the distance column records whatever the platform computed rather than anything checkable — the row order is the part that carries meaning. The fix would be to keep the exact ladders for Float64 and UInt8 and drop _distance from the Float16 blocks, keeping the "self-distance is not exactly zero" observation as a tolerance-based assertion instead of a pinned literal.

If it is a different tag, that guess is wrong and I will dig into it from the log.

zhangstar333 · 2026-08-26T04:35:57Z

@zhangstar333 Thanks for running it. I can't read the failure myself — the TeamCity instance asks for a login — so could you paste the assertion block from the log? The part I need is:
Check tag '...' failed, line N, ... mismatch.
Expect cell is: ...
But real is   : ...
The tag name alone would probably be enough to pin it down.

For context on where I'd expect trouble: that suite covers three tables, and only the Float16 one records values that are not exactly representable.

f64_l2 / f64_l2_flat record the closed-form L2 ladder 0, 16, 64, 144, 256

u8_hamming / u8_hamming_flat record the hamming ladder 0, 1, 2, 3, 4

Both are exact integers on any platform. The Float16 blocks are different — they record float32 cosine distances printed to 7-8 significant digits (1.1920929E-7, 0.2166698, 0.29892504, ...), computed from Float16 inputs.

I regenerated the goldens on macOS/arm64 after rebuilding the fixture. If the failing tag is f16_cosine or f16_cosine_flat, the likely cause is simply that those last digits come out differently on the CI's x86_64 build. In that case pinning them is the bug, not the values: that table uses directional data with no closed-form expected distance, so the distance column records whatever the platform computed rather than anything checkable — the row order is the part that carries meaning. The fix would be to keep the exact ladders for Float64 and UInt8 and drop _distance from the Float16 blocks, keeping the "self-distance is not exactly zero" observation as a tolerance-based assertion instead of a pinned literal.

If it is a different tag, that guess is wrong and I will dig into it from the log.

Exception:
java.lang.IllegalStateException: Check tag 'f16_cosine' failed:
Check tag 'f16_cosine' failed, line 1, FLOAT result mismatch.
Expect cell is: 1.1920929E-7
But real is : -1.1920929E-7
relative error is: 2.0, bigger than 1.0E-6
line 1 mismatch
ExpectRow: [1, item-0001, 1.1920929E-7]
RealRow : [1, item-0001, -1.1920929E-7]
sql:
SELECT row_id, label, _distance
FROM vector_search("table"="test_lance_vector_search_indexed_element_types.doris.vs_ivf_flat_f16_cosine", "column"="embedding", "query_vector"="[0.5212,-1.0089,1.0115,-0.8052,0.9277,-0.6251,0.7094,0.7868,-0.9736,0.6707,-0.4194,0.5045,0.5806,-0.3585,-0.7997,0.9973]", "top_k"="5", "metric"="cosine", "nprobes"="4", "use_index"="true")
ORDER BY _distance, row_id

@FANNG1 seems everyone could login as guest for TeamCity. no need admin account‌. u could have a try

FANNG1 · 2026-08-26T05:17:26Z

@FANNG1 seems everyone could login as guest for TeamCity. no need admin account‌. u could have a try

Got it, thanks!

…e and metric The Lance vector suites were all Float32 + L2. Doris only plans an indexed split when the query metric equals the metric the index was built with (LanceScanNode.metricMatches), so the cosine and dot branches of that comparison had never run against a real index, and no indexed table existed for any element type other than Float32. Two tiers now cover the whole matrix. Twelve tables carry it in depth: all six algorithms on Float32 + L2, plus cosine and dot, plus Float64, Float16 and UInt8. Each has goldens, a closed-form distance ladder where the data shape allows one, and a discriminator that proves the search parameter reached the index rather than falling back to a scan. Their data shape is per metric, because one shape cannot serve all of them - the collinear ladder is degenerate under cosine (directions converge, the top distances collapse to zero) and under dot (the answer ignores the query), so those tables use a directional shape, and UInt8 uses a thermometer code that makes hamming an exact ladder. One table, doris.vs_index_matrix, carries the remaining 44 cells at 64 rows, one vector column per cell. One column per cell rather than several indexes on one column, because only the first index built on a column is reachable - Lance answers the others with a silent brute-force scan, and Doris gets there by another route, since selectIndexSegments keeps only the segments of the first index it finds for the column's field id. Each cell is asserted to plan an indexed split with no fragment left scanned unindexed, to answer differently at nprobes=1 than at nprobes=4 (a scan has no partitions, so this is what distinguishes a real indexed search from a fallback), and to return what an exhaustive scan returns once its candidates are reranked. No goldens, so a fixture rebuild leaves it untouched. The generator's self-check is the whole contract for a fixture whose bytes are not reproducible, so it now pins a digest of every profile's vectors, reads rows back from each table, asserts the metric each index was actually built with (read from stats["indices"][*]["metric_type"], not searched for - a cosine IVF_PQ reports l2 on its sub-index), keeps the indexed-vs-flat comparison bit-exact for the integer ladders, and enforces the pylance and lance-namespace pins before building or verifying. Discriminator margins are reported and warned on when thin, and --repin lets a rebuild complete so those rows can be re-measured against the fixture that was actually promoted. buildable_combos() is a claim about Lance, so it was checked against Lance: all 96 cells of the matrix were built one per subprocess, 56 succeeded, and those 56 are exactly what it yields. The 40 failures are documented next to it and deliberately not attempted at build time - eighteen are Rust panics, and this script writes the fixture that gets committed. Finally, the fixture is published with `mc mirror --remove` instead of `mc cp -r`. The __manifest version is a commit count that a rebuild can lower, and Lance names version files u64::MAX - version so a listing resolves newest first; a stale higher-versioned manifest left in the bucket therefore wins over the one just published. The MinIO in a live test environment had accumulated three. Claude-Session: https://claude.ai/code/session_01J55jh5bX3t1DvTqbWx1qEN

FANNG1 · 2026-08-26T06:13:14Z

Thanks — that log made it clear, and the guest login works, so I can read TeamCity myself from now on.

The cause is narrower than the digit drift I guessed at. Cosine distance is 1 - cos_sim, and the cosine of a row against itself is mathematically exactly 1, so the distance is 0. It does not come out that way for float16: the query literal is the value before half-precision rounding while the column stores the value after, so the two are not bit-identical, cos_sim lands on 1 ± 1ulp, and the distance is ±2^-23 — which is exactly the 1.1920929E-7 in the golden. Which sign it takes depends on the accumulation order of the build running the query.

That is why the framework's 1e-6 relative tolerance did not absorb it: the rows below it passed, but relative error is meaningless around zero, and +eps against -eps comes out as 2.0.

Two things confirm the diagnosis:

Float32 cosine is unaffected. test_lance_vector_search_metrics runs the same metric over the same directional data and records a clean 0.0 self-distance, and it passed on this run. There the query literal and the stored value are bit-identical, so cos_sim really is 1.
Float16 L2 is unaffected. (x - x)^2 is 0 whatever the rounding; only a metric that divides by the vector norms introduces the ulp.

Fix

The Float16 table is the one table in this suite whose data has no closed-form expected distance — Float64 records the L2 ladder 0, 16, 64, 144, 256 and UInt8 the hamming ladder 0, 1, 2, 3, 4, both exact integers on any platform. So for Float16 the distance column was never recording a checkable value, only whatever the platform computed. Its two blocks now record row_id, label and the row order, which is the part that carries meaning. Float64 and UInt8 keep their distances.

The magnitude is still worth pinning — a backend that widened Float16 to Float32 before comparing, or lost the query vector's precision, would move it off one ulp — so it is now an assertion on the absolute value with a tolerance, rather than a literal whose sign a golden cannot avoid recording:

assertTrue(Math.abs(f16SelfDistance) < 1e-5, ...)

I also checked the rest of external_table_p0/lance for the same shape: this was the only near-zero float literal in any golden there, and f16_cosine_flat — added in this PR, and never reached on CI because the suite aborts at the first failure — had the identical exposure. Both are fixed.

Full external_table_p0/lance run locally on the new head: 13/14. The one failure is test_lance_show_index, which needs the FE-side Lance index metadata from #66637; the FE I have running locally predates it, and CI builds the FE from this branch, which does contain it.

zhangstar333 · 2026-08-26T06:27:30Z

run buildall

zhangstar333 · 2026-08-26T06:31:40Z

Thanks — that log made it clear, and the guest login works, so I can read TeamCity myself from now on.

The cause is narrower than the digit drift I guessed at. Cosine distance is 1 - cos_sim, and the cosine of a row against itself is mathematically exactly 1, so the distance is 0. It does not come out that way for float16: the query literal is the value before half-precision rounding while the column stores the value after, so the two are not bit-identical, cos_sim lands on 1 ± 1ulp, and the distance is ±2^-23 — which is exactly the 1.1920929E-7 in the golden. Which sign it takes depends on the accumulation order of the build running the query.

That is why the framework's 1e-6 relative tolerance did not absorb it: the rows below it passed, but relative error is meaningless around zero, and +eps against -eps comes out as 2.0.

Two things confirm the diagnosis:

Float32 cosine is unaffected. test_lance_vector_search_metrics runs the same metric over the same directional data and records a clean 0.0 self-distance, and it passed on this run. There the query literal and the stored value are bit-identical, so cos_sim really is 1.

Float16 L2 is unaffected. (x - x)^2 is 0 whatever the rounding; only a metric that divides by the vector norms introduces the ulp.

Fix

The Float16 table is the one table in this suite whose data has no closed-form expected distance — Float64 records the L2 ladder 0, 16, 64, 144, 256 and UInt8 the hamming ladder 0, 1, 2, 3, 4, both exact integers on any platform. So for Float16 the distance column was never recording a checkable value, only whatever the platform computed. Its two blocks now record row_id, label and the row order, which is the part that carries meaning. Float64 and UInt8 keep their distances.

The magnitude is still worth pinning — a backend that widened Float16 to Float32 before comparing, or lost the query vector's precision, would move it off one ulp — so it is now an assertion on the absolute value with a tolerance, rather than a literal whose sign a golden cannot avoid recording:
assertTrue(Math.abs(f16SelfDistance) < 1e-5, ...)
I also checked the rest of external_table_p0/lance for the same shape: this was the only near-zero float literal in any golden there, and f16_cosine_flat — added in this PR, and never reached on CI because the suite aborts at the first failure — had the identical exposure. Both are fixed.

Full external_table_p0/lance run locally on the new head: 13/14. The one failure is test_lance_show_index, which needs the FE-side Lance index metadata from #66637; the FE I have running locally predates it, and CI builds the FE from this branch, which does contain it.

get it. let's test again in TeamCity. @FANNG1
and the TeamCity could triggered‌ by comment,
Comment run buildall on your PR to trigger all TeamCity pipelines.
Comment /review on your PR to trigger AI 'code-review'

…y matrix (#4082) Documents the Lance vector index compatibility matrix requested in [apache/doris#66495](apache/doris#66495), whose completion criteria include "User documentation contains a supported algorithm compatibility matrix". Follows up on the [review comment on apache/doris#66512](apache/doris#66512 (comment)) asking for the integration capability matrix to be added to this page. Companion code PR: [apache/doris#67039](apache/doris#67039), which adds the regression coverage for the combinations documented here. Both `docs/` and the `zh-CN` translation are updated together. ## What is added **A vector element type x distance metric matrix** under "Supported Vector Index Types", describing which combinations can serve an index-backed search with the embedded Lance version, with footnotes for: - the Float16 L2 index-build limitation (and that Flat Search over Float16 is fine); - UInt8 being hamming-only, and available only for `IVF_FLAT` and `IVF_HNSW_FLAT`; - Int8 having no complete index build path, plus the nullable-Int8 crash and the upstream fix it is waiting on. **A "The Query Metric Must Match the Index Metric" section**, because it is the most likely user-visible surprise: Doris selects a vector index only when the requested `metric` equals the metric the index was built with, and otherwise runs Flat Search. `EXPLAIN` reports this as `lanceSearchIndexSegments=0`. It also notes that only one index per vector column is considered. ## Two corrections to existing text 1. **Version binding.** The page said the BE data reader is bound to Lance `9.1.0-beta.3` at commit `e934cc2c`. That is the `lance-java` version the **FE** uses (`fe/pom.xml`). The BE reader is `lance-c v0.1.6`, which pins the Lance Rust crates at **`7.0.0-beta.7`**, Lance commit **`e0e977a6`**. Both are now stated separately. The `data_storage_version` table below it stays correct: `rust/lance-encoding/src/version.rs` at `e0e977a6` has default `V2_1`, `Stable -> V2_1`, `Next -> V2_3`, and `2.2` stable but not default, which is what the table already describes. 2. **Default `metric`.** The `metric` parameter row and the paragraph after the parameter table both said that omitting `metric` makes Doris use the metric the index was created with. `LanceScanNode.metricMatches` maps an unset metric to `l2`, so on a cosine-indexed table an omitted `metric` plans a Flat Search instead. The documented behavior now matches the code, and says to pass `metric` explicitly for a non-L2 index.

github-actions · 2026-08-27T06:27:31Z

PR approved by anyone and no changes requested.

github-actions · 2026-08-27T06:27:33Z

PR approved by at least one committer and no changes requested.

FANNG1 requested a review from yiguolei as a code owner August 21, 2026 23:25

FANNG1 mentioned this pull request Aug 21, 2026

[doc](lance) Document the vector element type and metric compatibility matrix apache/doris-website#4082

Merged

FANNG1 marked this pull request as draft August 22, 2026 01:41

FANNG1 force-pushed the lance-metrics-elemtypes-66495 branch 2 times, most recently from af9fa35 to 54b999f Compare August 25, 2026 07:21

FANNG1 mentioned this pull request Aug 25, 2026

[Bug](lance) Vector search misses a later same-column index matching the query metric #67117

Open

7 tasks

FANNG1 force-pushed the lance-metrics-elemtypes-66495 branch from 54b999f to 75e20b8 Compare August 25, 2026 08:58

FANNG1 marked this pull request as ready for review August 25, 2026 09:12

FANNG1 force-pushed the lance-metrics-elemtypes-66495 branch from 75e20b8 to 2c329df Compare August 26, 2026 06:12

yiguolei approved these changes Aug 27, 2026

View reviewed changes

yiguolei merged commit 8a560dd into apache:branch-4.1 Aug 27, 2026
29 of 31 checks passed

github-actions Bot added the reviewed label Aug 27, 2026

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2026

FANNG1 deleted the lance-metrics-elemtypes-66495 branch August 27, 2026 08:27

Navigation Menu

Uh oh!

[test](lance) Add indexed coverage for cosine, dot and non-Float32 element types - #67039

[test](lance) Add indexed coverage for cosine, dot and non-Float32 element types#67039
yiguolei merged 1 commit into
apache:branch-4.1from
FANNG1:lance-metrics-elemtypes-66495

FANNG1 commented Aug 21, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

FANNG1 commented Aug 25, 2026

Uh oh!

zhangstar333 commented Aug 25, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026

Uh oh!

FANNG1 commented Aug 26, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026 •

edited

Loading

Uh oh!

FANNG1 commented Aug 26, 2026

Uh oh!

FANNG1 commented Aug 26, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026

Fix

Uh oh!

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

FANNG1 commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed?

The matrix is now covered in full, in two tiers

Fixture self-check

Fixture publishing

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

FANNG1 commented Aug 25, 2026

Uh oh!

zhangstar333 commented Aug 25, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026

Uh oh!

FANNG1 commented Aug 26, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FANNG1 commented Aug 26, 2026

Uh oh!

FANNG1 commented Aug 26, 2026

Fix

Uh oh!

zhangstar333 commented Aug 26, 2026

Uh oh!

zhangstar333 commented Aug 26, 2026

Fix

Uh oh!

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FANNG1 commented Aug 21, 2026 •

edited

Loading

zhangstar333 commented Aug 26, 2026 •

edited

Loading