[test](lance) Build the missing IVF_PQ regression fixture by FANNG1 · Pull Request #66779 · apache/doris

FANNG1 · 2026-08-14T10:04:02Z

What problem does this PR solve?

Issue Number: Part of #66495

Problem Summary: test_lance_vector_search documents its doris.vector_search
fixture as carrying an IVF_PQ index, but the fixture SQL
(run07_create_vector_types.sql) delegated index creation to a companion
create_vector_search_index.py that was never committed, and lance-spark-bundle
0.4.0 cannot create vector indexes through SQL. The table therefore had no
index at all, so every use_index / nprobes / refine_factor query in the suite
silently executed a flat KNN scan while the goldens still looked correct.
Nothing in the repository could observe the difference.

Reproduction: build the table the old fixture built and probe it with
nprobes=1. Lance ignores nprobes on an unindexed dataset and returns exactly
the flat top-10 (rows 256,255,257,254,258,253,259,252,260,251 for the boundary
query) - identical to the flat baseline, which is why the defect was invisible.

Fix: replace the Spark-created table with an offline-generated Directory
Namespace V2 catalog that carries a real IVF_PQ index, and add the evidence
that the index is actually used.

lance_build_preinstalled_catalog.py rebuilds the committed fixture and
self-checks it: exactly one IVF_PQ index named embedding_ivf_pq_f32 covering
every fragment, ANNSubIndex and ANNIvfPartition present in the indexed plan,
KNNVectorDistance and no ANN node in the flat plan, and the exact
16 * (n - r)^2 distance ladder that every golden and comment encodes, so a
change to the data shape fails here instead of surfacing as an opaque golden
diff. Index creation goes through the physical dataset because
DirectoryNamespace.create_table_index raises UnsupportedOperationError.
doris.vs_ivf_pq_f32 replaces doris.vector_search: 1024 rows in two
fragments, 16-dimensional Float32 embedding[j] = (row_id - 1) + j, so the
exact squared L2 distance between rows r and n is 16 * (n - r)^2 and the
head/tail queries have no distance ties. Columns are declared NOT NULL to
match the fixture being replaced, keeping the only non-nullable Lance column
mapping recorded by any Lance suite's DESC golden. The vs__
name encodes one cell of the algorithm x element type matrix, so a missing
combination is visible from the table list alone.
The suite gains a silent-fallback discriminator. Row 256 sits on the first
IVF partition boundary, so a genuine single-partition probe must miss true
neighbours from the next partition. The suite asserts that the nprobes=1
distance sequence differs from flat search; on the previous unindexed
fixture the two are identical and the assertion fails. Distances are
compared rather than row ids because the boundary query is symmetric and
rows r-d and r+d tie. top_k is 9 there, the last cut that lands on a
complete tie pair: at 10 the pair at distance 400 is split, so the golden
would pin an arbitrary winner that any change to Lance's top-k selection
could flip. Which partition edge row 256 lands next to changes on every
retrain, so no measured range is hardcoded; --check prints it instead.
IVF_PQ is lossy, so every indexed query uses refine_factor and the suite
documents indexed/flat agreement as an observed property of this frozen
fixture and pinned Lance version, not an algorithm guarantee.

The fixture is generated with the pins in lance_fixture_requirements.txt.
Its readers do not all run the same Lance version - a BE built from source uses
lance-c v0.1.2 (lance-rs 4.0.1) per thirdparty/vars.sh, the BE in CI comes from
the prebuilt doris-thirdparty package and is already on lance-c v0.1.6
(lance-rs 7.0.0-beta), and Spark writes into the same __manifest through
lance-java 4.0.0. The writer is therefore pinned to the oldest Lance in that
set, which every reader can read. Verified that this does not make the goldens
version-dependent: pylance 7.0.0 reads the committed fixture with results
identical to pylance 4.0.1 - same index, same refined top-5, same nprobes=1
boundary rows, same IVF partition ranges.

Index training is not bit-reproducible, so regenerating the fixture changes
the binary output; the reproducible properties are asserted by the generator
self-check instead. IVF_FLAT, IVF_SQ, IVF_HNSW_* and the other vector element
types are follow-up work for #66495.

Release note

None

Check List (For Author)

Test: Regression test
- Fixture generator self-check with the pinned dependencies
- test_lance_vector_search regenerated with -forceGenOut, then passed the
  normal golden comparison
- The whole external_table_p0/lance directory passed (6 suites, 0 failed),
  covering the pre-existing suites that share the regenerated __manifest
- Cross-checked that the nprobes=1 golden row order matches what pylance
  records probing the same physical index directly
Behavior changed: No, test fixture and regression coverage only
Does this need documentation: No

hello-stephen · 2026-08-14T10:04:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: Part of apache#66495 Problem Summary: test_lance_vector_search documents its doris.vector_search fixture as carrying an IVF_PQ index, but the fixture SQL (run07_create_vector_types.sql) delegated index creation to a companion create_vector_search_index.py that was never committed, and lance-spark-bundle 0.4.0 cannot create vector indexes through SQL. The table therefore had no index at all, so every use_index / nprobes / refine_factor query in the suite silently executed a flat KNN scan while the goldens still looked correct. Nothing in the repository could observe the difference. Reproduction: build the table the old fixture built and probe it with nprobes=1. Lance ignores nprobes on an unindexed dataset and returns exactly the flat top-10 (rows 256,255,257,254,258,253,259,252,260,251 for the boundary query) - identical to the flat baseline, which is why the defect was invisible. Fix: replace the Spark-created table with an offline-generated Directory Namespace V2 catalog that carries a real IVF_PQ index, and add the evidence that the index is actually used. - lance_build_preinstalled_catalog.py rebuilds the committed fixture and self-checks it: exactly one IVF_PQ index named embedding_ivf_pq_f32 covering every fragment, ANNSubIndex and ANNIvfPartition present in the indexed plan, KNNVectorDistance and no ANN node in the flat plan, and the exact 16 * (n - r)^2 distance ladder that every golden and comment encodes, so a change to the data shape fails here instead of surfacing as an opaque golden diff. Index creation goes through the physical dataset because DirectoryNamespace.create_table_index raises UnsupportedOperationError. - doris.vs_ivf_pq_f32 replaces doris.vector_search: 1024 rows in two fragments, 16-dimensional Float32 embedding[j] = (row_id - 1) + j, so the exact squared L2 distance between rows r and n is 16 * (n - r)^2 and the head/tail queries have no distance ties. Columns are declared NOT NULL to match the fixture being replaced, keeping the only non-nullable Lance column mapping recorded by any Lance suite's DESC golden. The vs_<algorithm>_<element type> name encodes one cell of the algorithm x element type matrix, so a missing combination is visible from the table list alone. - The suite gains a silent-fallback discriminator. Row 256 sits on the first IVF partition boundary, so a genuine single-partition probe must miss true neighbours from the next partition. The suite asserts that the nprobes=1 distance sequence differs from flat search; on the previous unindexed fixture the two are identical and the assertion fails. Distances are compared rather than row ids because the boundary query is symmetric and rows r-d and r+d tie. top_k is 9 there, the last cut that lands on a complete tie pair: at 10 the pair at distance 400 is split, so the golden would pin an arbitrary winner that any change to Lance's top-k selection could flip. Which partition edge row 256 lands next to changes on every retrain, so no measured range is hardcoded; --check prints it instead. - IVF_PQ is lossy, so every indexed query uses refine_factor and the suite documents indexed/flat agreement as an observed property of this frozen fixture and pinned Lance version, not an algorithm guarantee. The fixture is generated with the pins in lance_fixture_requirements.txt. Its readers do not all run the same Lance version - a BE built from source uses lance-c v0.1.2 (lance-rs 4.0.1) per thirdparty/vars.sh, the BE in CI comes from the prebuilt doris-thirdparty package and is already on lance-c v0.1.6 (lance-rs 7.0.0-beta), and Spark writes into the same __manifest through lance-java 4.0.0. The writer is therefore pinned to the oldest Lance in that set, which every reader can read. Verified that this does not make the goldens version-dependent: pylance 7.0.0 reads the committed fixture with results identical to pylance 4.0.1 - same index, same refined top-5, same nprobes=1 boundary rows, same IVF partition ranges. Index training is not bit-reproducible, so regenerating the fixture changes the binary output; the reproducible properties are asserted by the generator self-check instead. IVF_FLAT, IVF_SQ, IVF_HNSW_* and the other vector element types are follow-up work for apache#66495. ### Release note None ### Check List (For Author) - Test: Regression test - Fixture generator self-check with the pinned dependencies - test_lance_vector_search regenerated with -forceGenOut, then passed the normal golden comparison - The whole external_table_p0/lance directory passed (6 suites, 0 failed), covering the pre-existing suites that share the regenerated __manifest - Cross-checked that the nprobes=1 golden row order matches what pylance records probing the same physical index directly - Behavior changed: No, test fixture and regression coverage only - Does this need documentation: No

Gabriel39 · 2026-08-15T07:38:18Z

run buildall

github-actions · 2026-08-15T08:10:51Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-08-15T08:10:54Z

PR approved by anyone and no changes requested.

branch-4.1 apache#66779 replaced the Spark-built vector fixture with the committed offline catalog: doris.vector_search is now doris.vs_ivf_pq_f32 with index embedding_ivf_pq_f32 (num_bits=4, num_sub_vectors=4), and the run07 SQL plus its companion Java builder are gone. The nested_index/BTree and __lance_frag_reuse coverage now lives in lance_build_preinstalled_catalog.py and the committed fixture. Golden regenerated with -forceGenOut and verified against a fresh local cluster plus a freshly rebuilt docker MinIO fixture; the full external_table_p0/lance directory passes (7/7). For apache#66497.

Directory-only golden coverage for the physical entry TVF against the preinstalled fixture (apache#66779): doris.vs_ivf_pq_f32 yields exactly one user entry (embedding_ivf_pq_f32), doris.nested_index yields nested_label_btree while its reserved __lance_frag_reuse system entry is filtered, the unindexed table yields zero rows, an ordinary predicate filters the bounded result, REST catalogs are rejected with a fixed pre-initialization error, and a user without SHOW privilege is denied. For apache#66497 PR2 slice 2.

Issue Number: Part of apache#66495 The vector_search() path is algorithm-agnostic - the BE forwards metric, nprobes, refine_factor and ef to lance-c and lets Lance choose the index - but the only algorithm it has ever been exercised against is IVF_PQ (apache#66779). Nothing in the repository could say whether Doris reads an IVF_FLAT, IVF_SQ or graph index at all, so five of the six entries on the issue checklist rested on an argument about the code path rather than on a test. Extend the committed fixture with one table per remaining algorithm and add the suites that query them: - lance_build_preinstalled_catalog.py gains five VECTOR_TABLES entries - vs_ivf_flat_f32, vs_ivf_sq_f32, vs_ivf_hnsw_flat_f32, vs_ivf_hnsw_sq_f32 and vs_ivf_hnsw_pq_f32 - each carrying one index of its own algorithm over the same 1024-row, two-fragment, 16-dim Float32 data the IVF_PQ table already uses, so a table is exactly one cell of the algorithm matrix. The self-check verifies every index the same way it verified IVF_PQ: exactly one index of the expected type and name, covering every fragment, ANNSubIndex and ANNIvfPartition in the indexed plan, KNNVectorDistance and no ANN node in the flat plan, and the row-256 nprobes=1 discriminator that fails if a query silently falls back to a flat scan. - Only IVF_FLAT gets its indexed-equals-flat equality asserted, since it stores the original vectors and a full-partition probe is an exhaustive scan by another name. IVF_SQ, IVF_HNSW_* and IVF_PQ quantize or traverse a graph, so their agreement with the flat search is recorded, never asserted, and their queries use refine_factor. - The graph indexes need a second discriminator, because nprobes alone cannot show that ef reached the index. Measured on this data, a query at row 512 loses a true neighbour at ef=5 that ef=50 finds - but only on IVF_HNSW_SQ; on 1024 collinear vectors the FLAT and PQ graphs still return the exact rows at ef=5. The generator therefore asserts the ef discriminator on the one table that can carry it, records it for the others, and the suite queries that same table. - BOUNDARY_TOP_K makes the generator probe the boundary at the k the suites actually use. It checked k=10 while the suites query k=9, so a retrained index could have passed the generator and failed the suites. test_lance_vector_search_ivf_flat asserts the IVF_FLAT guarantee against Doris at both ends of the dataset; test_lance_vector_search_index_types covers the other four with per-table nprobes=1 discriminators, the ef pair, and the "ef must be greater than or equal to k" error a too-narrow refined graph query must keep producing. The writer pin stays at pylance 4.0.1, now with the evidence that matters after apache#66786 moved branch-4.1 to lance-c v0.1.6: pylance 7.0.0 - the Lance generation the BE reads through v0.1.6 - reproduces the generator self-check line for line on all six tables, and all 35 queries these three suites issue return identical rows and distances under both. The goldens therefore depend on the frozen fixture bytes, not on which Lance version reads them. Claude-Session: https://claude.ai/code/session_01BQAjAdyfr5Rr94WA4WE45S