[feature](lance) Support SHOW INDEX for Lance Directory tables

[u70b3]

What problem does this PR solve?

Issue Number: #66497

Related PR: #65730

Lance external tables can contain logical vector and scalar indexes, but SHOW INDEX currently only handles internal OlapTable metadata and returns an empty result for Lance tables.

This change reads authoritative logical index descriptions from one latest Lance Directory Dataset snapshot, resolves field IDs against the schema from that same snapshot, and exposes deterministic, bounded metadata through the existing 13-column SHOW INDEX result without changing internal-table behavior.

What changed?

• Add immutable Lance logical-index metadata and a loader backed by one latest Dataset snapshot.
• Support SHOW INDEX, SHOW INDEXES, SHOW KEY, and SHOW KEYS for Lance filesystem/Directory Catalog tables.
• Preserve existing internal-table behavior and permission-check ordering (privilege check runs before any catalog initialization).
• Return a fixed unsupported error for Lance REST catalogs in this phase.
• Sanitize Dataset URI, storage credential, and REST secret values from index metadata errors.
• Defensively reject non-Lance tables resolved from a Lance catalog instead of failing on a raw cast.
• Add unit and regression coverage for normalization, formatting, permissions, aliases, empty-index tables, and error handling.

Design walkthrough (diagrams)

SHOW INDEX for Lance Directory tables is a small read pipeline whose main concern is object ownership across the JNI boundary: SDK objects (Dataset, IndexDescription, Arrow allocators) never escape the loader, and the command layer only sees immutable LanceLogicalIndex values.

Layered view — three new classes (LanceLogicalIndex, LanceIndexMetadataLoader, LanceMetadataReadExecutor) and three touch points (ShowIndexCommand, LanceExternalTable, LanceExternalCatalog):

flowchart TB
  subgraph CMD["Command layer · Nereids"]
    A["ShowIndexCommand<br/>analyze() privilege check (:83)<br/>handleShowIndex() dispatch (:111)<br/>buildLanceRows() row mapping (:159)"]
  end
  subgraph EXT["External table layer"]
    B["LanceExternalTable<br/>loadIndexMetadata() (:69) — thin delegate"]
  end
  subgraph CAT["Catalog layer · caller thread"]
    C["LanceExternalCatalog<br/>isRestCatalogConfigured() (:172) early REST rejection<br/>resolveTableAccess() (:371) namespace resolution"]
    C2["sanitizedRootCauseMessage() (:473)<br/>URI / credential / token redaction"]
  end
  subgraph BND["Execution boundary · lance-metadata-read pool"]
    D["LanceMetadataReadExecutor (:50)<br/>4 concurrent · 16 queued · deadline ≤ 60s"]
    E["LanceIndexMetadataLoader<br/>load() (:82) → normalize() (:206)"]
  end
  subgraph NAT["native · JNI"]
    F["Dataset.open(uri, latest)<br/>schema + describeIndices from one snapshot"]
  end
  G["LanceLogicalIndex<br/>immutable value object · pure-Java boundary"]
  A --> B --> C --> D --> E --> F
  E --> G
  C -.->|any failure| C2
  C2 -.->|redacted exception only| A

Loading

Request timeline and resource ownership — namespace resolution stays on the caller thread (it owns the catalog's shared namespace/allocator), the JNI read runs on a bounded worker with a task-owned allocator, and a timed-out caller never cancels the in-flight native call:

sequenceDiagram
  autonumber
  participant U as User
  participant C as Caller thread<br/>ShowIndexCommand
  participant K as Caller thread<br/>LanceExternalCatalog
  participant X as Executor<br/>boundary
  participant W as Worker thread<br/>lance-metadata-read
  participant J as Lance JNI<br/>Dataset
  U->>C: SHOW INDEX FROM lance_db.t
  C->>C: analyze() → checkTblPriv(SHOW)
  Note over C: privilege check runs before any catalog initialization
  C->>K: isRestCatalogConfigured()
  Note over K: reads normalized properties only, no namespace init —<br/>REST catalogs rejected here with a fixed error
  C->>K: resolveTableAccess()
  Note over K: shared namespace / allocator owned by the caller thread,<br/>so catalog close stays safe
  K-->>C: ResolvedTableAccess(uri, options)
  C->>X: execute(task)
  X->>W: submit (wrapped task checks deadline first)
  W->>W: expired in queue? fail before entering JNI
  W->>J: open(latest) + schema + describeIndices
  Note over W,J: task-owned RootAllocator (256MB) —<br/>catalog close cannot release it early
  J-->>W: IndexDescription + LanceSchema
  W-->>X: immutable index list
  X-->>C: future.get(remaining deadline)
  Note over C,X: timeout / interrupt never cancels the Future —<br/>the worker stays sole owner of native resources
  C->>C: buildLanceRows() → 13 columns
  C-->>U: ShowResultSet

Loading

Loader internals — one snapshot feeds both the schema mapping and the index descriptions; normalization is fail-closed and produces deterministic, bounded output:

flowchart LR
  O["Dataset.open(uri, latest)<br/>index cache 0 · metadata cache 64MB<br/>schema and indexes from one snapshot"]
  O --> S["getLanceSchema().fields()"]
  O --> L["listIndexes()<br/>physical entries ≤ 16384"]
  subgraph SB["schema branch"]
    S --> DFS["buildFieldNamesById() (:148)<br/>DFS depth ≤ 64 · fields ≤ 16384<br/>field id → dotted path, quoted when needed"]
  end
  subgraph IB["index branch"]
    L --> FL["drop system indexes<br/>__lance_frag_reuse · __lance_mem_wal"]
    FL --> DD["dedupe by logical name ≤ 256"]
    DD --> CR["describeUserIndexes() (:95)<br/>per-name describeIndices(hasName)<br/>must return exactly 1, name must match"]
  end
  DFS --> V
  CR --> V
  subgraph NB["normalize() (:206) · fail-closed"]
    V["known, unique field ids<br/>columns ≤ 64 · names ≤ 16KB total"] --> P["details JSON allowlist<br/>TreeMap-sorted · ≤ 400B"]
    P --> R["exact duplicate names rejected<br/>sorted by (name, position)"]
  end
  R --> OUT["LanceLogicalIndex list<br/>immutable · deterministic"]

Loading

Fail-closed semantics (intentional)

Three strict behaviors are deliberate and locked by unit tests — the command reports an error rather than showing potentially misleading metadata:

• An index referencing an unknown or nested field ID fails the whole SHOW INDEX (testRejectsUnknownNestedDuplicateNullAndEmptyFieldIds).
• One malformed index entry (invalid details JSON, oversize fields, duplicate name) fails the whole result instead of silently dropping rows (testRejectsMalformedAndNonObjectJsonWithoutEchoingInput).
• Index names differing only by case are preserved as distinct entries; only exact duplicates are rejected (testRejectsExactDuplicateNameButPreservesCaseOnlyNames).

User/developer impact

Users can inspect logical Lance indexes through the standard 13-column SHOW INDEX interface for filesystem/Directory catalogs. Lance REST index inspection remains unsupported.

Release note

Support SHOW INDEX for Lance filesystem/Directory Catalog tables.

Check List (For Author)

• Unit tests: 28 tests passed
• Full FE build: all 27 modules passed
• Lance regression test: generation and normal verification modes passed
• git diff --check
• Behavior changed: SHOW INDEX now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs
• Documentation needed: document the supported Directory-only scope and REST limitation

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[u70b3]

/review

[u70b3]

run buildall

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 14.46% (59/408) 🎉
Increment coverage report
Complete coverage report

[yiguolei]

@u70b3 the new added regression test failed，please take a look

[u70b3]

Thanks for pointing this out. The issue has been fixed and pushed in commit 0ef748c49. I also verified the fix in a clean environment with no residual fixture data.

[u70b3]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 74.67% (171/229) 🎉
Increment coverage report
Complete coverage report

[u70b3]

run buildall

[u70b3]

@yiguolei @zhangstar333 friendly ping 🙂 Could one of you help approve the pending CI workflows on this PR when convenient? Seven checks (License Check, Code Formatter, FE Code Style Checker, Dependency License Review, BE UT macOS, Build Third Party Libraries, Build Extensions) are waiting on maintainer approval. The regression failure flagged earlier has been fixed and verified in a clean environment. Thanks a lot!

[yiguolei]

/review

[github-actions]

Requesting changes for four correctness gaps in the new Lance SHOW INDEX path.

Critical checkpoints:

• Correctness and compatibility: internal-table behavior and the four SHOW aliases remain intact, but valid nested-field indexes are rejected, PQ/HNSW properties are silently discarded, and valid Lance system indexes can abort the all-index description.
• Privilege, security, and lifecycle: SHOW privilege is checked before catalog I/O; table-vended credential overlay/redaction, one-Dataset snapshot consistency, allocator ownership, and cleanup look sound. The REST unsupported gate is too late and can perform namespace/table I/O first.
• Tests and scope: the new tests cover flat schemas, formatting, aliases, permissions, and basic errors, but they encode flattened provider JSON and nested-ID rejection, and miss system-plus-user-index and unreachable-REST routes. No additional user focus was supplied.
• Review completeness: the full 13-file authoritative diff and the relevant exact upstream version contracts were reviewed. The second normal/risk-focused round converged with no new non-duplicate findings. Builds/tests were not run under the review constraints.

[github-actions]

[P1] Parse the SDK's nested vector details. In the pinned Lance version, IVF-PQ details are serialized with a nested compression object, and HNSW settings are nested under hnsw; none of the flattened keys in this allowlist are emitted. Consequently those objects are silently dropped—the new regression already shows only metric_type although its fixture creates PQ with 8 bits and 2 sub-vectors. Please parse a bounded allowlist within the actual nested objects and test with the SDK-shaped JSON.

[u70b3]

Fixed in 8b58182. normalizeProperties now parses the SDK's actual nested shape: a top-level allowlist (metric_type, target_partition_size) plus bounded nested allowlists for compression (type/num_bits/num_sub_vectors/rotation_type) and hnsw (construction_ef/max_connections/max_level). The regression now asserts the nested form, e.g. {"compression":{"num_bits":4,"num_sub_vectors":4,"type":"pq"},"metric_type":"L2"}.

[github-actions]

[P1] Resolve valid nested field IDs. Lance indexes can target nested fields, and describeIndices() returns the indexed leaf ID, but this map contains only schema roots, so normalization later rejects the ID and fails SHOW INDEX for the entire table. The pinned Lance tag exposes LanceField.getChildren() and constructs canonical escaped paths from field ancestry (including names containing dots). Please recurse the schema into a complete ID-to-path map and replace the negative nested-ID test with a real nested-index case.

[u70b3]

Fixed in 8b58182. buildFieldNamesById now DFS-traverses the full schema (depth ≤ 64, ≤ 16,384 fields) and maps every field ID to its canonical dotted path with backtick-escaped segments, so indexes on nested fields resolve instead of failing the whole SHOW INDEX. Covered by a real nested-index regression case (nested_label_btree on a child field whose name contains a dot).

[github-actions]

[P2] Handle valid Lance system indexes before describing all entries. In the exact pinned SDK, optimized/MemWAL datasets can contain __lance_frag_reuse or __mem_wal with empty fields. describe_indices(None) includes them, but JNI unconditionally renders each entry's details and these system detail types have no scalar plugin, so this call aborts before any user index is returned; the local empty-field check would reject them after an SDK fix as well. Please pin/use a conversion that handles system entries and deliberately exclude them from SHOW INDEX, with a system-plus-user-index regression.

[u70b3]

Fixed in 8b58182. Bulk describeIndices() is no longer used. describeUserIndexes() lists physical entries via listIndexes(), filters __lance_frag_reuse / __lance_mem_wal, dedupes by logical name, then describes each user index through an exact IndexCriteria.hasName lookup (validating exactly-one cardinality and name match), so system entries never reach the details path.

[github-actions]

[P2] Gate REST catalogs before database resolution. On a cache miss this call populates the external meta-cache by listing Lance namespaces, and table resolution can then list remote tables; the REST-unsupported check is not reached until loadTableIndexMetadata. An unavailable REST endpoint therefore returns an unknown-database/provider failure after unnecessary I/O instead of the fixed unsupported error promised by this change. After the privilege check, reject the locally configured REST type before resolving the database/table, and cover the route with an unreachable namespace.

[u70b3]

Fixed in 8b58182. handleShowIndex now rejects a configured Lance REST catalog immediately after the privilege check and before getDbOrAnalysisException (ShowIndexCommand.java:117), so the fixed unsupported error is returned without any namespace or table I/O.

[u70b3]

run buildall

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 53.41% (196/367) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 31.36% (196/625) 🎉
Increment coverage report
Complete coverage report

[u70b3]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 74.66% (274/367) 🎉
Increment coverage report
Complete coverage report

[u70b3]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 74.66% (274/367) 🎉
Increment coverage report
Complete coverage report

[zhangstar333]

do not cancel or interrupt the Future
Will this cause the thread pool to be exhausted, making it unusable for everyone?

[u70b3]

Good question — this boundary is the trade-off the pool is built around. Short answer: temporary exhaustion under pathological storage failure is possible and intentional (bounded fail-fast); permanent exhaustion is not, and cancelling the Future would change neither.

Why cancelling doesn't help. cancel(true) only delivers Thread.interrupt(). The worker is blocked inside a synchronous JNI call; the Lance native runtime does not observe Java interrupts, so the thread stays occupied until the native call returns regardless. Cancellation would only flip the Future's state while the worker — the sole owner of the Dataset and its task-scoped 256MB allocator — is still running, re-introducing the ownership race this boundary exists to prevent. It doesn't even free the queue slot: a cancelled FutureTask stays in the queue until a worker dequeues it.

Why the pool still recovers. A stuck native call is already time-bounded inside the pinned SDK (lance-core 9.1.0-beta.3 → object_store 0.13.2): every S3 attempt terminates via connect_timeout=5s / timeout=30s defaults, and the retry loop stops at max_retries=3 or retry_timeout=180s elapsed. So one object-store op returns in ≲3.5 min even against an endpoint that hangs every request, and one SHOW INDEX task issues only a bounded number of such ops (describeIndices reads the already-open snapshot's manifest). Worst-case worker occupancy is minutes, not forever; expired queued tasks then fail before entering JNI and the pool drains within one deadline.

What users see during that window. Submissions beyond 4 running + 16 queued fail immediately with MetadataReadCapacityException — deliberate fail-fast backpressure rather than unbounded queuing (each in-flight task owns a 256MB Arrow allocator, so an unbounded pool is a native-memory risk). The waiting caller is bounded by min(query_timeout, 60s), and the blast radius is limited to Lance metadata reads on this FE; query execution and other catalogs never touch this pool.

Residual risk I consider acceptable: file:// datasets on a hung filesystem (NFS hard mount) can pin a worker indefinitely — local disk I/O has no timeout, and no pool design, cancellation included, can reclaim a thread in uninterruptible sleep.

If you prefer a tighter bound, we can pass an explicit client_retry_timeout (e.g. 30s) with the SHOW INDEX read options as a Doris-side default, so a stuck worker returns within roughly the caller deadline instead of the 180s SDK default. WDYT?

[zhangstar333]

get, maybe keep SDK default is ok

[u70b3]

/review

[zhangstar333]

@u70b3 some file conflicts, u could rebase code, and we merge this pr firstly.

[zhangstar333]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 74.25% (274/369) 🎉
Increment coverage report
Complete coverage report

[u70b3]

run buildall

[u70b3]

run buildall

[u70b3]

/review

[zhangstar333]

run buildall

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 53.66% (198/369) 🎉
Increment coverage report
Complete coverage report

np

[zhangstar333]

hi @u70b3
https://github.com/apache/doris-website/blob/master/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
u could update the doc about index, after this user could know how to use command.