Skip to content

fix: preserve complete blob v2 logical schemas - #8929

Merged
Xuanwo merged 12 commits into
mainfrom
xuanwo/harden-blob-v2-normalization
Sep 2, 2026
Merged

fix: preserve complete blob v2 logical schemas#8929
Xuanwo merged 12 commits into
mainfrom
xuanwo/harden-blob-v2-normalization

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 1, 2026

Copy link
Copy Markdown
Member

Blob v2 schema normalization must distinguish logical writer input from the prepared writer intermediate. Rebuilding an already-logical field can collapse the complete data, uri, position, size shape to the minimal form and lose schema properties. Prepared child IDs must also follow the semantic data and uri fields instead of their positions in the prepared layout.

Logical minimal and complete schemas now pass through normalization unchanged, while prepared input alone normalizes to the minimal logical shape with IDs matched by child name. Descriptor and malformed layouts remain explicit errors. The contract is exercised through create, append, merge-insert, external-range, and nested Rust/Python paths and is documented as public behavior.

Blob v2 is beta, so this enforces the complete invariant directly without compatibility handling for intermediate beta schemas.

A mutation check that routes logical input through the prepared normalization branch makes all logical identity matrix cases fail; restoring the intended branch makes the full matrix pass.

@github-actions github-actions Bot added A-python Python bindings A-docs Documentation bug Something isn't working labels Sep 1, 2026
@Xuanwo
Xuanwo marked this pull request as ready for review September 1, 2026 17:05
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 1, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 1, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 1, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026
Comment thread docs/src/guide/blob.md
requires `uri`, and an explicit range must have `size > 0`. Use inline `b""` for
an empty blob; a URI without range fields still represents the complete external
object, including an empty object. Python's `blob_field` and `BlobType` use the
complete shape. Lance preserves an accepted logical shape, including child

@BubbleCal BubbleCal Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two documented logical shapes are still not append-compatible. Rust blob_field creates the minimal shape, while Python produces the complete shape; appending complete input to a minimal dataset fails with unexpected=[blob.position, blob.size] (the reverse direction succeeds). Please make Blob schema compatibility symmetric and add a mixed-shape append test; otherwise a Rust-created dataset cannot be appended with the standard Python BlobType.

Comment thread python/python/lance/blob.py Outdated
This is the "logical" type users write. Lance will store it in a compact
descriptor format, and reads will return descriptors by default.
descriptor format, and reads will return descriptors by default. Its storage
type is ``Struct<data: LargeBinary?, uri: Utf8?, position: UInt64?,

@BubbleCal BubbleCal Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BlobType.__arrow_ext_deserialize__ ignores the supplied storage_type and always returns the canonical four-child type. A minimal schema round-trips as complete, while child metadata and required position/size nullability are lost. Please reconstruct from a validated storage type and assert exact storage-field equality in the round-trip tests.

@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

All three previously blocking boundaries are now covered: accepted Blob v2 logical shapes append symmetrically, Python preserves validated storage schemas through Arrow reconstruction, and empty external sources return before issuing an empty range read. The focused cross-language, IPC, pickle, malformed-layout, and backend-independent regressions support the end-to-end contract.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@Xuanwo
Xuanwo merged commit 9c99a39 into main Sep 2, 2026
38 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/harden-blob-v2-normalization branch September 2, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-docs Documentation A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants