fix: preserve complete blob v2 logical schemas - #8929
Conversation
…ion' into xuanwo/harden-blob-v2-normalization
| requires `uri`, and an explicit range must have `size > 0`. Use inline `b""` for | ||
| an empty blob; a URI without range fields still represents the complete external | ||
| object, including an empty object. Python's `blob_field` and `BlobType` use the | ||
| complete shape. Lance preserves an accepted logical shape, including child |
There was a problem hiding this comment.
The two documented logical shapes are still not append-compatible. Rust blob_field creates the minimal shape, while Python produces the complete shape; appending complete input to a minimal dataset fails with unexpected=[blob.position, blob.size] (the reverse direction succeeds). Please make Blob schema compatibility symmetric and add a mixed-shape append test; otherwise a Rust-created dataset cannot be appended with the standard Python BlobType.
| This is the "logical" type users write. Lance will store it in a compact | ||
| descriptor format, and reads will return descriptors by default. | ||
| descriptor format, and reads will return descriptors by default. Its storage | ||
| type is ``Struct<data: LargeBinary?, uri: Utf8?, position: UInt64?, |
There was a problem hiding this comment.
BlobType.__arrow_ext_deserialize__ ignores the supplied storage_type and always returns the canonical four-child type. A minimal schema round-trips as complete, while child metadata and required position/size nullability are lost. Please reconstruct from a validated storage type and assert exact storage-field equality in the round-trip tests.
There was a problem hiding this comment.
✅ Gate recommendation: approve.
All three previously blocking boundaries are now covered: accepted Blob v2 logical shapes append symmetrically, Python preserves validated storage schemas through Arrow reconstruction, and empty external sources return before issuing an empty range read. The focused cross-language, IPC, pickle, malformed-layout, and backend-independent regressions support the end-to-end contract.
Blob v2 schema normalization must distinguish logical writer input from the prepared writer intermediate. Rebuilding an already-logical field can collapse the complete
data, uri, position, sizeshape to the minimal form and lose schema properties. Prepared child IDs must also follow the semanticdataandurifields instead of their positions in the prepared layout.Logical minimal and complete schemas now pass through normalization unchanged, while prepared input alone normalizes to the minimal logical shape with IDs matched by child name. Descriptor and malformed layouts remain explicit errors. The contract is exercised through create, append, merge-insert, external-range, and nested Rust/Python paths and is documented as public behavior.
Blob v2 is beta, so this enforces the complete invariant directly without compatibility handling for intermediate beta schemas.
A mutation check that routes logical input through the prepared normalization branch makes all logical identity matrix cases fail; restoring the intended branch makes the full matrix pass.