Proposal: Encoded File Concatenation and Column Slices

[Xuanwo]

Summary

This proposal adds two related low-level capabilities:

1. A single lance-file primitive for concatenating compatible encoded Lance files without decoding and re-encoding their page payloads.
2. A FileFragment API for writing independently computed physical-row slices of one or more existing columns and combining a complete set of slices into a DataReplacementGroup.

The API is intentionally a mechanism, not a distributed execution framework. Lance does not own task planning, scheduling, retries, progress, recovery, or commit granularity. Those remain the responsibility of Ray, Spark, or another caller-controlled coordinator.

No Lance file layout, manifest feature, reader layout, or transaction operation is added.

Motivation

Binary-copy compaction currently owns both compatibility checks and buffer-copy rewriting inside the dataset layer. This duplicates file-format knowledge and risks eligibility checks drifting from the actual rewrite behavior.

Distributed column recomputation has the same underlying file operation. Workers can independently compute and encode row-aligned pieces of one or more columns, but the final replacement needs a regular Lance file covering the entire fragment. Concatenating those already encoded files should not require decoding Arrow arrays or rerunning the UDF.

These two callers should share the encoded-file transformation while retaining their own dataset semantics:

• Compaction owns fragment grouping, deletions, row IDs, index remapping, and transactions.
• A distributed column writer owns work partitioning, retries, result collection, and publication.

Encoded file concatenation

The proposed one-shot primitive is conceptually:

concat_files(
    target,
    ordered_inputs: [EncodedFileInput],
    output_factory,
    options,
) -> Written(FileConcatOutput)
   | Reused(input_index, FileConcatOutput)
   | Unsupported(FileConcatReason)

Each input is a complete immutable Lance file. The primitive does not expose an internal plan and does not slice ranges out of a file.

It reads metadata once, determines the actions required for page, column, and file buffers, copies page payloads unchanged, relocates offsets and row-relative metadata, and writes a new footer. Exact-version details such as structural headers, alignment, and footer behavior remain in version-specific hooks.

Unsupported means the files cannot be safely concatenated without re-encoding. Missing or corrupt inputs, short reads, and I/O failures remain errors and must not be treated as a compatibility fallback. output_factory is only invoked after compatibility has been established. A single compatible full input can return Reused without creating another object.

FileFragment API

The proposed Rust surface is:

FileFragment::write_columns(
    data,
    schema,
) -> DataReplacementGroup

FileFragment::write_columns_slice(
    rows: Range<u64>,
    data,
    schema,
) -> ColumnSlice

FileFragment::concat_column_slices(
    slices: [ColumnSlice],
) -> DataReplacementGroup

write_columns is the full-fragment path. The plural name reflects that schema may contain one or more existing top-level fields.

write_columns_slice writes the same fields for a fragment-local, half-open physical row interval. slice is used deliberately: Lance range reads can have logical semantics and apply deletion vectors, while this API is positional. The input must contain exactly rows.end - rows.start rows, including values at deleted positions.

ColumnSlice is an opaque immutable value describing a completed ordinary Lance file. It contains only the storage facts needed to validate and combine it, such as:

• fragment ID and source read version;
• physical row interval and row count;
• ordered target field IDs;
• the staged DataFile.

It is not a plan, task, writer, progress record, retry policy, or retention handle. It must support a version-tagged bytes round trip so workers can return it to a coordinator. The Python object should be picklable and expose equivalent to_bytes / from_bytes methods.

concat_column_slices sorts slices by physical start position and requires all inputs to:

• come from the same fragment snapshot;
• cover exactly the same ordered target fields;
• cover [0, fragment.physical_rows()) without gaps or overlaps.

Duplicate, overlapping, incomplete, or mixed-field inputs are errors; there is no last-write-wins rule. The method re-reads the actual file metadata instead of trusting only serialized declarations. It calls concat_files in physical-row order, uses the single-file reuse path when possible, and falls back to ordered decode/re-encode on Unsupported. Missing or corrupt slice files remain errors.

Neither successful nor failed concatenation deletes the input files. Their lifecycle remains caller-owned. Only complete fragment coverage can produce a publishable DataReplacementGroup.

Python API

Python should ship with the Rust API, using existing LanceFragment conventions:

LanceFragment.write_columns(
    data_obj,
    columns: list[str],
) -> DataReplacementGroup

LanceFragment.write_columns_slice(
    data_obj,
    columns: list[str],
    rows: slice,
) -> ColumnSlice

LanceFragment.concat_column_slices(
    slices: list[ColumnSlice],
) -> DataReplacementGroup

The Python binding resolves column names against the dataset schema and delegates schema, field-ID, row-count, and coverage validation to Rust. rows must have finite non-negative start and stop, with step equal to None or 1.

Example

Workers can independently write multiple columns for disjoint physical slices:

part = fragment.write_columns_slice(
    data,
    columns=["embedding", "score"],
    rows=slice(0, 100_000),
)
payload = part.to_bytes()

The coordinator collects the results, combines complete fragment coverage, and commits using the existing transaction API:

parts = [lance.fragment.ColumnSlice.from_bytes(value) for value in payloads]
replacement = fragment.concat_column_slices(parts)

operation = lance.LanceOperation.DataReplacement([replacement])
dataset = lance.LanceDataset.commit(
    dataset.uri,
    operation,
    read_version=read_version,
)

Multiple fragments can be processed independently and their DataReplacementGroups committed in one transaction.

Compaction reuse

Compaction does not call the column-slice APIs. It only passes each output group's complete data files, in fragment row order, to concat_files:

compaction adapter ---------+
                            +--> lance_file::concat_files
column-slice concatenation -+

The existing try-binary-copy mode falls back to decode/re-encode on Unsupported; a forced binary-copy mode returns a typed error. File compatibility, buffer transformation, page-priority relocation, and footer handling have one authority in lance-file.

Relationship to existing proposals

• Two‑Tier Manifest Architecture to Enable Lock‑free Parallel Column Writes #6641 proposes a two-tier manifest to change commit coordination for parallel column writers. This proposal does not change manifest or commit semantics. It supplies a lower-level mechanism that an external coordinator can use before one existing DataReplacement commit.
• Proposal: Data Overlay files for fast targeted writes #7401 adds overlay files for sparse cell-level updates. Column slices instead produce a complete positional replacement for one or more columns of a fragment and add no read-time overlay.

[Xuanwo]

Can also read: #5433

[zhangyue19921010]

Thanks @Xuanwo for putting this together — strong +1 on the overall direction, especially the mechanism-vs-coordination boundary. Keeping concat_files / ColumnSlice as pure, deterministic file-format primitives and leaving scheduling, retries, and transactions to external coordinators is exactly the right contract.

From the lance-spark perspective, this fixes a real structural limitation. Today's distributed ADD/UPDATE COLUMNS backfill is parallelized at the fragment level: a large fragment always lands on a single task, with its new column values fully buffered in executor memory — meaning stragglers and OOM risk. With write_columns_slice, a backfill may become a map-only job over row ranges with bounded per-task memory, followed by an embarrassingly parallel per-fragment concat_column_slices stage. A much better fit for expensive-compute workloads like embedding backfill.

One small question: write amplification on the concat step. The new column bytes hit storage twice — once when a worker writes its slice file, and again when concat copies the payload into the final file. For wide columns like embeddings (4KB+ per row), that second copy is real IO and object-store cost.

This shouldn't block the proposal, though. Would it be possible to treat zero-rewrite stitching — making slice files independently referenceable so concat becomes metadata-only — as an explicit phase 2 if necessary?