Skip to content

HIVE-29829: Iceberg: Stream partition column statistics on write, prune them on read - #6707

Merged
deniskuzZ merged 1 commit into
apache:masterfrom
deniskuzZ:col_stats_refactor
Sep 8, 2026
Merged

HIVE-29829: Iceberg: Stream partition column statistics on write, prune them on read#6707
deniskuzZ merged 1 commit into
apache:masterfrom
deniskuzZ:col_stats_refactor

Conversation

@deniskuzZ

@deniskuzZ deniskuzZ commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Per-partition column statistics for Iceberg tables have been reworked end to end, covering how they are stored, written, and read during scans, as well as how readers determine whether the statistics still accurately describe the underlying data.

Why are the changes needed?

Performance driven optimization and design refactor

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • TestIcebergColStatsFormat
  • TestIcebergColStatsWritePolicy
  • TestHiveIcebergStatistics

qtests:

  • iceberg_colstats_analyze_evolution.q
  • iceberg_colstats_folded.q
  • iceberg_colstats_granularity.q
  • iceberg_colstats_staleness.q
  • iceberg_colstats_write_paths.q,
  • iceberg_compaction_colstats_compute.q

depends on #6711

@deniskuzZ
deniskuzZ marked this pull request as draft August 18, 2026 17:14
@deniskuzZ
deniskuzZ force-pushed the col_stats_refactor branch 2 times, most recently from f73473a to 04270e5 Compare August 19, 2026 20:52
@deniskuzZ deniskuzZ changed the title WIP: Iceberg col stats refactor HIVE-29829: Iceberg: Partition-level column statistics refactor Aug 21, 2026
@deniskuzZ deniskuzZ mentioned this pull request Aug 23, 2026
Open
@deniskuzZ deniskuzZ mentioned this pull request Aug 29, 2026
Open
@Aggarwal-Raghav

Aggarwal-Raghav commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

This PR is a bit overwhelming, so I’m mostly reviewing to learn and understand the new flow!

difin
difin reviewed Sep 1, 2026
View reviewed changes
Comment thread common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
difin
difin reviewed Sep 1, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java Outdated
difin
difin reviewed Sep 2, 2026
View reviewed changes
Comment thread ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java Outdated
difin
difin approved these changes Sep 2, 2026
View reviewed changes

@difin difin left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

I left a few comments.

@difin

difin commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR description lists TestIcebergManifestColStats, but I don't see this class in the diff.

@difin

difin commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

There are 43 Sonar issues, some looks actionable, like replacing String concatenation blocks with String.format(), adding try-with-resources.

@deniskuzZ

deniskuzZ commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

PR description lists TestIcebergManifestColStats, but I don't see this class in the diff.

removed, it was extracted into separate PR

@deniskuzZ
HIVE-29829: Iceberg: Partition-level column statistics
d9ab5cc 
A partitioned table's column statistics are stored one Puffin blob per
partition - every column's entry framed behind its field id and length,
bit vector included - together with table-level aggregates, one entry
per column, folded from the partitions. A gather streams through the
writer a batch at a time, and a merge carries forward, byte for byte,
the blobs of partitions no write has changed. The storage-handler SPI
takes the gather as an iterator where it took a list.

The table metadata registers only what a read can answer without
opening the file: the table-level aggregates and one partition entry
carrying the combined column list and a numPartitions property. Blob
offsets live in the file's own footer; a pruned read filters it by
partition name, coalesces the wanted blobs into runs, and fetches them
in one vectored call - S3A merges the ranges and issues them four at a
time. A whole-table read is served from the aggregates alone, and an
ask of another partition count is turned away from the metadata with
no I/O. A read decodes the columns it was asked for and steps over
the rest.

Whether stored statistics still describe the data is decided from the
snapshot the file was written for: ANALYZE stores what it read, an
overwrite what it replaced, a major compaction of one current-spec
partition refreshes it; a plain INSERT maintains none, and its
partitions read as stale until recomputed. A branch's statistics stay
on the branch.
@sonarqubecloud

sonarqubecloud Bot commented Sep 6, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
40 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@difin difin difin approved these changes
Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
+1 more reviewer
@Aggarwal-Raghav Aggarwal-Raghav Aggarwal-Raghav left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

tests passed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@deniskuzZ @Aggarwal-Raghav @difin @asf-ci-hive