feat: make StatisticsRegistry the default operator-statistics path by asolimando · Pull Request #23651 · apache/datafusion

asolimando · 2026-07-16T19:34:52Z

Which issue does this PR close?

Rationale for this change

#23051 unified statistics into one StatisticsContext walk. This folds the
previously opt-in StatisticsRegistry (#21483) into that walk as the default, so
providers are consulted inline and use_statistics_registry is no longer needed.
Consumed today by join ordering and EXPLAIN; more CBO rules are follow-up.

What changes are included in this PR?

Consult the provider chain in the StatisticsContext walk (first Computed
wins, else statistics_from_inputs); used by join_selection.
Deprecate use_statistics_registry (ignored, warns) and
DefaultStatisticsProvider; add partition-aware
StatisticsProvider::compute_statistics_with_args.
Propagate provider Extensions through the walk via a side cache (no cost
without providers); add StatisticsContext::compute_extended, deprecate
StatisticsRegistry::compute/compute_base.
EXPLAIN and EXPLAIN ANALYZE show_statistics now reflect the session registry.
Add the join_reorder example.

Are these changes tested?

Unit tests, statistics_registry.slt, the example, the full extended suite, and
compute_statistics at benchmark parity.

Are there any user-facing changes?

use_statistics_registry deprecated and ignored (default behavior unchanged).
StatisticsRegistry::compute/compute_base deprecated (55.0.0) for
StatisticsContext; compute_extended added; DefaultStatisticsProvider
deprecated. Nothing removed. See the 55.0.0 upgrade guide.

Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

github-actions · 2026-07-16T19:45:37Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 103.326s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [ 103.207s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.617s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 208.953s] datafusion
    Building datafusion-common v54.0.0 (current)
       Built [  33.356s] (current)
     Parsing datafusion-common v54.0.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-common v54.0.0 (baseline)
       Built [  33.047s] (baseline)
     Parsing datafusion-common v54.0.0 (baseline)
      Parsed [   0.061s] (baseline)
    Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.769s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  68.597s] datafusion-common
    Building datafusion-physical-optimizer v54.0.0 (current)
       Built [  38.498s] (current)
     Parsing datafusion-physical-optimizer v54.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-physical-optimizer v54.0.0 (baseline)
       Built [  38.552s] (baseline)
     Parsing datafusion-physical-optimizer v54.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-physical-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.137s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  78.395s] datafusion-physical-optimizer
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  36.563s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.141s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  36.364s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.142s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.675s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure type_marked_deprecated: #[deprecated] added on type ---

Description:
A type is now #[deprecated]. Downstream crates will get a compiler warning when using this type.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/type_marked_deprecated.ron

Failed in:
  Struct DefaultStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:298

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_physical_plan::operator_statistics::StatisticsRegistry::compute in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:402
  method datafusion_physical_plan::operator_statistics::StatisticsRegistry::compute_base in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:416

     Summary semver requires new minor version: 0 major and 2 minor checks failed
    Finished [  75.181s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 181.499s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 183.064s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.090s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 367.243s] datafusion-sqllogictest

codecov-commenter · 2026-07-16T20:04:54Z

Codecov Report

❌ Patch coverage is 80.27397% with 72 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@7f6cc60). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...usion/physical-plan/src/operator_statistics/mod.rs	64.63%	19 Missing and 39 partials ⚠️
datafusion/physical-plan/src/statistics.rs	92.41%	8 Missing and 3 partials ⚠️
datafusion/physical-plan/src/analyze.rs	83.33%	2 Missing ⚠️
datafusion/physical-plan/src/display.rs	93.33%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23651   +/-   ##
=======================================
  Coverage        ?   80.66%           
=======================================
  Files           ?     1087           
  Lines           ?   367595           
  Branches        ?   367595           
=======================================
  Hits            ?   296535           
  Misses          ?    53406           
  Partials        ?    17654

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

StatisticsContext (the single statistics walk, apache#23051) now consults the StatisticsRegistry (apache#21483) at each node before falling back to the operator's statistics_from_inputs, using the provider result's base Statistics. The registry is always-on: datafusion.optimizer.use_statistics_registry is deprecated and ignored (register providers on the SessionState instead). The now-redundant DefaultStatisticsProvider is deprecated, since the walk falls back natively.

The walk keeps core `Statistics` as its currency; a per-walk side cache carries provider `Extensions` separately, so a walk with no providers has no per-node overhead. A provider's `Computed` extensions are cached and reach parent-node providers as their `child_stats`; the built-in `statistics_from_inputs` fallback yields none, so extensions survive only across an unbroken chain of provider-handled nodes. Adds `StatisticsContext::compute_extended` (statistics + extensions) alongside `compute` (core statistics only). `StatisticsRegistry::compute` preserves extensions again; it and `compute_base` are deprecated in favor of `StatisticsContext`.

EXPLAIN ... show_statistics computed statistics with an empty registry, so registered providers were not reflected in the statistics column. Thread the session's StatisticsRegistry into the displayable plan; the column now reflects provider-supplied estimates when providers are registered, and is unchanged when none are.

A runnable example showing a StatisticsProvider that supplies and refines column statistics (post-filter distinct count via the survival formula) to flip a join's build side, with a before/after EXPLAIN.

asolimando · 2026-07-17T18:47:49Z

 STORED AS PARQUET
 LOCATION 'test_files/scratch/statistics_registry/dim_small.parquet';

-# -- Without registry --------------------------------------------------------


Having removed use_statistics_registry config knob means the registry can't be turned off within an SLT. The no-provider path is covered by existing tests indirectly, and the with/without contrast now lives in the join_reorder example

asolimando · 2026-07-17T18:53:11Z

-    }
+struct StatsCache {
+    statistics: HashMap<CacheKey, Arc<Statistics>>,
+    extensions: HashMap<CacheKey, Extensions>,


I kept https://github.com/apache/datafusion/pull/23651/changes#diff-149914d58cc354ddd4d84e283de4f8f07f3295b39137d850b3344448d4f6dc7a separated on purpose in case reviewers prefer to drop it from this PR.

The reason to have two caches is that caching on ExtendedStatistics at every node regressed the compute_statistics benchmark ~6% to 28% (median ~13%) on the no-provider path (a wrapper allocation per node).

The committed version keeps core Statistics in the walk and holds Extensions in a side map, populated only when a provider returns some. Benchmark parity when no providers are registered, extensions still reach parent-node providers.

I think this commit is worth keeping for feature parity w.r.t. StatisticsRegistry as existing today.

asolimando · 2026-07-17T18:55:43Z

            metric_types: self.metric_types,
            metric_categories: self.metric_categories,
            format: self.format,
+            statistics_registry: self.statistics_registry,


The registry here is needed to keep EXPLAIN ANALYZE and EXPLAIN in sync when statistics providers are in use.

asolimando · 2026-07-17T19:30:20Z

As discussed in #21122 (comment), making StatisticsRegistry part of the default path was suggested by @alamb, who also suggested to showcase it via an example, which I have added, showing a better plan (join build/probe flip) over built-in defaults, via a statistics provider.

cc:

@kosiew as the original reviewer of feat: Add pluggable StatisticsRegistry for operator-level statistics propagation #21483
@xudong963 who gently reviewed many related contributions (notably refactor(physical-plan): externalize statistics traversal into StatisticsContext #23051)
@alamb as the original idea behind this PR is yours, it would be great if you could confirm this matches what you meant

Note: the diff reads large (+1055 / -332), but ~620 of the added lines are examples and tests, ~38 are docs. The core change is ~400 lines, which goes down to a more reasonable ~300 if you exclude mechanical fixes.

github-actions Bot added documentation Improvements or additions to documentation optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) common Related to common crate physical-plan Changes to the physical-plan crate labels Jul 16, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jul 16, 2026

asolimando force-pushed the asolimando/statistics-registry-default branch from 1f2d39d to 38d5bb1 Compare July 16, 2026 20:19

asolimando added 4 commits July 17, 2026 09:14

feat(examples): add StatisticsRegistry example

ded4292

A runnable example showing a StatisticsProvider that supplies and refines column statistics (post-filter distinct count via the survival formula) to flip a join's build side, with a before/after EXPLAIN.

asolimando force-pushed the asolimando/statistics-registry-default branch from 38d5bb1 to ded4292 Compare July 17, 2026 18:38

github-actions Bot added the core Core DataFusion crate label Jul 17, 2026

asolimando commented Jul 17, 2026

View reviewed changes

asolimando marked this pull request as ready for review July 17, 2026 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: make StatisticsRegistry the default operator-statistics path#23651

feat: make StatisticsRegistry the default operator-statistics path#23651
asolimando wants to merge 4 commits into
apache:mainfrom
asolimando:asolimando/statistics-registry-default

asolimando commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 16, 2026 •

edited

Loading

Uh oh!

asolimando Jul 17, 2026

Uh oh!

asolimando Jul 17, 2026

Uh oh!

asolimando Jul 17, 2026

Uh oh!

asolimando commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

asolimando commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

asolimando Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

asolimando Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

asolimando Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

asolimando commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

asolimando commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

codecov-commenter commented Jul 16, 2026 •

edited

Loading