Skip to content

HDDS-16173. Add cluster utilization analysis for container balancer CLI commands - #11037

Open
sravani-revuri wants to merge 7 commits into
apache:masterfrom
sravani-revuri:HDDS-16173
Open

HDDS-16173. Add cluster utilization analysis for container balancer CLI commands#11037
sravani-revuri wants to merge 7 commits into
apache:masterfrom
sravani-revuri:HDDS-16173

Conversation

@sravani-revuri

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

We are adding three new CLI commands — assessment, dry-run, and recommend — that all need the same cluster view. These commands run at client-side and fetch datanode usage info via the existing getDatanodeUsageInfo RPC.

This Jira builds the shared foundation that is a common module that takes datanode usage protos,

Responsibilities:

Apply include/exclude filters on the proto list
Compute cluster average utilization
Compute upper/lower utilization limits from cluster average and threshold
Compute each eligible node’s utilization (utilization = (capacity - remaining) / capacity) and classify each as source, target, or balanced by comparing with the upper/lower limits.
Accumulate source/target counts and totalOverUtilizedBytes/totalUnderUtilizedBytes
Compute MaxUtilization, MinUtilization
Compute cluster imbalance and bytes to move
Build and return ContainerBalancerClusterSnapshot
The logic should match the way the running balancer uses in ContainerBalancerTask.initializeIteration(), so CLI output and actual balancer behaviour stay consistent.

Analyzer should return an object(ContainerBalancerClusterSnapshot) containing totalEligibleDatanodes count, clusterAvgUtilization, clusterCapacityBytes, MaxUtilization, MinUtilization, upperLimit, lowerLimit, sourceCount, targetCount, totalOverUtilizedBytes, totalUnderUtilizedBytes, bytesToMove, Imbalance (drift), top 5 sourceNodes hostnames, bottom 5 targetNodes hostnames.

We should also test this analysis.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16173

How was this patch tested?

Written Unit Tests.

@sravani-revuri

Copy link
Copy Markdown
Contributor Author

@ashishkumar50 , @sreejasahithi could you please review these changes?

@sreejasahithi sreejasahithi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sravani-revuri for this PR,
left some initial comments.

@sreejasahithi sreejasahithi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sravani-revuri for updating the patch

@sarvekshayr sarvekshayr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sravani-revuri for working on this.

@sravani-revuri
sravani-revuri marked this pull request as ready for review August 24, 2026 05:18

@ashishkumar50 ashishkumar50 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sravani-revuri Thanks for the patch.
The purpose of this analyzer is to ensure that CLI analysis and the running balancer use the same rules, having two implementations of the classification logic can cause them to diverge. Can you make all common logic in single place and CLI/ContainerBalancerTask specific logic in their respective module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants