HDDS-16173. Add cluster utilization analysis for container balancer CLI commands - #11037
Open
sravani-revuri wants to merge 7 commits into
Open
HDDS-16173. Add cluster utilization analysis for container balancer CLI commands#11037sravani-revuri wants to merge 7 commits into
sravani-revuri wants to merge 7 commits into
Conversation
Contributor
Author
|
@ashishkumar50 , @sreejasahithi could you please review these changes? |
sreejasahithi
left a comment
Contributor
There was a problem hiding this comment.
Thanks @sravani-revuri for this PR,
left some initial comments.
sreejasahithi
left a comment
Contributor
There was a problem hiding this comment.
Thanks @sravani-revuri for updating the patch
sarvekshayr
reviewed
Aug 19, 2026
sarvekshayr
left a comment
Contributor
There was a problem hiding this comment.
Thanks @sravani-revuri for working on this.
sravani-revuri
marked this pull request as ready for review
August 24, 2026 05:18
ashishkumar50
left a comment
Contributor
There was a problem hiding this comment.
@sravani-revuri Thanks for the patch.
The purpose of this analyzer is to ensure that CLI analysis and the running balancer use the same rules, having two implementations of the classification logic can cause them to diverge. Can you make all common logic in single place and CLI/ContainerBalancerTask specific logic in their respective module.
sravani-revuri
force-pushed
the
HDDS-16173
branch
from
September 8, 2026 08:50
2e9fb85 to
2d528b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
We are adding three new CLI commands — assessment, dry-run, and recommend — that all need the same cluster view. These commands run at client-side and fetch datanode usage info via the existing getDatanodeUsageInfo RPC.
This Jira builds the shared foundation that is a common module that takes datanode usage protos,
Responsibilities:
Apply include/exclude filters on the proto list
Compute cluster average utilization
Compute upper/lower utilization limits from cluster average and threshold
Compute each eligible node’s utilization (utilization = (capacity - remaining) / capacity) and classify each as source, target, or balanced by comparing with the upper/lower limits.
Accumulate source/target counts and totalOverUtilizedBytes/totalUnderUtilizedBytes
Compute MaxUtilization, MinUtilization
Compute cluster imbalance and bytes to move
Build and return ContainerBalancerClusterSnapshot
The logic should match the way the running balancer uses in ContainerBalancerTask.initializeIteration(), so CLI output and actual balancer behaviour stay consistent.
Analyzer should return an object(ContainerBalancerClusterSnapshot) containing totalEligibleDatanodes count, clusterAvgUtilization, clusterCapacityBytes, MaxUtilization, MinUtilization, upperLimit, lowerLimit, sourceCount, targetCount, totalOverUtilizedBytes, totalUnderUtilizedBytes, bytesToMove, Imbalance (drift), top 5 sourceNodes hostnames, bottom 5 targetNodes hostnames.
We should also test this analysis.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16173
How was this patch tested?
Written Unit Tests.