HDDS-16300. Allow the Ozone Manager to dynamically reconfigure its SCM node list (ozone.scm.nodes / ozone.scm.address) without a restart

[hani-fouladgar]

What changes were proposed in this pull request?

When an SCM is added to (or removed from) an SCM-HA ring, the OM only learns the new membership from ozone.scm.nodes.<serviceId> / ozone.scm.address.<serviceId>.<nodeId> at startup. The OM's SCM block and container failover proxy providers build their node list once in their constructor, so reaching a newly added SCM previously required restarting the OM. This makes SCM scale-out/migration disruptive for the OM.

Please describe your PR in detail:
Approach
Make the OM's SCM proxy providers reloadable and drive the reload through the existing reconfiguration mechanism, so no restart is needed.

• SCMFailoverProxyProviderBase.changeConfig() (new): reloads the node list and addresses from the (already-updated) configuration.
  • loadConfigs() now builds the new node list / proxy-info map in temporaries and commits them only after the whole config parses successfully. Adding an SCM requires two properties (the node list and the new node's address); they may be applied in either order, and if the node list is updated first the reload throws and leaves the previous state intact so the operator can retry.
  • Cached proxies for removed nodes, or nodes whose address changed, are stopped (RPC.stopProxy) so the next call dials the fresh address.
  • The current-proxy pointer is kept valid: it re-syncs its index to the rebuilt list, falling back to the first node if the node it referenced was removed.
• HAUtils: added overloads of getScmBlockClient / getScmContainerClient that accept a caller-supplied proxy provider, so the OM can keep the reference needed to reload it.
• ScmClient.reloadScmNodes() (new): delegates to changeConfig() on the block and container providers; a no-op when providers are absent (e.g. mock-constructed clients).
• OzoneManager: constructs and retains the block/container proxy providers, passes them to ScmClient, and — only when an SCM service id is configured (SCM HA) — registers ozone.scm.nodes.<serviceId> and, as a prefix, ozone.scm.address.<serviceId>. as reconfigurable. The reconfScmNodes callback rejects an empty node list and otherwise triggers reloadScmNodes(). The address keys are registered as a prefix because a newly added node's key does not exist at startup and cannot be registered by name in advance.

This mirrors the existing datanode-side SCM reconfiguration (HDDS-16.x, TestDatanodeSCMNodesReconfiguration) and keeps service boundaries intact — the OM only reloads its own client proxies.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16300

How was this patch tested?

Testing

• TestSCMFailoverProxyProviderChangeConfig (unit): changeConfig() adds a node, removes a node while keeping the current-proxy pointer valid, and fails cleanly (leaving state intact) when a referenced node's address is missing.
• TestScmClient (unit): reloadScmNodes() delegates to both providers; no-op when providers are null.
• TestOmSCMNodesReconfiguration (integration, SCM-HA MiniOzoneCluster): the SCM node list and per-node address prefix are reconfigurable on a running OM; an empty node list is rejected; a valid reconfigure drives the proxy reload end-to-end.

Manual Testing

1. Manually shrink the SCM node list 3 -> 2

bash-5.1$ sed -i 's/scm1,scm2,scm3/scm1,scm2/' /etc/hadoop/ozone-site.xmlp/ozone-site.xml

2. Get the value of ozone.scm.nodes.scmservice from OM (in memory)

bash-5.1$ curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE '<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2,scm3</value>

3. Trigger reconfig the OM to get the new value from the config

bash-5.1$ ozone admin reconfig --service OM --address om1:9862 start   
OM: Started reconfiguration task on node [om1:9862].

4. Check the reconfig status

bash-5.1$ ozone admin reconfig --service OM --address om1:9862 status 2>&1 | grep -vE "TracingUtil|Sampling" 
OM: Reconfiguring status for node [om1:9862]: started at Thu Sep 03 20:53:38 UTC 2026 and finished at Thu Sep 03 20:53:38 UTC 2026.
SUCCESS: Changed property ozone.scm.nodes.scmservice
	From: "scm1,scm2,scm3"
	To: "scm1,scm2"

5. Get the value of ozone.scm.nodes.scmservice from OM (in memory)

bash-5.1$ curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE '<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2</value>

6. Add scm3 back

bash-5.1$ sed -i 's/scm1,scm2/scm1,scm2,scm3/' /etc/hadoop/ozone-site.xml

7. Get the value of ozone.scm.nodes.scmservice from OM (in memory)

bash-5.1$ curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE '<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2</value>

8. Trigger reconfig the OM to get the new value from the config

bash-5.1$ ozone admin reconfig --service OM --address om1:9862 start 
OM: Started reconfiguration task on node [om1:9862].

9. Check the reconfig status

bash-5.1$ ozone admin reconfig --service OM --address om1:9862 status 2>&1 | grep -vE "TracingUtil|Sampling" 
OM: Reconfiguring status for node [om1:9862]: started at Thu Sep 03 21:01:59 UTC 2026 and finished at Thu Sep 03 21:01:59 UTC 2026.
SUCCESS: Changed property ozone.scm.nodes.scmservice
	From: "scm1,scm2"
	To: "scm1,scm2,scm3"

10. Get the value of ozone.scm.nodes.scmservice from OM (in memory)
    curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE 'ozone.scm.(address|nodes).scmservice\s*[^<]+'
    ozone.scm.nodes.scmservicescm1,scm2,scm3
    bash-5.1$