From 858d60d32eba1b7a88e652fba67fee966e6abcab Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Wed, 2 Jul 2025 18:43:39 -0700 Subject: [PATCH 1/5] HDDS-13373. [Docs] Add leader transfer command and service role listing command in the "OM High Availability" page Change-Id: I397296679415e7908c1a218b4a4783ccba8aef8d --- hadoop-hdds/docs/content/feature/OM-HA.md | 78 ++++++++++++++++++++++ hadoop-hdds/docs/content/feature/SCM-HA.md | 77 +++++++++++++++++++++ 2 files changed, 155 insertions(+) diff --git a/hadoop-hdds/docs/content/feature/OM-HA.md b/hadoop-hdds/docs/content/feature/OM-HA.md index 7eb83c5e5302..8419ef760817 100644 --- a/hadoop-hdds/docs/content/feature/OM-HA.md +++ b/hadoop-hdds/docs/content/feature/OM-HA.md @@ -125,6 +125,84 @@ ozone om [global options (optional)] --bootstrap --force Note that using the _force_ option during bootstrap could crash the OM process if it does not have updated configurations. +## OM Leader Transfer + +The `ozone admin om transfer` command allows you to manually transfer the leadership of the Ozone Manager (OM) Raft group to a specific OM node or to a randomly chosen follower. + +### Usage + +```bash +ozone admin om transfer -id -n +ozone admin om transfer -id -r +``` + +* `-id, --service-id`: Specifies the Ozone Manager Service ID. +* `-n, --newLeaderId, --new-leader-id`: The node ID of the OM to which leadership will be transferred (e.g., `om1`). +* `-r, --random`: Randomly chooses a follower to transfer leadership to. + +### Example + +To transfer leadership to `om2` in a cluster with service ID `cluster1`: + +```bash +ozone admin om transfer -id cluster1 -n om2 +``` + +To transfer leadership to a random follower: + +```bash +ozone admin om transfer -id cluster1 -r +``` + +## OM Service Roles Listing + +The `ozone admin om roles` command lists all Ozone Managers and their respective Raft server roles (leader, follower, or candidate). + +### Usage + +```bash +ozone admin om roles [-id ] [--json | --table] +``` + +* `-id, --service-id`: (Optional) Specifies the Ozone Manager Service ID. +* `--json`: (Optional) Formats the output as JSON. +* `--table`: (Optional) Formats the output as a table. + +### Example + +To list OM roles for `cluster1`: + +```bash +ozone admin om roles -id cluster1 +``` + +Example output: + +``` +om1 : LEADER (host1) +om2 : FOLLOWER (host2) +om3 : FOLLOWER (host3) +``` + +To list OM roles as a table: + +```bash +ozone admin om roles -id cluster1 --table +``` + +Example table output: + +``` +Ozone Manager Roles +------------------- +Host Name | Node ID | Role +------------------- +host1 | om1 | LEADER +host2 | om2 | FOLLOWER +host3 | om3 | FOLLOWER +------------------- +``` + ## Automatic Snapshot Installation for Stale Ozone Managers Sometimes an OM follower node may be offline or fall far behind the OM leader's raft log. diff --git a/hadoop-hdds/docs/content/feature/SCM-HA.md b/hadoop-hdds/docs/content/feature/SCM-HA.md index 2b6ee72b7cfc..fb5b5bf38ebf 100644 --- a/hadoop-hdds/docs/content/feature/SCM-HA.md +++ b/hadoop-hdds/docs/content/feature/SCM-HA.md @@ -90,6 +90,83 @@ ozone scm --bootstrap Note: both commands perform one-time initialization. SCM still needs to be started by running `ozone --daemon start scm`. +## SCM Leader Transfer + +The `ozone admin scm transfer` command allows you to manually transfer the leadership of the Storage Container Manager (SCM) Raft group to a specific SCM node or to a randomly chosen follower. + +### Usage + +```bash +ozone admin scm transfer -id -n +ozone admin scm transfer -id -r +``` + +* `-id, --service-id`: Specifies the SCM Service ID. +* `-n, --newLeaderId, --new-leader-id`: The node ID of the SCM to which leadership will be transferred (e.g., `scm1`). +* `-r, --random`: Randomly chooses a follower to transfer leadership to. + +### Example + +To transfer leadership to `scm2` in a cluster with service ID `cluster1`: + +```bash +ozone admin scm transfer -id cluster1 -n scm2 +``` + +To transfer leadership to a random follower: + +```bash +ozone admin scm transfer -id cluster1 -r +``` + +## SCM Service Roles Listing + +The `ozone admin scm roles` command lists all Storage Container Managers and their respective Raft server roles (leader, follower, or candidate). + +### Usage + +```bash +ozone admin scm roles [--json | --table] +``` + +* `--json`: (Optional) Formats the output as JSON. +* `--table`: (Optional) Formats the output as a table. + +### Example + +To list SCM roles: + +```bash +ozone admin scm roles +``` + +Example output: + +``` +host1:9876:LEADER:scm1:192.168.1.1 +host2:9876:FOLLOWER:scm2:192.168.1.2 +host3:9876:FOLLOWER:scm3:192.168.1.3 +``` + +To list SCM roles as a table: + +```bash +ozone admin scm roles --table +``` + +Example table output: + +``` +Storage Container Manager Roles +--------------------------------------------------- +Host Name | Ratis Port | Role | Node ID | Host Address +--------------------------------------------------- +host1 | 9876 | LEADER | scm1 | 192.168.1.1 +host2 | 9876 | FOLLOWER | scm2 | 192.168.1.2 +host3 | 9876 | FOLLOWER | scm3 | 192.168.1.3 +--------------------------------------------------- +``` + ## Auto-bootstrap In some environments (e.g. Kubernetes) we need to have a common, unified way to initialize SCM HA quorum. As a reminder, the standard initialization flow is the following: From 632e887098923c7fa1a871c564cec723eecaebe6 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Wed, 2 Jul 2025 20:53:37 -0700 Subject: [PATCH 2/5] Update hadoop-hdds/docs/content/feature/SCM-HA.md Co-authored-by: Peter Lee --- hadoop-hdds/docs/content/feature/SCM-HA.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hadoop-hdds/docs/content/feature/SCM-HA.md b/hadoop-hdds/docs/content/feature/SCM-HA.md index fb5b5bf38ebf..4bea1097bc21 100644 --- a/hadoop-hdds/docs/content/feature/SCM-HA.md +++ b/hadoop-hdds/docs/content/feature/SCM-HA.md @@ -94,6 +94,8 @@ Note: both commands perform one-time initialization. SCM still needs to be star The `ozone admin scm transfer` command allows you to manually transfer the leadership of the Storage Container Manager (SCM) Raft group to a specific SCM node or to a randomly chosen follower. +Be aware of the node's status(eg. Safemode, Operational status), ozone currently has no ability to check the target node's status before transfering the leadership. + ### Usage ```bash From 461e1d39ad92c4411e7785cb850c2d0dfd9446d2 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Sun, 6 Jul 2025 16:49:33 -0700 Subject: [PATCH 3/5] Use SCM UUID for transfer leadership command Change-Id: I91f667f66525e0df7def9f333aa566a187fd8dde --- hadoop-hdds/docs/content/feature/SCM-HA.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/feature/SCM-HA.md b/hadoop-hdds/docs/content/feature/SCM-HA.md index 4bea1097bc21..94bb12732d4d 100644 --- a/hadoop-hdds/docs/content/feature/SCM-HA.md +++ b/hadoop-hdds/docs/content/feature/SCM-HA.md @@ -104,7 +104,7 @@ ozone admin scm transfer -id -r ``` * `-id, --service-id`: Specifies the SCM Service ID. -* `-n, --newLeaderId, --new-leader-id`: The node ID of the SCM to which leadership will be transferred (e.g., `scm1`). +* `-n, --newLeaderId, --new-leader-id`: The SCM UUID (Raft peer ID) of the SCM to which leadership will be transferred (e.g., `e6877ce5-56cd-4f0b-ad60-4c8ef9000882`). * `-r, --random`: Randomly chooses a follower to transfer leadership to. ### Example From 3a93fff9037132688ac4f667bd19c0a24013ef75 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Wed, 9 Jul 2025 07:01:10 -0700 Subject: [PATCH 4/5] Update SCM UUID Change-Id: I05d091e0afc02c80701348e53230fe5c0217b57a --- hadoop-hdds/docs/content/feature/SCM-HA.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/hadoop-hdds/docs/content/feature/SCM-HA.md b/hadoop-hdds/docs/content/feature/SCM-HA.md index 94bb12732d4d..c856e392e0e5 100644 --- a/hadoop-hdds/docs/content/feature/SCM-HA.md +++ b/hadoop-hdds/docs/content/feature/SCM-HA.md @@ -39,7 +39,7 @@ This logical name is called `serviceId` and can be configured in the `ozone-site Most of the time you need to set only the values of your current cluster: - ```XML + ```xml ozone.scm.service.ids cluster1 @@ -48,7 +48,7 @@ Most of the time you need to set only the values of your current cluster: For each of the defined `serviceId` a logical configuration name should be defined for each of the servers -```XML +```xml ozone.scm.nodes.cluster1 scm1,scm2,scm3 @@ -57,7 +57,7 @@ For each of the defined `serviceId` a logical configuration name should be defin The defined prefixes can be used to define the address of each of the SCM services: -```XML +```xml ozone.scm.address.cluster1.scm1 host1 @@ -109,10 +109,10 @@ ozone admin scm transfer -id -r ### Example -To transfer leadership to `scm2` in a cluster with service ID `cluster1`: +To transfer leadership to a specific SCM in a cluster with service ID `cluster1`: ```bash -ozone admin scm transfer -id cluster1 -n scm2 +ozone admin scm transfer -id cluster1 -n e6877ce5-56cd-4f0b-ad60-4c8ef9000882 ``` To transfer leadership to a random follower: @@ -178,7 +178,7 @@ In some environments (e.g. Kubernetes) we need to have a common, unified way to This can be improved: primordial SCM can be configured by setting `ozone.scm.primordial.node.id` in the config to one of the nodes. -```XML +```xml ozone.scm.primordial.node.id scm1 @@ -215,14 +215,14 @@ Sub-CA on the SCM’s are used to issue signed certificates for OM/DN in the clu ### How to enable security: -```XML +```xml -ozone.security.enable +ozone.security.enable true -hdds.grpc.tls.enabled +hdds.grpc.tls.enabled true ``` @@ -305,4 +305,4 @@ bin/ozone debug ldb --db=/tmp/metadata/scm.db scan --column-family=containers Add additional SCM nodes and extend the cluster configuration to reflect the newly added nodes. Bootstrap the newly added SCM nodes with `scm --bootstrap` command and start the SCM service. -Note: Make sure that the `ozone.scm.primordial.node.id` property is pointed to the existing SCM before you run the `bootstrap` command on the newly added SCM nodes. +Note: Make sure that the `ozone.scm.primordial.node.id` property is pointed to the existing SCM before you run the `bootstrap` command on the newly added SCM nodes. \ No newline at end of file From d1a2fee7535bf7c5dc2a5e4fe7674847d9a62336 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Wed, 9 Jul 2025 20:32:34 -0700 Subject: [PATCH 5/5] docs(scm): Update SCM-HA.md to use UUIDs Change-Id: Iea0316735f96dd35a708e3f4606cb4400f83c8e5 --- hadoop-hdds/docs/content/feature/SCM-HA.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/hadoop-hdds/docs/content/feature/SCM-HA.md b/hadoop-hdds/docs/content/feature/SCM-HA.md index c856e392e0e5..7f9396fafe69 100644 --- a/hadoop-hdds/docs/content/feature/SCM-HA.md +++ b/hadoop-hdds/docs/content/feature/SCM-HA.md @@ -145,9 +145,9 @@ ozone admin scm roles Example output: ``` -host1:9876:LEADER:scm1:192.168.1.1 -host2:9876:FOLLOWER:scm2:192.168.1.2 -host3:9876:FOLLOWER:scm3:192.168.1.3 +host1:9876:LEADER:e6877ce5-56cd-4f0b-ad60-4c8ef9000882:192.168.1.1 +host2:9876:FOLLOWER:f2ba1c28-486a-4192-9a5f-b3752fe27d6b:192.168.1.2 +host3:9876:FOLLOWER:9c8d2b3a-e4f5-6789-0123-456789abcdef:192.168.1.3 ``` To list SCM roles as a table: @@ -160,13 +160,13 @@ Example table output: ``` Storage Container Manager Roles ---------------------------------------------------- -Host Name | Ratis Port | Role | Node ID | Host Address ---------------------------------------------------- -host1 | 9876 | LEADER | scm1 | 192.168.1.1 -host2 | 9876 | FOLLOWER | scm2 | 192.168.1.2 -host3 | 9876 | FOLLOWER | scm3 | 192.168.1.3 ---------------------------------------------------- +------------------------------------------------------------------------------------ +Host Name | Ratis Port | Role | Node ID | Host Address +------------------------------------------------------------------------------------ +host1 | 9876 | LEADER | e6877ce5-56cd-4f0b-ad60-4c8ef9000882 | 192.168.1.1 +host2 | 9876 | FOLLOWER | f2ba1c28-486a-4192-9a5f-b3752fe27d6b | 192.168.1.2 +host3 | 9876 | FOLLOWER | 9c8d2b3a-e4f5-6789-0123-456789abcdef | 192.168.1.3 +------------------------------------------------------------------------------------ ``` ## Auto-bootstrap @@ -305,4 +305,4 @@ bin/ozone debug ldb --db=/tmp/metadata/scm.db scan --column-family=containers Add additional SCM nodes and extend the cluster configuration to reflect the newly added nodes. Bootstrap the newly added SCM nodes with `scm --bootstrap` command and start the SCM service. -Note: Make sure that the `ozone.scm.primordial.node.id` property is pointed to the existing SCM before you run the `bootstrap` command on the newly added SCM nodes. \ No newline at end of file +Note: Make sure that the `ozone.scm.primordial.node.id` property is pointed to the existing SCM before you run the `bootstrap` command on the newly added SCM nodes.