HDDS-16398. Fix intermittent AlreadyClosedException in TestCommitWatcher

[yandrey321]

What changes were proposed in this pull request?

TestCommitWatcher#testReleaseBuffersOnException (and testReleaseBuffers, which shares the same setup) intermittently fail because the test starts writing to a freshly-allocated RATIS THREE pipeline before that pipeline's Ratis group has elected a leader.

@beforeeach calls cluster.waitForClusterToBeReady(), which waits only for datanode registration and SCM readiness — not for the pipeline's leader election. When the first putBlock/watch lands before a leader is settled, Ratis returns NotLeaderException; under the test's deliberately short (3s) request/watch timeouts the retries are exhausted and the client is closed, surfacing as:

NotLeaderException
-> RaftRetryFailureException
-> AlreadyClosedException: SlidingWindow$Client:client-...->RAFT is closed.
The failure occurs at future1.get() during setup, before the test reaches the datanode-shutdown scenario it actually exercises.

The fix adds one call in @beforeeach, after the cluster is ready, to wait for the RATIS THREE pipeline to reach OPEN before any writes:

cluster.waitForClusterToBeReady();
// Wait for the RATIS THREE pipeline to reach OPEN state before any writes.
// A pipeline only opens once it is healthy, which requires an elected Ratis
// leader; otherwise the first write can race leader election and fail with
// NotLeaderException -> RaftRetryFailureException -> AlreadyClosedException.
cluster.waitForPipelineTobeReady(HddsProtos.ReplicationFactor.THREE, 60000);
This closes the race deterministically: SCM transitions a RATIS pipeline to OPEN only when Pipeline.isHealthy() is true, and for a RATIS pipeline isHealthy() requires all datanodes to have reported and leaderId != null. So once the pipeline is OPEN, a leader has been elected and reported — exactly the precondition the writes depend on. The change reuses the existing MiniOzoneCluster.waitForPipelineTobeReady(...) helper (no new abstraction) and, being in @beforeeach, protects both tests in the class.

This is a test-only change; no production code is affected.

Generated-by: Claude Code (Claude Opus 4.8)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16398

How was this patch tested?

CI: https://github.com/yandrey321/ozone/actions/runs/34411454199/job/102906678997

[chihsuan]

Thanks for looking into this flaky test! @yandrey321 Could you share the CI log of the failure you investigated? The Jira stack trace is truncated.

Would it also be worth running the intermittent-test-check workflow on your branch to confirm it's stable?

Small nit: the code snippets in the description could use ``` fences so they render properly. Thanks!

[yandrey321]

Thanks for looking into this flaky test! @yandrey321 Could you share the CI log of the failure you investigated? The Jira stack trace is truncated.

Would it also be worth running the intermittent-test-check workflow on your branch to confirm it's stable?

Small nit: the code snippets in the description could use ``` fences so they render properly. Thanks!

here is the link to the failed CI: https://github.com/yandrey321/ozone/actions/runs/34395742534/job/102619847831

I saw multiple CI runs with the same signature.

[chihsuan]

here is the link to the failed CI: https://github.com/yandrey321/ozone/actions/runs/34395742534/job/102619847831

Thanks for the link! @yandrey321 I ran the intermittent-test-check on your branch and it still failed 3 times: https://github.com/chihsuan/ozone/actions/runs/34612123189

The failures have different signatures, so I suspect the root cause is different and there may be multiple issues. Could you take a look?