Skip to content

Bound the consolidation master call by wait + margin - #98

Merged
ristik merged 3 commits into
mainfrom
fix/long-poll-parity-202610261300
Oct 26, 2026
Merged

ristik merged 3 commits into
mainfrom
fix/long-poll-parity-202610261300

Conversation

@ristik

@ristik ristik commented Oct 26, 2026

Copy link
Copy Markdown
Member

Parity with state-transition-sdk-rust#70

Only item 5 was a defect here; items 1-4 were checked and are already correct.

# Item Status Evidence
1 zero-interval floor ok LongPoll.pacedPause floors with MIN_POLL_GAP_MILLIS
2 unrequested echo ok JsonRpcAggregatorClient.parseInclusionProofResult rejects waitedMs unless a wait was asked; parseBatch throws on it without waitAsked; rpcFailure reports it only when asked
3 transport failure vs overload ok InclusionProofWaiter.onTransportFailure is separate from onOverload and ends as a transport TimeoutException
4 -32021 and unknown grace ok, N/A the grace is elapsedMillis() from the start of the wait, so there is nothing to reset
5 master per-call timeout fixed the master batch call had no call timeout; BatchRequestOptions.withCallTimeout is new and the consolidator sets min(remaining, wait + margin)

Test: TokenProofClientTest.theMasterCallTimeoutIsBoundedByWaitPlusMargin. ./gradlew build passes.

Parity with state-transition-sdk-rust#70 (item 5).
@MastaP

MastaP commented Oct 26, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T11:39:03.091066Z f3ef3b7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@MastaP MastaP left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at a0661ec after reading the whole production diff (BatchRequestOptions, JsonRpcAggregatorClient, ProofConsolidator). This is the Java counterpart of Rust #70 item 5 (Rust #71).

  • BatchRequestOptions.withCallTimeout is validated (positive, at most MAX_CALL_TIMEOUT) and carried by every with* copy. JsonRpcAggregatorClient passes it on to the call's RequestOptions.
  • ProofConsolidator sets it only on the master's latest request that waits: min(wait + margin, attempt.remaining()), capped at MAX_CALL_TIMEOUT, and at least 1 ms, so a spent budget is left to the attempt deadline.
  • Items 1–4 match my notes from the Rust #45 review: Java floors the pause at MIN_POLL_GAP_MILLIS, rejects an unrequested echo, keeps transport failures apart from overload, and counts the unknown grace in time.

Tests. ./gradlew build passes (668 tests, 0 failures). Removing the withCallTimeout(callTimeout(wait)) call fails TokenProofClientTest (1 failure).

…ound with the repeat

The consolidator maps a call that outlived its bound to TIMEOUT, and the
repeat without waitMs uses only what is left of the original call timeout.

@MastaP MastaP left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at c97e1f1. This replaces my approval of a0661ec.

The repeat-without-waitMs change is fine. The repeat now uses only what is left of the call bound (bound − elapsed, failing at once when none is left). withoutWait has a single caller, so dropping the callTimeout copy there changes nothing else.

The new timedOut mapping makes a peer's transport timeout a non-retryable CANCELLED. requestFailure now maps any InterruptedIOException in the cause chain to ConsolidationException.Kind.TIMEOUT. That is not limited to the master's wait-plus-margin bound: it also catches a peer's ordinary transport timeout in wave 2. There, TIMEOUT meets two existing rules:

  1. In fetchAll, the first failing peer sets aborted and cancels inner, so the peers still in flight complete with CancellationException, which requestFailure maps to CANCELLED (retryable=false).
  2. finishFetch treats both CANCELLED and TIMEOUT as consequences ("cancellation") and, when no other error exists, reports the first error by index.

So a peer timeout at a higher index than a peer that was still in flight is reported as that peer's CANCELLED. Before this commit the timeout was TRANSPORT, which finishFetch picks as the origin.

Repro. The shards-8 corpus case (master plus 3 peers), with the first peer call left in flight and the second failing with a SocketTimeoutException cause:

Head Reported failure
a0661ec TRANSPORT, retryable=true, message timeout
c97e1f1 CANCELLED, retryable=false, message The request was cancelled.

A caller that stops on non-retryable failures now gives up on a transient network timeout.

Possible fixes:

  • map only the master's own bounded call to TIMEOUT (the only call that carries a callTimeout, e.g. check master && wait > 0); or
  • keep a per-call timeout out of finishFetch's "cancellation" class, which is meant for the attempt deadline and the follow-on cancellations.

Either way, a test like the one below should pass (it prints the reported kind; assert isRetryable() and a non-CANCELLED kind).

Repro test (org.unicitylabs.sdk.consolidation, uses CorpusSupport)
package org.unicitylabs.sdk.consolidation;

import com.fasterxml.jackson.databind.JsonNode;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import org.junit.jupiter.api.Test;
import org.unicitylabs.sdk.api.BatchAggregatorClient;
import org.unicitylabs.sdk.api.BatchProofException;
import org.unicitylabs.sdk.api.BatchRequestOptions;
import org.unicitylabs.sdk.api.InclusionProofBatch;
import org.unicitylabs.sdk.api.StateId;
import org.unicitylabs.sdk.api.bft.RootTrustBase;
import org.unicitylabs.sdk.api.bft.ShardId;
import org.unicitylabs.sdk.api.bft.ShardConfigurationPin;
import org.unicitylabs.sdk.util.HexConverter;
import org.unicitylabs.sdk.api.bft.verification.OperationUnicityCertificateVerifier;
import org.unicitylabs.sdk.api.bft.PartitionViewPolicy;
import org.unicitylabs.sdk.api.bft.verification.UnicityCertificateVerifier;
import org.unicitylabs.sdk.api.bft.verification.VerificationPolicy;

// One peer's transport times out while another peer is in flight.
class ZzReviewPeerTimeoutTest {

  @Test
  void aPeerTransportTimeoutIsReportedAsTheRetryableCause() throws Exception {
    CorpusSupport.Case c = new CorpusSupport.Case("shards-8");
    JsonNode context = c.json("context.json");
    JsonNode all = c.json("responses.json");
    String masterBits = context.get("master").asText();
    Map<String, JsonNode> responses = new HashMap<>();
    responses.put(masterBits, all.get("master"));
    for (JsonNode peer : all.get("peers")) {
      responses.put(peer.get("shard").asText(), peer.get("response"));
    }
    List<ShardId> scheme = new ArrayList<>();
    for (JsonNode bits : context.get("shards")) {
      scheme.add(CorpusSupport.shard(bits.asText()));
    }
    List<CorpusSupport.Call> calls = CorpusSupport.newCalls();
    List<ShardEndpoint> endpoints = new ArrayList<>();
    int[] peerSeen = {0};
    for (JsonNode bits : context.get("participants")) {
      String b = bits.asText();
      BatchAggregatorClient real = new CorpusSupport.FakeShardClient(b, responses.get(b), calls);
      BatchAggregatorClient client = b.equals(masterBits) ? real
          : (BatchAggregatorClient) java.lang.reflect.Proxy.newProxyInstance(
              BatchAggregatorClient.class.getClassLoader(), new Class<?>[] {BatchAggregatorClient.class},
              (proxy, method, args) -> {
                if (!method.getName().equals("getInclusionProofs")) {
                  return method.invoke(real, args);
                }
                int n = peerSeen[0]++;
                CompletableFuture<InclusionProofBatch> f = new CompletableFuture<>();
                if (n == 0) {
                  return f; // in flight until the wave is cancelled
                }
                f.completeExceptionally(new BatchProofException(BatchProofException.Kind.TRANSPORT,
                    "timeout", null, null, null, true, 1, null,
                    new java.net.SocketTimeoutException("read timed out")));
                return f;
              });
      endpoints.add(new ShardEndpoint(CorpusSupport.shard(b), client));
    }
    List<ShardConfigurationPin> pins = new ArrayList<>();
    for (JsonNode config : context.get("shardConfigs")) {
      pins.add(new ShardConfigurationPin(CorpusSupport.shard(config.get("shard").asText()),
          HexConverter.decode(config.get("hash").asText())));
    }
    RootTrustBase trustBase = CorpusSupport.trustBase(c.bytes("trust-base.cbor"));
    OperationUnicityCertificateVerifier verifier = (OperationUnicityCertificateVerifier)
        new UnicityCertificateVerifier().beginOperation(VerificationPolicy.LEGACY_QUORUM,
            PartitionViewPolicy.none());
    ProofConsolidator consolidator = new ProofConsolidator(new ShardRouting(endpoints, scheme, pins),
        trustBase, verifier);
    ConsolidationException failure;
    try {
      consolidator.consolidate(CorpusSupport.evidenceOf(c.json("requests.json")),
          ConsolidationOptions.defaults().withMaster(CorpusSupport.shard(masterBits))
              .withTimeout(Duration.ofSeconds(20)).withExecutor(Runnable::run)).join();
      throw new AssertionError("expected a failure");
    } catch (CompletionException e) {
      failure = (ConsolidationException) e.getCause();
    }
    System.out.println("ZZPROBE peers called=" + peerSeen[0] + " failure kind=" + failure.getKind()
        + " retryable=" + failure.isRetryable() + " message=" + failure.getMessage());
  }
}

The rest of my approval of a0661ec still holds. I will re-approve once the classification is fixed.

…ut stays TRANSPORT

A peer's SocketTimeoutException made fetchAll's cancellation class report a
non-retryable CANCELLED from an in-flight peer.
@MastaP

MastaP commented Oct 26, 2026

Copy link
Copy Markdown
Member

@codex review

@ristik

ristik commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

Fixed in f3ef3b7. requestFailure now maps an InterruptedIOException to TIMEOUT only for the master's own bounded call, the only call that carries a callTimeout (fetch passes a bounded flag, true only when wait > 0). A peer's ordinary transport timeout stays a retryable TRANSPORT, so fetchAll/finishFetch pick it as the origin again instead of reporting an in-flight peer's non-retryable CANCELLED. Your repro is now PeerTransportTimeoutTest (shards-8, first peer in flight, second a SocketTimeoutException) asserting TRANSPORT and isRetryable(); the master-bound TIMEOUT test is unchanged. CI passes.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: f3ef3b71d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@MastaP MastaP left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at f3ef3b7; this replaces my change request at c97e1f1.

The fix. requestFailure maps an InterruptedIOException to TIMEOUT only when the call carried its own call bound (bounded = requestOptions.getCallTimeout() != null). That is only the master's waiting latest request. Peer exact-root calls never carry a bound, so their transport timeouts stay a retryable TRANSPORT, which finishFetch reports as the origin. Both requestFailure call sites pass the flag: the call's own failure passes bounded, and the setup failure in fetch passes false.

Re-run of my repro (shards-8, the first peer call left in flight, the second failing with SocketTimeoutException): the failure is now TRANSPORT, retryable=true, message timeout, the same as a0661ec. It was CANCELLED, retryable=false at c97e1f1.

Mutation check. Dropping the bounded && condition fails PeerTransportTimeoutTest. The master-bound TIMEOUT case keeps passing in TokenProofClientTest.

Tests. ./gradlew build passes: 671 tests, 0 failures.

@ristik
ristik merged commit 78fb7ed into main Oct 26, 2026
1 check passed
@ristik
ristik deleted the fix/long-poll-parity-202610261300 branch October 26, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants