Repository navigation
Bound the consolidation master call by wait + margin - #98
Conversation
Parity with state-transition-sdk-rust#70 (item 5).
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. You're on a roll. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
MastaP
left a comment
There was a problem hiding this comment.
Approving at a0661ec after reading the whole production diff (BatchRequestOptions, JsonRpcAggregatorClient, ProofConsolidator). This is the Java counterpart of Rust #70 item 5 (Rust #71).
BatchRequestOptions.withCallTimeoutis validated (positive, at mostMAX_CALL_TIMEOUT) and carried by everywith*copy.JsonRpcAggregatorClientpasses it on to the call'sRequestOptions.ProofConsolidatorsets it only on the master's latest request that waits:min(wait + margin, attempt.remaining()), capped atMAX_CALL_TIMEOUT, and at least 1 ms, so a spent budget is left to the attempt deadline.- Items 1–4 match my notes from the Rust #45 review: Java floors the pause at
MIN_POLL_GAP_MILLIS, rejects an unrequested echo, keeps transport failures apart from overload, and counts the unknown grace in time.
Tests. ./gradlew build passes (668 tests, 0 failures). Removing the withCallTimeout(callTimeout(wait)) call fails TokenProofClientTest (1 failure).
…ound with the repeat The consolidator maps a call that outlived its bound to TIMEOUT, and the repeat without waitMs uses only what is left of the original call timeout.
MastaP
left a comment
There was a problem hiding this comment.
Requesting changes at c97e1f1. This replaces my approval of a0661ec.
The repeat-without-waitMs change is fine. The repeat now uses only what is left of the call bound (bound − elapsed, failing at once when none is left). withoutWait has a single caller, so dropping the callTimeout copy there changes nothing else.
The new timedOut mapping makes a peer's transport timeout a non-retryable CANCELLED. requestFailure now maps any InterruptedIOException in the cause chain to ConsolidationException.Kind.TIMEOUT. That is not limited to the master's wait-plus-margin bound: it also catches a peer's ordinary transport timeout in wave 2. There, TIMEOUT meets two existing rules:
- In
fetchAll, the first failing peer setsabortedand cancelsinner, so the peers still in flight complete withCancellationException, whichrequestFailuremaps toCANCELLED(retryable=false). finishFetchtreats bothCANCELLEDandTIMEOUTas consequences ("cancellation") and, when no other error exists, reports the first error by index.
So a peer timeout at a higher index than a peer that was still in flight is reported as that peer's CANCELLED. Before this commit the timeout was TRANSPORT, which finishFetch picks as the origin.
Repro. The shards-8 corpus case (master plus 3 peers), with the first peer call left in flight and the second failing with a SocketTimeoutException cause:
| Head | Reported failure |
|---|---|
| a0661ec | TRANSPORT, retryable=true, message timeout |
| c97e1f1 | CANCELLED, retryable=false, message The request was cancelled. |
A caller that stops on non-retryable failures now gives up on a transient network timeout.
Possible fixes:
- map only the master's own bounded call to
TIMEOUT(the only call that carries acallTimeout, e.g. checkmaster && wait > 0); or - keep a per-call timeout out of
finishFetch's "cancellation" class, which is meant for the attempt deadline and the follow-on cancellations.
Either way, a test like the one below should pass (it prints the reported kind; assert isRetryable() and a non-CANCELLED kind).
Repro test (org.unicitylabs.sdk.consolidation, uses CorpusSupport)
package org.unicitylabs.sdk.consolidation;
import com.fasterxml.jackson.databind.JsonNode;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import org.junit.jupiter.api.Test;
import org.unicitylabs.sdk.api.BatchAggregatorClient;
import org.unicitylabs.sdk.api.BatchProofException;
import org.unicitylabs.sdk.api.BatchRequestOptions;
import org.unicitylabs.sdk.api.InclusionProofBatch;
import org.unicitylabs.sdk.api.StateId;
import org.unicitylabs.sdk.api.bft.RootTrustBase;
import org.unicitylabs.sdk.api.bft.ShardId;
import org.unicitylabs.sdk.api.bft.ShardConfigurationPin;
import org.unicitylabs.sdk.util.HexConverter;
import org.unicitylabs.sdk.api.bft.verification.OperationUnicityCertificateVerifier;
import org.unicitylabs.sdk.api.bft.PartitionViewPolicy;
import org.unicitylabs.sdk.api.bft.verification.UnicityCertificateVerifier;
import org.unicitylabs.sdk.api.bft.verification.VerificationPolicy;
// One peer's transport times out while another peer is in flight.
class ZzReviewPeerTimeoutTest {
@Test
void aPeerTransportTimeoutIsReportedAsTheRetryableCause() throws Exception {
CorpusSupport.Case c = new CorpusSupport.Case("shards-8");
JsonNode context = c.json("context.json");
JsonNode all = c.json("responses.json");
String masterBits = context.get("master").asText();
Map<String, JsonNode> responses = new HashMap<>();
responses.put(masterBits, all.get("master"));
for (JsonNode peer : all.get("peers")) {
responses.put(peer.get("shard").asText(), peer.get("response"));
}
List<ShardId> scheme = new ArrayList<>();
for (JsonNode bits : context.get("shards")) {
scheme.add(CorpusSupport.shard(bits.asText()));
}
List<CorpusSupport.Call> calls = CorpusSupport.newCalls();
List<ShardEndpoint> endpoints = new ArrayList<>();
int[] peerSeen = {0};
for (JsonNode bits : context.get("participants")) {
String b = bits.asText();
BatchAggregatorClient real = new CorpusSupport.FakeShardClient(b, responses.get(b), calls);
BatchAggregatorClient client = b.equals(masterBits) ? real
: (BatchAggregatorClient) java.lang.reflect.Proxy.newProxyInstance(
BatchAggregatorClient.class.getClassLoader(), new Class<?>[] {BatchAggregatorClient.class},
(proxy, method, args) -> {
if (!method.getName().equals("getInclusionProofs")) {
return method.invoke(real, args);
}
int n = peerSeen[0]++;
CompletableFuture<InclusionProofBatch> f = new CompletableFuture<>();
if (n == 0) {
return f; // in flight until the wave is cancelled
}
f.completeExceptionally(new BatchProofException(BatchProofException.Kind.TRANSPORT,
"timeout", null, null, null, true, 1, null,
new java.net.SocketTimeoutException("read timed out")));
return f;
});
endpoints.add(new ShardEndpoint(CorpusSupport.shard(b), client));
}
List<ShardConfigurationPin> pins = new ArrayList<>();
for (JsonNode config : context.get("shardConfigs")) {
pins.add(new ShardConfigurationPin(CorpusSupport.shard(config.get("shard").asText()),
HexConverter.decode(config.get("hash").asText())));
}
RootTrustBase trustBase = CorpusSupport.trustBase(c.bytes("trust-base.cbor"));
OperationUnicityCertificateVerifier verifier = (OperationUnicityCertificateVerifier)
new UnicityCertificateVerifier().beginOperation(VerificationPolicy.LEGACY_QUORUM,
PartitionViewPolicy.none());
ProofConsolidator consolidator = new ProofConsolidator(new ShardRouting(endpoints, scheme, pins),
trustBase, verifier);
ConsolidationException failure;
try {
consolidator.consolidate(CorpusSupport.evidenceOf(c.json("requests.json")),
ConsolidationOptions.defaults().withMaster(CorpusSupport.shard(masterBits))
.withTimeout(Duration.ofSeconds(20)).withExecutor(Runnable::run)).join();
throw new AssertionError("expected a failure");
} catch (CompletionException e) {
failure = (ConsolidationException) e.getCause();
}
System.out.println("ZZPROBE peers called=" + peerSeen[0] + " failure kind=" + failure.getKind()
+ " retryable=" + failure.isRetryable() + " message=" + failure.getMessage());
}
}The rest of my approval of a0661ec still holds. I will re-approve once the classification is fixed.
…ut stays TRANSPORT A peer's SocketTimeoutException made fetchAll's cancellation class report a non-retryable CANCELLED from an in-flight peer.
|
@codex review |
|
Fixed in f3ef3b7. |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
MastaP
left a comment
There was a problem hiding this comment.
Approving at f3ef3b7; this replaces my change request at c97e1f1.
The fix. requestFailure maps an InterruptedIOException to TIMEOUT only when the call carried its own call bound (bounded = requestOptions.getCallTimeout() != null). That is only the master's waiting latest request. Peer exact-root calls never carry a bound, so their transport timeouts stay a retryable TRANSPORT, which finishFetch reports as the origin. Both requestFailure call sites pass the flag: the call's own failure passes bounded, and the setup failure in fetch passes false.
Re-run of my repro (shards-8, the first peer call left in flight, the second failing with SocketTimeoutException): the failure is now TRANSPORT, retryable=true, message timeout, the same as a0661ec. It was CANCELLED, retryable=false at c97e1f1.
Mutation check. Dropping the bounded && condition fails PeerTransportTimeoutTest. The master-bound TIMEOUT case keeps passing in TokenProofClientTest.
Tests. ./gradlew build passes: 671 tests, 0 failures.
Parity with state-transition-sdk-rust#70
Only item 5 was a defect here; items 1-4 were checked and are already correct.
LongPoll.pacedPausefloors withMIN_POLL_GAP_MILLISJsonRpcAggregatorClient.parseInclusionProofResultrejectswaitedMsunless a wait was asked;parseBatchthrows on it withoutwaitAsked;rpcFailurereports it only when askedInclusionProofWaiter.onTransportFailureis separate fromonOverloadand ends as a transportTimeoutException-32021and unknown graceelapsedMillis()from the start of the wait, so there is nothing to resetBatchRequestOptions.withCallTimeoutis new and the consolidator setsmin(remaining, wait + margin)Test:
TokenProofClientTest.theMasterCallTimeoutIsBoundedByWaitPlusMargin../gradlew buildpasses.