Skip to content

UTB parity 5/8: GetTrustBase page decoder, HTTP source, fetch loop and verify-update-retry - #4

Merged
ristik merged 5 commits into
mainfrom
dev2/java-5-fetch
Nov 8, 2026
Merged

ristik merged 5 commits into
mainfrom
dev2/java-5-fetch

Conversation

@ristik

@ristik ristik commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Part of the Java UTB parity stack (owner §51, plan briefs/java-utb-parity-plan.md PR 5). Stacked on #3 (dev3/java-3-history); retarget to main after #3's chain merges. Mirrors JS TrustBaseRange.ts, TrustBaseUpdater.ts, ITrustBaseSource.ts, verifyWithUpdate.ts, and Rust trust_base_{range,update}.rs, client/http.rs (trust base GET), tests/{gettrustbase_vectors,trust_base_http_source,trust_base_update}.rs. Rust is the semantic reference where JS and Rust differ (see below). Nothing here touches unicitynetwork/*.

What it adds (package org.unicitylabs.sdk.api.bft.trustbase unless noted)

  • TrustBaseStatus (the 8 wire statuses, 8+ refused), TrustBaseRangeLimits (default 1 MiB / 64 entries), TrustBaseRequest (+ Snapshot), TrustBaseRange.decode(bytes, limits, expectedNetwork): strict decode of the tag-39015 page (byte budget first, entry count from the array header, network, status/cursor rules, first epoch, consecutive epochs, links, end id), TrustBaseResponseException for a well-formed but inconsistent page. Unsigned-64 epochs throughout.
  • TrustBaseSource (CompletableFuture<byte[]> getTrustBasePage(request); cancelling the future cancels the request) and org.unicitylabs.sdk.api.HttpTrustBaseSource: GET <origin>/api/v1/trustbases?from=E&maxEntries=N[&snapshotEnd=E&snapshotEntryId=0x..], Accept: application/cbor, response capped at 1 MiB + 1 KiB while streaming, redirects never followed, non-2xx is a refusal, X-SDK-Version per SdkVersionHeader (default/override/off, from UTB parity 1/4: network names (TESTNET2) and the X-SDK-Version header #1), API key as Authorization: Bearer (the JSON-RPC client's scheme) and only over HTTPS unless insecure transport is allowed.
  • TrustBaseUpdater.update / updateAsync(…, executor): frozen/echoed snapshot, cursor progress, sources tried in order, 2 rounds with a pause, busy/unavailable asked again, other refusals final for the call, "nothing newer" only when every source says it and no snapshot is frozen, an entry the history refuses keeps the authenticated prefix and moves to the next source, an honest source that refuses a fake echoed snapshot (HISTORY_MISMATCH) is asked again without it (Rust behaviour, not in the JS file), cancellation of the in-flight request and the pause. TrustBaseUpdateException carries Failure, per-source Causes and the appended count; an authenticated TrustBaseConflictException is rethrown, not treated as one source's refusal.
  • TrustBaseUpdater.verifyWithUpdate(target, sources, options, verify, isMissingEntry): verify; only on the typed no-entry result one update, then at most one more verification; a useful prefix allows the retry even if a later page failed. It is generic in the result and takes the no-entry test as a predicate, so this PR does not depend on UTB parity 5/8: GetTrustBase page decoder, HTTP source, fetch loop and verify-update-retry #4; UTB parity 5/8: GetTrustBase page decoder, HTTP source, fetch loop and verify-update-retry #4 supplies NoTrustBaseEntryResult as the predicate.
  • TrustBaseUpdateTarget (network, tip, append) with of(TrustBaseHistory); PR 6's session will implement it.

Decisions to review

Evidence

  • src/test/resources/vectors/gettrustbase/vectors.json is byte-identical to bft-core #928 (5d9a8eaa211b), sha256 3f9732fa5d5bed202d326f452f732d625040dbfac81857e19574df25bde3734c, pinned by a test. All 45 cases decode or are refused with the vector's class; exact budgets; the shared maximum-size entry (95,365 bytes) as a one-entry page, and an item over 128 KiB refused.
  • HttpTrustBaseSourceTest (MockWebServer, 9 tests): path/query, unsigned epochs and echoed snapshot, version header default/override/off, bearer key, plain-HTTP key refused, redirect refused, 503, the size cap, cancellation.
  • TrustBaseUpdaterTest (16 tests) over scripted sources mirroring Rust trust_base_update.rs, with signed synthetic chains. Mutants, each killed: snapshot-mismatch retry removed; "nothing newer" accepted with a snapshot; refusal not final; busy treated as final; appended count lost on refusal; authenticated fork swallowed; page start unchecked.
  • Built and tested on a real JDK 11 (Temurin 11.0.33, JAVA_HOME): compileJava, the trustbase and HTTP source tests (repeated 3 times), animalsnifferMain (Android API 31) clean, checkstyle clean for the main files (test files carry the same naming warnings as the existing tests).
  • No new dependency.

@ristik

ristik commented Nov 8, 2026

Copy link
Copy Markdown
Owner Author

java#4 (PR5/8) at 1d74173: no blockers in the code I read. APPROVE held, because nothing has run: there's no CI and I couldn't build locally.

Why nothing has run: this fork has no registered workflows (GET /actions/workflows returns none, and no run has ever started), so the CI never ran on #1 to #4. Someone with admin rights has to open the fork's Actions tab once and confirm "I understand my workflows, go ahead and enable them". Locally, only JDK 25 is installed, which this Gradle version refuses ("A problem occurred configuring root project: 25.0.5.0.3"). So I've run none of your tests or mutants myself.

Checked by reading, against JS TrustBaseUpdater.ts and Rust trust_base_update.rs:

  • Vectors: src/test/resources/vectors/gettrustbase/vectors.json is byte-identical to bft-core #928's gettrustbase/testdata/v1/vectors.json (SHA-256 3f9732fa…).
  • Fetch loop: the same semantics as both SDKs.
    • The cursor and the frozen snapshot belong to the whole call, and reset on a refused entry or when the target moves.
    • "Current" needs every source to say NOT_YET_AVAILABLE with an end below the cursor.
    • BUSY and UNAVAILABLE are retried on the next pass, and other statuses are final.
    • The HISTORY_MISMATCH-with-snapshot hardening drops the snapshot and asks the same source again, recording the cause.
    • The authenticated prefix is kept.
    • Unsigned 64-bit comparisons are used, and nextFrom saturates at 2^64 − 1.
  • verifyWithUpdate: retries only when the entry is missing, and verifies the prefix after a failed update that appended entries.
  • HTTP source: uses the origin's /api/v1/trustbases with from/maxEntries/snapshotEnd/snapshotEntryId=0x… and Accept: application/cbor. Redirects are off, the body is capped at 1 MiB + 1 KiB, and any non-2xx answer is refused. The API key is only sent over HTTPS unless overridden, and X-SDK-Version comes from SdkVersionHeader. The 30 s default call timeout also closes the gap JS still has (no default page timeout).
  • Cancellation: propagates to the in-flight Call.

Parity notes (not blocking):

  1. Authenticated conflict: Java rethrows TrustBaseConflictException straight out of update when the target reports an authenticated fork. JS records any TrustBaseError from append as a refused cause and goes to the next source, with the session latching the conflict. Rust's update records HistoryError::Conflict as a refused cause too. All three refuse the fork, but the error a caller sees differs. Java's is arguably the clearest, so align on purpose: either note it in the plan or follow JS and Rust.
  2. API key header: Java sends Authorization: Bearer <key>, which is consistent with Java's own JsonRpcAggregatorClient. JS and Rust send X-API-Key. The aggregator doesn't read either on this endpoint, but a gateway in front of it might. Worth one line in the plan.

Stack: this PR sits on dev3's #1 to #3, which haven't come to me for review yet. My verdict here covers only this PR's own delta (1d74173f against de67e4e0).

@ristik ristik closed this Nov 8, 2026
@ristik ristik reopened this Nov 8, 2026
@ristik

ristik commented Nov 8, 2026

Copy link
Copy Markdown
Owner Author

Update (rev3): my note 1 was wrong. I'd read JS from a clone that predated js#26. Current JS main (TrustBaseUpdater.ts:176-182) also rethrows an authenticated TrustBaseConflictError and HistoryConflictedError instead of recording a refused cause, so Java matches JS, and only Rust differs because its update takes a bare history. Note 1 is withdrawn. Note 2 (Bearer versus X-API-Key) stays as an owner-level question; Java's choice is consistent with its own client.

Ran locally on JDK 11 (/Users/risto/uni/agre/jdk11): test --tests '*trustbase*' --tests '*HttpTrustBaseSource*' compileJava animalsnifferMain passes, 81 tests and 0 failures. Mutants: with the HISTORY_MISMATCH-with-snapshot hardening disabled, 1 updater test fails; with the "current" boundary moved from < to <=, 1 fails. I'll approve once CI (now running after the Actions fix) is green at 1d74173.

@ristik

ristik commented Nov 8, 2026

Copy link
Copy Markdown
Owner Author

java#4 at 1d74173: APPROVE, for the owner's list. CI build is green at this head, the first run since Actions were enabled. Locally on JDK 11, the trust base tests, compileJava and animalsnifferMain pass, and my two updater mutants are caught. My review covers this PR's own delta; dev3's #1 to #3 below it are reviewed separately.

@ristik

ristik commented Nov 8, 2026

Copy link
Copy Markdown
Owner Author

java#4 at fb814af: APPROVE carried. The only change since 1d74173 is a merge of dev3's #7 (424b44d), and git merge-tree of the parents reproduces exactly its tree (e049f864), so nothing of yours changed. CI is green, and ./gradlew build passes at #6's head, which contains this (875 tests). Base: this PR still targets dev3/java-3-history, so its diff now also shows #7's changes. Retargeting it to dev3/java-4-verification would make the owner's view show only this PR's own delta, and the merge order is then #1, #2, #3, #7, #4, #5, #6.

@ristik
ristik changed the base branch from dev3/java-3-history to dev3/java-4-verification November 8, 2026 01:38
@ristik
ristik changed the base branch from dev3/java-4-verification to main November 8, 2026 03:07
@ristik
ristik merged commit 6addd05 into main Nov 8, 2026
1 check passed
@ristik
ristik deleted the dev2/java-5-fetch branch November 14, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant