Repository navigation
Unicity Trust Base v2: weighted BFT quorum (unequal votes), seals judged by the entry of their own epoch, release pins, cache and file store, GetTrustBase fetch loop, X-SDK-Version header (API-breaking, 4.0.0) - #101
Conversation
…ged by the entry of their own epoch, release pins, cache and file store, GetTrustBase fetch loop, X-SDK-Version header (API-breaking, 4.0.0) Signed trust-base entries (tag 39000) and bundles (tag 39015). Entries and seals use the BFT Core quorum with unequal votes: stake-weighted members, threshold floor(2W/3)+1 of the total weight W. Seals are verified against the entry of their own epoch; release pins for mainnet and testnet2; trust-base cache with a file store; GetTrustBase fetch loop and HTTP source; X-SDK-Version header. API-breaking (release version 4.0.0): RootTrustBase and VerificationPolicy removed, NetworkId.TESTNET removed, TESTNET2 added, beginOperation(PartitionViewPolicy). Unchanged: the token format, the proof and certificate wire formats and existing tokens. A seal with more than 256 signatures is refused at decode, and CBOR text must be valid UTF-8. See the PR description.
MastaP
left a comment
There was a problem hiding this comment.
Reviewed at a80b89c. The new commit changes docs only; the code is the same as at 39c3405, where I read every changed production line. Not approving yet: item 1.
What I checked:
./gradlew build(JDK 21): 886 tests, 0 failures, 4 skipped.- The core rules match bft-go-base#20 on a 245-case cross-implementation corpus, including unsigned 64-bit values, UTF-16 vs UTF-8 ordering and strict UTF-8.
- The release pins are byte-identical to unicity-ids#11.
- Real mainnet and testnet2 seals verify offline through
openPublic. - A 302 isn't followed and the key isn't replayed. The size cap applies to the actual stream.
- A fake snapshot end above the honest tips makes honest sources final for the call (
TrustBaseUpdater.java:244-267).- aggregator-go#273 answers
snapshotEndabove its tip with NotYetAvailable (internal/trustbase/utb/history.go:445-448). The drop-the-snapshot retry handles onlyHISTORY_MISMATCH(:254), so the honest source falls through tofinallyRefused(:263-265). - Against the real #273 handler serving 5 epochs: the honest source alone appends 4 (tip 5).
- With a hostile first source serving one real entry, a cursor and a fake
effectiveEnd:NONE_SERVED, appended 1, tip 2. TrustBaseUpdaterTest.anHonestSourceRefusingAFakeSnapshotIsAskedAgainWithoutItmodels the honest server answering HistoryMismatch for an end above its tip, which #273 doesn't do.- JS #206 and Rust #73 have the same logic.
- aggregator-go#273 answers
Lower:
verifySuccessorandTrustBaseHistory.appenddon't validate the successor's structure (TrustBaseEntry.java:624-648). Go'sVerifySuccessorrunsValidate()first.- Repro: an entry built with the public constructor, with
quorumThreshold1 where 3 is required for weight 4, and signed by the predecessor quorum. It is APPENDED, and a one-signer seal at that epoch then authenticates OK. - Go refuses the same entry: "quorum threshold 1, expected 3".
- Every wire path decodes, and so validates, first, so this is not reachable remotely. Calling
validate()at the start ofverifySuccessorcloses it.
- Repro: an entry built with the public constructor, with
- Non-inclusion loses the typed missing-entry signal.
NonInclusionVerifier.java:198-212reportsNO_TRUST_BASE_ENTRYasAUTHORITY_REJECTEDwith no cause;ProofConsolidator.java:534-546maps it toUNAUTHENTICATED.- So a non-inclusion proof sealed in an epoch the client doesn't hold yet looks final.
- With the real testnet2 vector and an empty pinned history, the certificate check gives
NoTrustBaseEntryResult, while non-inclusion givesreason=AUTHORITY_REJECTED cause=null.
- Test gaps.
- Removing the network check in
verifySuccessor(:629) fails none of the 886 tests. The "another network" case also changes the genesis, so the previous-id check fires first. - Removing the epoch check (
:633) also fails none.append's gap logic masks it, but a directverifySuccessorcall then accepts prev+2.
- Removing the network check in
Notes:
- Today's gateways (V1 body): you get PAGE causes, which are retryable and do no damage, and
openPublicdoesn't fetch. But with any V1 source listed,refreshcan't report "current". - Namespace check:
openPrivaterefuses only the exact stringpublic;Public,PUBLICandlocalare accepted. This is harmless with the shipped stores, because the keys include the genesis id. - Merge and release: merge after unicity-ids#11, and release with
-Pversion=4.0.0.
The non-inclusion corpus is re-vendored byte for byte from ristik/aggregator-go risto/ni-corpus-utb-authority (generator commit 7350e8f): cases unchanged, authorities are one-entry trust base v2 bundles named by trustBaseBundle, authority/*.json gone; the source lock and the pinned manifest hash follow. TrustBaseTestUtils.fromLegacyJson and src/test/resources/trust-base.json are removed: the consolidation and signature-recovery fixtures build their genesis entry directly, TokenE2ETest and AggregatorStack open the bundle the aggregator under test serves (GET /api/v1/trustbases, the local-network setup), and the JS interop script uses the 4.0.0 JS API with that bundle.
The docker stack is the TypeScript SDK's integration stack (src/test/resources/integration: compose file, aggregator Dockerfile, stack.env and build-images.sh): both images are built from pinned commits of the ristik forks, the root runs with the signed trust base history and the aggregator serves its bundle; AggregatorStack opens that bundle as a local session and checks its genesis against the one the stack wrote. The only trust-base.json left in the stack is the node-side input of the bft-core binaries. Non-inclusion and uc-consolidation corpora are re-vendored byte for byte from ristik/aggregator-go risto/ni-corpus-utb-authority (locks and pins updated): authorities are one-entry bundles on the local network (3), and the corpus support code opens bundles. The split-vector README records the 4.0.0 check.
…hority f072f4a (merged up with main: the bft-core pin of unicitynetwork#16); locks and pins updated
…(NOT_YET_AVAILABLE too), a successor is validated first, non-inclusion and consolidation keep the typed missing-entry cause, release pins at the merged unicity-ids ef00ff25 - TrustBaseUpdater: a source that answers a snapshot-carrying request with HISTORY_MISMATCH or NOT_YET_AVAILABLE (what aggregator-go answers for an end above its tip) is asked again from the tip without the snapshot. - TrustBaseEntry.verifySuccessor and TrustBaseHistory.append validate the entry's structure first, as Go's VerifySuccessor does. - NoTrustBaseEntryException: the typed missing-entry signal is the cause of a non-inclusion rejection and of a consolidation failure for certificate evidence. - Release pins: manifest source commit ef00ff25 (bundles unchanged, byte for byte). - Tests: the updater against the real server's answer, one test per successor rule (structure, network, epoch), the typed cause on the real testnet2 vector and in the consolidation corpus.
|
Thanks for the review. Pushed
Also: release pins re-pointed to the merged unicity-ids Tests only, no code change needed: the cross-SDK CBOR text-string vectors (generated by Go at bft-go-base dbf94198, shared with the JS and Rust SDKs) are vendored under and pass: a leading U+FEFF is kept, invalid UTF-8 is refused, and a member id that differs only by the leading U+FEFF is a member of its own whose signature counts. Notes: the V1-gateway and namespace-check points are unchanged (harmless as you describe). Verification: all 889 tests of the unit suites (without the Testcontainers, cucumber and JS-interop suites) pass with |
…e snapshot Dropping the snapshot after HISTORY_MISMATCH / NOT_YET_AVAILABLE no longer forgets that a higher end was announced: a not-yet-available answer without a snapshot whose end is below the next epoch counts as current only when no end at or above it was announced; otherwise it is transient and asked again in the next round. Reset where the snapshot is reset because an entry was refused or the target moved on.
…valid UTF-8 refused, BOM twin members count), generated by Go at bft-go-base dbf94198
MastaP
left a comment
There was a problem hiding this comment.
Re-reviewed at 0c24027. I read every production change since a80b89c (8 files). Items 1, 2 and 4 are fixed. Item 3 is fixed for non-inclusion and for evidence seals, but not for the other two seal checks of a consolidation. Not approving yet, for that one item.
Fixed:
- NotYetAvailable on a snapshot request: against the real aggregator-go#273 handler (at its fix head), my earlier scenario now ends at tip 5, the same as the honest source alone. Before the fix it ended at tip 2.
- Validation:
verifySuccessorandTrustBaseHistory.appendvalidate the entry first. - Tests:
TrustBaseSuccessorRulesTestcovers the network, epoch and structure checks one at a time.
Still open: item 3, for two of the three seal checks.
- Direct-target master seal (
ProofConsolidator.java:624-627):nullis passed as the cause.- Repro on the consolidation guard harness: the token's own proofs are in epoch 1, which the client holds. The batch is re-certified with the seal moved to epoch 2 and re-signed by the same root key.
- The certificate check of that UC gives
NoTrustBaseEntryResult(epoch 2, held tip 1). - But
ProofConsolidatorandTokenProofClientboth fail withUNAUTHENTICATEDandcause=null; only the message saysNO_TRUST_BASE_ENTRY.
- Partition-view seal (
PartitionViewAuthenticator.java:234-238): it throwsPartitionViewException(AUTHENTICATION)with no cause, andauthenticateViewwraps that.- Repro with
TestChain: a view sealed by an epoch-2 authority, checked against a history holding epoch 1, givesreason=AUTHENTICATION cause=null. - Control: with epoch 2 appended, the view authenticates.
- Repro with
The master seal is the newest seal in a consolidation, so after a committee rotation it is the one most likely to be in an epoch the client doesn't hold yet. JS #206 carries the typed cause at all three checks, and Rust #73 keeps the typed VerificationError at all three.
Checks:
./gradlew build(offline, JDK 21) passes, and CI passes.- The release pins are byte-identical to unicity-ids ef00ff25, and the manifest is
56b390ab…as in JS and Rust. - On the cross-implementation corpus, Java's verdicts are unchanged.
Notes: the corpora come from the ristik fork generator (details on JS #206), and the integration stack builds from ristik forks.
…d missing-entry cause A consolidation checks three seals; the evidence seal already carried the typed NoTrustBaseEntryException as the cause. The other two now do too: - ProofConsolidator.authenticateDirect: a direct-target master seal of an epoch the history does not hold fails UNAUTHENTICATED with the typed cause. - PartitionViewAuthenticator: AUTHENTICATION with the typed cause; the consolidation's view path passes it on as the direct cause, as at the other two checks (the message is unchanged). Final seal failures (bad quorum, wrong network) carry no typed cause. Tests: an epoch-2 re-signed direct target (ProofConsolidator and TokenProofClient) with an epoch-1 history, control with epoch 2 appended; an epoch-2 partition view with an epoch-1 history, control with epoch 2 appended; below quorum and an alien key carry no typed cause.
|
Thanks for the re-review. I pushed
Tests:
Docs: I added one sentence to
|
MastaP
left a comment
There was a problem hiding this comment.
Re-reviewed at fb6fe57. I read the production changes since 0c24027 (ProofConsolidator, PartitionViewAuthenticator and the docs sentence). Item 3 is now fixed at all three seal checks of a consolidation. Approving.
Verified:
- Direct-target master seal: my repro (the batch re-certified with its seal moved to epoch 2 under the same root key, against a history holding epoch 1) now fails
UNAUTHENTICATEDwithNoTrustBaseEntryException(epoch 2, held tip 1) as the cause, through bothProofConsolidatorandTokenProofClient. At 0c24027 the cause was null. - Partition-view seal: my
TestChainrepro (an epoch-2 view checked against epoch 1) now givesAUTHENTICATIONwith the typed cause. - The new tests catch a revert: putting back
nullat the master seal,nullinPartitionViewAuthenticator, oreas the cause inviewFailureeach fails at least one of the new tests inConsolidationGuardTestandPartitionViewAuthenticatorTest. - Failures other than a missing entry keep their previous cause, and no code depends on the view failure's cause being a
PartitionViewException. ./gradlew clean build --offline(JDK 21): 899 tests, 0 failures, 4 skipped. CI passes on this head.
Nit (non-blocking): viewFailure was inserted below the Javadoc of UnavailableSignal (ProofConsolidator.java:127, "Service outcome that ends the attempt as unavailable…"), so that comment now dangles above viewFailure's own Javadoc.
The notes from my previous review still apply: the corpora come from the ristik fork generator, and the integration stack builds from ristik forks.
Sync staging with upstream main (unicitynetwork#101 merged)
What changed
Weighted quorum. Entries and seals use the BFT Core quorum with unequal votes: each member carries a stake-weighted vote, the threshold is exactly
floor(2W/3)+1of the total weight W, and every successor entry needs that weight of its predecessor's members (a seal needs it of the entry of its own epoch).Verification no longer reads the single-record JSON trust base (
RootTrustBase). It runs against a trust-base history: Unicity Trust Base entries (CBOR tag 39000), started from a genesis entry whose entry id the application pins, extended only by entries that carry a weighted quorum of signatures of their predecessor (tag 39015 bundles are the file and wire form).TrustBaseEntry,TrustBaseBundle,TrustBaseHistory,TrustBaseHistoryView): strict decoding (unique encoding), identity over the unsigned bytes (EntryID = SHA-256), append-only chain, weighted quorum (more than two thirds of the predecessor's weight; an unknown or bad signature fails the entry), typed errors (TrustBaseGapException,TrustBaseConflictException,TrustBasePinMismatchException, ...). Limits: 256 members, 128-byte node ids, 128 KiB per entry (TrustBaseEntry.MAX_ENTRY_BYTES), pages of 1 MiB / 64 entries. The largest valid entry (256 members, 128-byte ids, 256 predecessor signatures, 95,365 bytes) round-trips as a shared fixture.UnicitySealTrustBaseEntryRule,SealQuorumRule, the verification session): the entry of the seal's epoch is looked up, the seal's round must lie inside that epoch's interval, then the weighted quorum of that entry is checked (the upstream seal rule: unknown or bad extra signatures count zero). Only a seal whose epoch is above the held tip (or an empty history) fails with the typed, retryableNoTrustBaseEntryResult; a seal of another network or of an epoch before the genesis is an ordinary, final failure. A seal carries at most 256 supplied signatures (UnicitySeal.MAX_WITNESSES, checked at decode, structure, session and rule).TrustBaseSource,HttpTrustBaseSource,TrustBaseUpdater):GET /api/v1/trustbases, bounded read, no redirects, optional bearer key, cancellation; the first page freezes the snapshot (end epoch and end entry id) and later pages echo it; sources are untrusted; an authenticated prefix is kept when a later page fails;verifyWithUpdateverifies, fetches once on the no-entry result and verifies once more.API-breaking changes (no backwards compatibility)
What does not change: the token format, the proof and certificate wire formats (inclusion and non-inclusion proofs, unicity certificate and seal encodings) and existing tokens: no encoder of a token, proof, certificate or seal is touched. Only the SDK API changes:
NetworkId.TESTNETremoved, the seal and verification API (TrustBaseHistoryView,beginOperation(PartitionViewPolicy)) and trust base loading (a bundle instead of the JSONRootTrustBase). The new decode bounds: a seal with more than 256 signatures is refused, and CBOR text strings must be valid UTF-8 (previously decoded with the platform charset).RootTrustBase(JSON authority) is removed, withfromJson/toJsonandbft-trustbase.*.jsonfiles.TrustBaseHistoryViewreplaces it in every verifier and caller (UnicityCertificateVerifier.verify,verifyCertificate,NonInclusionVerifier,ProofConsolidator, ...). A trust base serializes as a bundle (toBundle(),TrustBaseBundle.decode).getNetworkId()on a history is anOptional(empty while the history holds no entry), so minting code usesgetNetworkId().get().VerificationPolicyand the legacy quorum-only path are removed:beginOperation(PartitionViewPolicy)replacesbeginOperation(VerificationPolicy, PartitionViewPolicy).NoTrustBaseEntryResult, never a verification against the latest entry. More than 256 seal signatures is refused at decode.NetworkId.TESTNET(2) is removed; there is no constant for the old testnet and id 2 stays a valid unnamed id.NetworkId.TESTNET2(4) is added.MAINNET(1) andLOCAL(3) are unchanged.Version
All three SDKs ship as 4.0.0 (owner decision); the release must use
-Pversion=4.0.0(release.ymlpasses the workflow input; the last upstream tag is1.4.2, so this is a major bump for an API-breaking release). This PR also sets 4.0.0 where a default or documented version lives: thebuild.gradle.ktsfallback (was3.0-SNAPSHOT, used when-Pversionis not passed) and the README dependency line. The generatedSdkVersion.VERSION(from the Gradle project version) is what theX-SDK-Versionheader reports (java/4.0.0), so it cannot drift from the artifact. Fork main gets the same through ristik#11; this branch is rebuilt from main once that merges.Release pins (tied to unicitynetwork/unicity-ids#11)
ReleasePinscarries the genesis trust base of mainnet (network 1) and testnet2 (network 4) as library resources (src/main/resources/release-pins/): the two bundlesunicity-trustbase.{mainnet,testnet2}.cborandmanifest.json, byte-identical to the files of unicitynetwork/unicity-ids#11, commit513dc4fcf01ee802a97d59a5bc4e50ba64786d45(recorded in the manifest'ssource); generated, verified at load (bundle digest, genesis against its pin, network, epochs). The manifest is byte-identical to the JS and Rust SDKs' (sha2567bbaad4f…).0b6f8143e19e2a585be3534e8f1062f36c3b9621e506158f10abfbc9a56f4e4c9d01b8713cab42dab269e1ad6e244cf11e7f8c4b580682a98402db348b37bd524b49eb2bb0729051b89a48668fd6ef1fda284aad2e10b7d5f23aad4b5faef37f5b0a4fb7bd6c2d71068c0e65659aa60733dcc9083e9290fc22c2dd8e8e60f08aunicitynetwork/unicity-ids#11 is still open at the time of writing (head
513dc4fc). This PR must not merge before it does, or the pins must be regenerated from whatever lands. Real mainnet and testnet2 seals verify through these bundles (test vectors).Cache and store API
TrustBaseSession.openPublic(network, override, store, ...)starts from the embedded release pin (a newer public pinReleasePinmust extend the release lineage);openPrivate(namespace, pin, store, ...)is a private network with a pin you supply (never the public namespace);openLocal(bundle, store, ...)is a local deployment, taking network, genesis and digest from the bundle (namespacelocal). Each returns anOpenedwith the session andCacheDiagnostics (a refused cache leaves the embedded history in place).session.append(entry)/appendAsync,refresh(sources, ...),history()(read-onlyTrustBaseHistoryView),exportBundle(),flush(),isDirty(),getConflict()(two histories that disagree at one epoch are conflict evidence and are never silently merged, sticky).TrustBaseStore:read(key, maxBytes)andcompareAndSwap(key, expectedRevision, bytes);flushretries up toMAX_CAS_ATTEMPTS= 4 and reports a failed save separately without losing the dirty state.MemoryTrustBaseStoreis in the core.FileTrustBaseStore(package...trustbase.file, optional, nothing in the core refers to it): one file per key in a caller-chosen directory, namedutb-<sha256(key)>.cborwith the lock directoryutb-<sha256(key)>.lock, the same names as the JS and Rust adapters; bounded reads, atomic replacement (same-directory temporary file and atomic move), a lock that is never waited on and never stolen (contention returns BUSY). On Android the application picks the directory and a non-UI executor.Version header
Every aggregator request (including the trust-base GET) carries
X-SDK-Version: java/<project version>by default (SdkVersionHeader), so the SDK versions in use can be measured; the aggregator never reads it.SdkVersionHeader.of(value)sets a custom printable-ASCII value andSdkVersionHeader.disabled()turns it off. Unlike the JS SDK there is no browser case, so it is on by default everywhere.Shared vectors
Vendored byte for byte, with provenance in the README beside each:
vectors/trust-base-v2/vectors.json(bft-go-baseutb/upstream-v2,c32454f),vectors/trust-base-v2/max-entry/v1/max-entry.json(bft-go-baseutb/gettrustbase-codec,81ee667, the 256-member fixture),vectors/gettrustbase/vectors.json(bft-core GetTrustBase response vectors, 45 cases),vectors/utb-cache/v1/vectors.json(the cache vectors shared with the Rust and JS SDKs),vectors/real-seals/(real mainnet and testnet2 seals and the two bundles).Evidence
Checked on
upstream-pr/utb-v2(39c3405, upstream main1496b9aas the base), 2026-11-08, with the Gradle wrapper (8.8) and Temurin JDKs:./gradlew build: BUILD SUCCESSFUL. 886 tests, 0 failures, 0 errors, 4 skipped. Thechecktask includesanimalsnifferMainagainst the Android API 31 signature (gummy-bears-api-31), which passed../gradlew clean build androidJar jvmJar sourcesJar javadocJar: BUILD SUCCESSFUL, the same 886 tests, 0 failures, 4 skipped; the Android and JVM jars, sources and javadoc build. The Android animal-sniffer check is the Android evidence; nothing was run on an Android device or emulator.integration, Docker/Testcontainers) are excluded from./gradlew testand were not run here. CI on fork main9fe6bf8(run 42463319580) is green.Known nits
None open. (The README networks-table wording was fixed in ristik#10, merged as 5071407.)
Not in this PR
Documentation: README.md keeps setup concise and identifies the build JDK; docs/api-reference.md covers sessions, pins, refresh, stores, limits and errors.
trustBaseBundlein the contexts), cases unchanged, re-vendored byte for byte from ristik/aggregator-goristo/ni-corpus-utb-authority.risto/ni-corpus-utb-authority(counterpart bft-core change: ucfixture: local network (3) and trust base v2 bundle authority; regenerated uc-consolidation producer subset ristik/bft-core#1000).