fix(eth2api): decode Transaction as a bare SSZ byte list - #569
Merged
Conversation
`Transaction` is a one-field struct deriving `Encode`/`Decode`, so `ssz_derive` gave it container framing: a spurious 4-byte offset prefix per transaction. The spec type is `ByteList[MAX_BYTES_PER_TRANSACTION]` with no framing (charon models it as a plain `type Transaction []byte`), so pluto could not SSZ-decode any block carrying even one transaction. The validator client publishes signed blocks as SSZ, so every non-empty block came back `400 invalid submitted block`. With no proposer partial signature from the pluto nodes, a 4-node 2-charon/2-pluto cluster lost the whole duty: threshold 3 needs at least one pluto signature. In a kurtosis mixed cluster only 27 of 87 proposer duties were broadcast, and all of pluto's successful proposals contained zero transactions. Adding `struct_behaviour = "transparent"` restores the bare-list encoding. `TreeHash` deliberately keeps the derived container behaviour: merkleizing a one-field container yields the single leaf unchanged, so the root already matched the bare list and the `*_beacon_block_body_root` spec vectors (whose fixtures carry transactions) are unaffected. The gap that hid this was test coverage: the payload fixtures only exercised `TreeHash` and JSON, never an SSZ round trip. Both are now covered. Measured on a kurtosis 2-charon/2-pluto cluster, proposer duties broadcast cluster-wide went from 27/87 to 57/71, with charon (57) and pluto (58) at parity and `400 invalid submitted block` eliminated. Pluto now proposes blocks with up to 606 transactions and 12 blobs. Also fixes two things that made this hard to diagnose: - `ApiError::into_response` dropped the `source` field, so the real decode error was never logged despite the doc comment claiming it was "surfaced in debug logs only". The cause had to be read off the validator client instead. It is now logged with its `source()` chain. - `map_dutydb_error` is shared by `attestation_data` and `await_proposal` but hardcoded attestation wording, so a timed-out block proposal reported "attestation duty expired before data was stored". It now names the duty awaited. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
iamquang95
approved these changes
Jul 30, 2026
| // path every error response takes, otherwise it is silently dropped. | ||
| if let Some(source) = &self.source { | ||
| if self.status_code.is_server_error() { | ||
| tracing::warn!( |
Collaborator
There was a problem hiding this comment.
should this be error instead?
Charon logs 5xx validator api responses at error level (log.Error in writeError, router.go); match that for parity instead of warn. Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
bellatrix::Transactionis a one-field struct derivingEncode/Decode, sossz_derivegave it container framing — a spurious 4-byte offset prefix on every transaction. The spec type isByteList[MAX_BYTES_PER_TRANSACTION], a bare list with no framing; charon models the same type as a plaintype Transaction []byte.Consequence: pluto cannot SSZ-decode any block carrying even one transaction. The validator client publishes signed blocks as SSZ (
content-type: application/octet-stream), so every non-empty block was rejected with400 invalid submitted block. Without a proposer partial signature from the pluto nodes, a 4-node 2-charon/2-pluto cluster loses the duty entirely: threshold 3 with only 2 charon nodes needs at least one pluto signature.Charon's tracker reports this from the other side:
where
annoyed-adult/brave-mobileare the two pluto nodes.Diagnosis
Isolated by replaying captured validator-client submissions against a live node, then rebuilding the nested SSZ with individual fields swapped.
400means decode failed; anything later means it decoded.40040040004 00 00 00prefix)The threshold is one transaction — not a size limit, and nothing to do with blobs. Blob-carrying blocks simply correlate with busy blocks. Every block pluto successfully proposed contained zero transactions (22/22, then 17/17 across two runs).
Transactionis the only type in the tree with this defect;extra_data, a bareSszList<u8, _>in the same payload, decodes correctly and is the contrast case.Fix
#[ssz(struct_behaviour = "transparent")]onTransaction, restoring the bare-list encoding.TreeHashdeliberately keeps the derived container behaviour: merkleizing a one-field container yields the single leaf unchanged, so the root already equalled the bare list's root. The*_beacon_block_body_rootand*_execution_payload_rootspec vectors — whose fixtures carry transactions — pass unchanged, confirming block roots were never affected.Results
Measured on a kurtosis mixed 2-charon/2-pluto cluster (lighthouse VCs, Fulu), matched sample sizes:
400 invalid submitted block408timeoutsAll four nodes now broadcast 57–58: charon (57) and pluto (58) at parity. Pluto proposes blocks with up to 606 transactions and 12 blobs; 39 of 60 proposals carry transactions, matching the network's distribution. The
408s collapsing alongside indicates they were largely downstream of the same bug.Test gap this closes
The execution-payload fixtures only exercised
TreeHashand JSON — never an SSZ round trip — which is why container framing went unnoticed. Added:transaction_ssz_is_the_bare_byte_list— byte-exact encoding with no framing; raw bytes decode; a zero-length transaction is valid (a container required ≥4 bytes for its offset).execution_payload_with_transactions_ssz_round_trips— round-trips a payload carrying transactions and asserts the encoded length is4·N + Σ len.Diagnosability (why this needed a packet capture)
ApiError::into_responsedropped thesourcefield entirely, so the real decode error was never logged despite the doc comment saying it was "surfaced in debug logs only". The cause had to be read off the validator client. Now logged with its fullsource()chain —warnfor 5xx,debugotherwise.map_dutydb_erroris shared byattestation_dataandawait_proposalbut hardcoded attestation wording, so a timed-out block proposal reportedattestation duty expired before data was stored. It now names the duty awaited.Checks
cargo test -p pluto-eth2api: 128 passed (4integration::*need a docker socket)cargo test -p pluto-core: 604 passedcargo clippy --workspace --all-targets --all-features -- -D warnings: cleancargo +nightly fmt --all --check: cleancargo deny check: advisories ok, bans ok, licenses ok, sources okNot covered
~19% of proposer duties still aren't broadcast, but charon shows the same rate (57 vs 58), so it is not pluto-specific — genesis warm-up plus normal QBFT round-1 proposer timeouts.
🤖 Generated with Claude Code