From 16eeb227693334702915b8be60abe289fde873a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Apr 2026 08:05:09 +0000 Subject: [PATCH 1/5] spec: history sync completion signal (EOSE-equivalent) Co-Authored-By: Claude --- docs/specs/2026-04-24-history-sync-eose.md | 256 +++++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 docs/specs/2026-04-24-history-sync-eose.md diff --git a/docs/specs/2026-04-24-history-sync-eose.md b/docs/specs/2026-04-24-history-sync-eose.md new file mode 100644 index 00000000..b5763fbf --- /dev/null +++ b/docs/specs/2026-04-24-history-sync-eose.md @@ -0,0 +1,256 @@ +# History Sync Completion Signal (EOSE-equivalent) + +> **One-sentence summary:** Add an explicit `HistorySyncComplete` message +> to `willow-transport` so clients can tell — per topic, per provider — +> when backfill has finished and the gossip feed is live, eliminating +> the "is history still loading?" ambiguity that today's protocol +> silently imposes on every freshly-joined client. + +## Motivation + +Today a joining client subscribes to a server's gossip topic and starts +receiving events: some are historical (replayed by the replay worker +from its 1000-event ring buffer, or streamed by the storage worker from +SQLite), some are live. The client has no way to tell these apart. +The UI therefore has to guess — usually with a loading spinner that +stays up for a fixed debounce window, or that flips off the first time +the event stream goes quiet for N ms. Both heuristics are wrong on a +slow network and both waste milliseconds on a fast one. + +Nostr solved this cleanly with NIP-01's `EOSE` marker: `["EOSE", +]` is a zero-payload frame the relay emits after all stored +events for a subscription have been flushed, drawing a bright line +between backfill and live tail. We want the same property in Willow. +The details differ — Willow has no subscription ids, no single +authoritative relay, and two distinct sync providers — but the user- +visible invariant we need is identical: **after the client has seen +one of these markers from a trusted provider, it knows the loading +state has resolved and the UI can commit.** + +## Wire format + +Introduce a new `MessageType::HistorySyncComplete = 7` variant in +`crates/transport/src/lib.rs:64` alongside the existing seven tags. +The payload is a new struct: + +```rust +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct HistorySyncComplete { + /// The TopicId (blake3 of the canonical topic string) this marker applies to. + pub topic_id: [u8; 32], + /// The Ed25519 PeerId of the provider emitting the marker. + pub provider_peer: EndpointId, + /// Hash of the last event the provider streamed before emitting the marker, + /// or `None` if the provider had zero stored events for this topic. + pub last_event_hash: Option, + /// Monotonically-increasing cursor scoped to (topic_id, provider_peer). + /// Incremented when the provider restarts and re-streams from zero. + pub epoch: u64, +} +``` + +`topic_id` is carried explicitly (rather than implicit in the gossip +topic) so that the marker survives relay-bridge forwarding and so a +single audit log can correlate markers across topics. `provider_peer` +is the provider's `EndpointId`, **not** a signed field — the signature +lives on the Ed25519-wrapped envelope the way ordinary chat events are +wrapped in `crates/identity/src/lib.rs`. `last_event_hash` lets the +client detect truncation: if the last event the client *received* from +that provider does not match the hash in the marker, the client MUST +treat the sync as incomplete (see "Sharp edges" below). `epoch` exists +so a provider that crashes and restarts mid-stream cannot confuse a +client still holding a stale marker from the previous run — mirroring +Nostr's one-EOSE-per-REQ guarantee without needing a subscription id. + +The struct serialises well under the existing 256 KB +`MAX_DESER_SIZE` ceiling in `crates/transport/src/lib.rs:36` — it is +~80 bytes on the wire and imposes no new size class. + +## Per-topic vs per-request + +Nostr's EOSE is scoped to a subscription id because the client +explicitly requested a filtered view. Willow has no such structure: +`Network::subscribe()` in `crates/network/src/traits.rs:132` joins a +gossip mesh by `TopicId`, and from that point every peer in the mesh +sees every event. There is nothing to "close." + +We therefore scope the marker to `(topic_id, provider_peer, epoch)`: + +- **Topic** is the unit of subscription a client actually has. +- **Provider** is needed because, unlike Nostr, Willow has multiple + concurrent providers per topic (see next section). +- **Epoch** distinguishes fresh streams from a restarted provider. + +Critically, `HistorySyncComplete` is **not** broadcast to the whole +mesh. It is sent as a direct `broadcast_neighbors`-style frame from a +provider to a newly-joined peer, so non-joining peers don't see +redundant traffic. The relay forwards these frames verbatim (it +already forwards opaque bytes by topic; the new `MessageType` tag is +invisible to relay-side logic, which is deliberately stateless per +`crates/relay/src/lib.rs:13`). + +## Multiple providers + +A single server has, in the current architecture, three candidate +providers for history: + +| Provider | Source | Completeness | +|----------|--------|--------------| +| **Replay worker** | in-memory ring buffer, last 1000 events | lossy, recent | +| **Storage worker** | SQLite archival log | authoritative, long tail | +| **Peer** | any peer with more state than us | opportunistic | + +A joining client can receive up to three `HistorySyncComplete` markers +for the same topic. We define "caught up" as a **first-trusted-wins** +rule, with a pluggable policy: + +1. **Default policy (fast UI):** the client considers itself caught up + for a topic as soon as it has received a valid marker from **any + peer granted `SyncProvider` permission** (the relay-worker role in + the state machine). This minimises perceived load time and matches + the UX the feature exists to deliver. +2. **Strict policy (opt-in):** wait until markers have arrived from + **a majority of currently-connected `SyncProvider` peers**, or from + the storage worker specifically if one is in the trust list. This + is the right choice for clients that are about to take an offline + snapshot or archive state. + +Untrusted peers' markers are ignored entirely — accepting them would +let any peer prematurely flip the UI's loading flag off. + +## Client API + +Emit a new variant on `ClientEvent` in +`crates/client/src/events.rs:10`: + +```rust +/// Backfill from at least one trusted SyncProvider has finished for this topic. +HistorySynced { + topic: String, + provider: EndpointId, + /// Number of additional trusted providers still streaming history + /// for this topic. `0` means no providers are still streaming. + still_pending: usize, +}, +``` + +The UI subscribes to this event exactly the way it already subscribes +to `MessageReceived`. The natural consumer is the +`SyncCompleted { ops_applied }` variant already at +`crates/client/src/events.rs:48`: that existing event is session-wide +and is emitted after a batch of operations lands, whereas +`HistorySynced` is topic-scoped and marks the boundary, not the work. +Both are kept — they answer different questions. + +## Provider-side emission + +Emission is triggered by `NeighborUp` events on the provider's +`TopicEvents` stream (`crates/network/src/traits.rs:52`): + +- **Replay worker**: after flushing its 1000-event ring buffer (or + zero events if empty), emit a marker with the hash of the last + flushed event, or `None` when empty. +- **Storage worker**: snapshot the watermark *first*, stream every + row up to that watermark via `SELECT ... ORDER BY seq ASC`, emit + the marker. Events that arrive at the worker after the watermark + was taken are forwarded as ordinary gossip after the marker — they + are live, not historical. +- **Peer-to-peer**: emit after the DAG-diff exchange from + `2026-04-01-per-author-merkle-dag-state-design.md` completes and + before relaying any live events to the new neighbour. + +The provider tracks which neighbours it has already sent a marker to +in this `epoch` so a reconnect loop cannot spam the UI. + +## Sharp edges + +- **Silent truncation.** Nostr's EOSE does not guarantee completeness + against `max_limit` truncation, and the 1000-event replay buffer + has the same problem. The `last_event_hash` field lets clients + detect this: if the DAG cursor the client was tracking does not + link to `last_event_hash` through a known parent chain, the client + SHOULD fall back to the storage worker or a peer with a deeper + history before flipping the UI flag. +- **Provider lies.** A compromised `SyncProvider` can emit a marker + before actually flushing its history. The worst-case effect is a + stale UI; it cannot forge events (signatures still verify). Clients + MAY set a floor (e.g. "wait at least 200 ms and see at least one + event *or* one marker") to cap the damage. +- **Filter-change analogue.** Nostr has to define what EOSE means + when the filter changes mid-subscription; Willow does not, because + a topic change is literally a new subscription. Joining a new + channel = new `TopicId` = fresh marker cycle. This is cleaner + precisely *because* Willow is topic-scoped. +- **Offline peers and resumption.** A client that disconnects and + reconnects requests history again; providers emit a new marker with + a new `epoch`. The client SHOULD discard markers whose + `(provider_peer, epoch)` is older than one it has already seen + since the last `NeighborUp` for that provider. + +## Testing + +**State-level** (`crates/state/src/tests.rs`, `just test-state`): no +state-machine change — `HistorySyncComplete` is a transport-layer +message, not an `EventKind`. A single round-trip test in +`crates/transport/src/lib.rs` covers serde + size bounds. + +**Client-level** (`crates/client/src/lib.rs` test module, +`just test-client`): + +- emitting a marker from `MemNetwork` produces exactly one + `ClientEvent::HistorySynced` per `(topic, provider)` pair; +- a marker from an untrusted peer produces **no** event; +- reconnect with a new `epoch` re-emits; reconnect with the same + `epoch` does not; +- a `last_event_hash` mismatch between the marker and the last + received event produces a `HistorySynced { still_pending: _ }` + with a warning logged but no false-positive completion. + +**Relay-level** (`crates/relay/tests/`, `just test-relay`): verify the +relay forwards `MessageType::HistorySyncComplete` bytes unchanged (it +should already — the relay is opaque at the message-type level — but +the test pins that contract so a future size-bounded filter does not +silently drop the marker). + +**Browser-level** (`crates/web/tests/browser.rs`, `just test-browser`): +the loading spinner component hides on receipt of `HistorySynced` +for the active channel's topic and stays hidden across subsequent +live events. + +## Interaction with existing specs + +- `2026-04-01-per-author-merkle-dag-state-design.md` — the DAG-diff + protocol produces a natural "I have sent you everything in my + frontier" moment; `HistorySyncComplete` is the wire encoding of + that moment. +- `2026-04-12-state-authority-and-mutations.md` — markers do **not** + flow through `apply_event`. They are not authority-bearing; they + cannot grant, revoke, or mutate `ServerState`. A malicious marker + is a UX bug at worst, never a state bug. +- `2026-03-27-worker-nodes-design.md` — replay and storage workers + gain a new emission obligation documented above; no change to their + trust status. + +## Open questions + +1. Should `last_event_hash` be mandatory rather than optional? Making + it required forces providers to decide "am I empty?" vs "am I done + streaming?" explicitly, at the cost of one more type-level state + in the provider. +2. Do we want a separate `HistorySyncFailed { topic, provider, + reason }` message, or is absence-of-marker + a client-side timeout + sufficient? Nostr gets away with absence; Willow has richer error + information available at the provider. +3. Should the marker carry the provider's current state hash, so + clients can cross-check against the "majority-agreed state" + mechanism from `CLAUDE.md`'s trust model? This would unify history + completion and state convergence into one signal. +4. Per-channel vs per-server scope: today `TopicId` is per-channel, + but a server-wide marker ("all channels have caught up") would + simplify the join flow. Do we want both? +5. Is `epoch: u64` overkill? A simple `ChaCha20`-derived nonce would + avoid the "did I remember to bump it?" bug class, at the cost of + not being orderable. +6. Should markers be rate-limited at the relay? Today the relay is + content-agnostic, but a compromised provider could emit markers + in a tight loop and force every subscribed UI to re-render. From 59b0e9c4dc81277d386ac8f3b7c219146417c79c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Apr 2026 07:01:08 +0000 Subject: [PATCH 2/5] spec(#214): resolve review directives - Drop `provider_peer` from payload; receiver derives from verified envelope signer (forecloses MITM/relay-rewrite class). - Rename `epoch` -> `stream_generation` to disambiguate from the crypto-epoch in #220. - Switch from `broadcast_neighbors` to `broadcast` with receiver- side dedup on `(provider, stream_generation)`; no new transport primitives needed. - Soften NIP-01 over-claims; explicitly note the truncation-detection contract is deliberately stronger than NIP-01. - Fix `crates/relay/src/lib.rs:13` line citation; reference the `Trust model` section in module docs instead. --- docs/specs/2026-04-24-history-sync-eose.md | 103 +++++++++++++-------- 1 file changed, 63 insertions(+), 40 deletions(-) diff --git a/docs/specs/2026-04-24-history-sync-eose.md b/docs/specs/2026-04-24-history-sync-eose.md index b5763fbf..e6ceab4c 100644 --- a/docs/specs/2026-04-24-history-sync-eose.md +++ b/docs/specs/2026-04-24-history-sync-eose.md @@ -38,29 +38,34 @@ The payload is a new struct: pub struct HistorySyncComplete { /// The TopicId (blake3 of the canonical topic string) this marker applies to. pub topic_id: [u8; 32], - /// The Ed25519 PeerId of the provider emitting the marker. - pub provider_peer: EndpointId, /// Hash of the last event the provider streamed before emitting the marker, /// or `None` if the provider had zero stored events for this topic. pub last_event_hash: Option, - /// Monotonically-increasing cursor scoped to (topic_id, provider_peer). + /// Monotonically-increasing cursor scoped to (topic_id, provider). /// Incremented when the provider restarts and re-streams from zero. - pub epoch: u64, + pub stream_generation: u64, } ``` `topic_id` is carried explicitly (rather than implicit in the gossip topic) so that the marker survives relay-bridge forwarding and so a -single audit log can correlate markers across topics. `provider_peer` -is the provider's `EndpointId`, **not** a signed field — the signature -lives on the Ed25519-wrapped envelope the way ordinary chat events are -wrapped in `crates/identity/src/lib.rs`. `last_event_hash` lets the -client detect truncation: if the last event the client *received* from -that provider does not match the hash in the marker, the client MUST -treat the sync as incomplete (see "Sharp edges" below). `epoch` exists -so a provider that crashes and restarts mid-stream cannot confuse a -client still holding a stale marker from the previous run — mirroring -Nostr's one-EOSE-per-REQ guarantee without needing a subscription id. +single audit log can correlate markers across topics. The provider's +identity is deliberately **not** carried in the payload: the marker +rides the same Ed25519-wrapped envelope as ordinary events +(`crates/identity/src/lib.rs`), so the receiver derives the provider's +`EndpointId` from the verified envelope signer at unpack time. This +forecloses a class of relay-rewrite / MITM attacks where a separate +`provider_peer` field could be edited to attribute the marker to a +different trusted provider, and saves the bytes besides. +`last_event_hash` lets the client detect truncation: if the last event +the client *received* from that provider does not match the hash in +the marker, the client MUST treat the sync as incomplete (see "Sharp +edges" below). `stream_generation` exists so a provider that crashes +and restarts mid-stream cannot confuse a client still holding a stale +marker from the previous run. The semantics are inspired by Nostr's +one-EOSE-per-REQ pattern; the actual contract is adapted to Willow's +subscription-less topic gossip and is intentionally stronger than +NIP-01's (see "Sharp edges"). The struct serialises well under the existing 256 KB `MAX_DESER_SIZE` ceiling in `crates/transport/src/lib.rs:36` — it is @@ -74,20 +79,35 @@ explicitly requested a filtered view. Willow has no such structure: gossip mesh by `TopicId`, and from that point every peer in the mesh sees every event. There is nothing to "close." -We therefore scope the marker to `(topic_id, provider_peer, epoch)`: +We therefore scope the marker to `(topic_id, provider, stream_generation)`, +where `provider` is recovered from the verified envelope signer at +unpack time: - **Topic** is the unit of subscription a client actually has. - **Provider** is needed because, unlike Nostr, Willow has multiple concurrent providers per topic (see next section). -- **Epoch** distinguishes fresh streams from a restarted provider. - -Critically, `HistorySyncComplete` is **not** broadcast to the whole -mesh. It is sent as a direct `broadcast_neighbors`-style frame from a -provider to a newly-joined peer, so non-joining peers don't see -redundant traffic. The relay forwards these frames verbatim (it -already forwards opaque bytes by topic; the new `MessageType` tag is -invisible to relay-side logic, which is deliberately stateless per -`crates/relay/src/lib.rs:13`). +- **Stream generation** distinguishes fresh streams from a restarted + provider. + +The marker is sent as a regular `TopicHandle::broadcast` frame on the +topic's gossip mesh — the same path ordinary events take. This costs +one mesh-wide message per `NeighborUp` on the provider, which is +acceptable overhead for a per-(topic, provider, generation) deduped +marker; existing peers ignore markers whose +`(provider, stream_generation)` they have already observed. Relays +forward the marker by their existing topic-routing rules: see the +`Trust model` section in the module-level documentation of +`crates/relay/src/lib.rs`, which establishes the relay as +content-agnostic at the wire level. + +Using `broadcast` rather than `broadcast_neighbors` is a deliberate +trade. `broadcast_neighbors` (`crates/network/src/traits.rs:72`) is +documented as not forwarded by gossip relays, and there is no per-peer +direct-send primitive in the `Network` trait — so a "send only to the +new joiner" frame would either fail to traverse the relay bridge or +require new transport plumbing. Mesh broadcast plus +`(provider, stream_generation)` deduplication at receivers gives the +same effect with no new primitives. ## Multiple providers @@ -160,17 +180,19 @@ Emission is triggered by `NeighborUp` events on the provider's before relaying any live events to the new neighbour. The provider tracks which neighbours it has already sent a marker to -in this `epoch` so a reconnect loop cannot spam the UI. +in this `stream_generation` so a reconnect loop cannot spam the UI. ## Sharp edges -- **Silent truncation.** Nostr's EOSE does not guarantee completeness - against `max_limit` truncation, and the 1000-event replay buffer - has the same problem. The `last_event_hash` field lets clients - detect this: if the DAG cursor the client was tracking does not - link to `last_event_hash` through a known parent chain, the client - SHOULD fall back to the storage worker or a peer with a deeper - history before flipping the UI flag. +- **Silent truncation.** Nostr's EOSE deliberately does *not* guarantee + completeness against `max_limit` truncation; clients are expected to + consume a partial reply silently. Willow's contract is intentionally + stronger: `last_event_hash` lets clients detect truncation against + the 1000-event replay buffer or any other ring-buffered source. If + the DAG cursor the client was tracking does not link to + `last_event_hash` through a known parent chain, the client SHOULD + fall back to the storage worker or a peer with a deeper history + before flipping the UI flag. - **Provider lies.** A compromised `SyncProvider` can emit a marker before actually flushing its history. The worst-case effect is a stale UI; it cannot forge events (signatures still verify). Clients @@ -183,9 +205,10 @@ in this `epoch` so a reconnect loop cannot spam the UI. precisely *because* Willow is topic-scoped. - **Offline peers and resumption.** A client that disconnects and reconnects requests history again; providers emit a new marker with - a new `epoch`. The client SHOULD discard markers whose - `(provider_peer, epoch)` is older than one it has already seen - since the last `NeighborUp` for that provider. + a new `stream_generation`. The client SHOULD discard markers whose + `(provider, stream_generation)` is older than one it has already + seen since the last `NeighborUp` for that provider — where + `provider` is the verified envelope signer. ## Testing @@ -200,8 +223,8 @@ message, not an `EventKind`. A single round-trip test in - emitting a marker from `MemNetwork` produces exactly one `ClientEvent::HistorySynced` per `(topic, provider)` pair; - a marker from an untrusted peer produces **no** event; -- reconnect with a new `epoch` re-emits; reconnect with the same - `epoch` does not; +- reconnect with a new `stream_generation` re-emits; reconnect with + the same `stream_generation` does not; - a `last_event_hash` mismatch between the marker and the last received event produces a `HistorySynced { still_pending: _ }` with a warning logged but no false-positive completion. @@ -248,9 +271,9 @@ live events. 4. Per-channel vs per-server scope: today `TopicId` is per-channel, but a server-wide marker ("all channels have caught up") would simplify the join flow. Do we want both? -5. Is `epoch: u64` overkill? A simple `ChaCha20`-derived nonce would - avoid the "did I remember to bump it?" bug class, at the cost of - not being orderable. +5. Is `stream_generation: u64` overkill? A simple `ChaCha20`-derived + nonce would avoid the "did I remember to bump it?" bug class, at + the cost of not being orderable. 6. Should markers be rate-limited at the relay? Today the relay is content-agnostic, but a compromised provider could emit markers in a tight loop and force every subscribed UI to re-render. From 0addbf697e27ef8e5ba400e28300d0d057d03c44 Mon Sep 17 00:00:00 2001 From: Noah Date: Sat, 25 Apr 2026 02:01:12 -0700 Subject: [PATCH 3/5] spec(#214): apply audit findings - round 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - move HistorySyncComplete from a new MessageType variant in willow-transport to a new WireMessage variant in willow-common, matching the production convention that all gossip traffic is routed through MessageType::Channel and avoids a transport→state dependency cycle - correct motivation to describe the actual two sync paths (SyncRequest/SyncBatch on _willow_server_ops + WorkerRequest::Sync/ History on _willow_workers) instead of a fictional "events replayed on the gossip topic" path - add "Which gossip topic carries the marker" subsection clarifying that workers only subscribe to WORKERS_TOPIC and SERVER_OPS_TOPIC in current code, so the marker travels on _willow_server_ops for worker-served history (not per-channel topics) - describe replay worker as per-author DAG with max_events_per_author cap (default 1000) plus LRU server cap, not a flat 1000-event ring buffer - clarify SyncProvider is a permission, not a role - name the storage path the marker draws from (sync_since / ORDER BY seq ASC, not history() / DESC) and explain why - replace coined "DAG-diff protocol" with the spec's actual SyncMessage::Advertise/Request/Response terminology - reconcile motivation's "two distinct sync providers" with the three provider classes listed below - fix line citations: events.rs:19 (was :10), events.rs:57 (was :48), traits.rs:56 / :84 (was :52), relay/src/lib.rs:9-22 Scope-section (was Trust-model section) - drop duplicated "verified envelope signer" parenthetical in Sharp edges section Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/specs/2026-04-24-history-sync-eose.md | 219 +++++++++++++-------- 1 file changed, 139 insertions(+), 80 deletions(-) diff --git a/docs/specs/2026-04-24-history-sync-eose.md b/docs/specs/2026-04-24-history-sync-eose.md index e6ceab4c..41194228 100644 --- a/docs/specs/2026-04-24-history-sync-eose.md +++ b/docs/specs/2026-04-24-history-sync-eose.md @@ -1,71 +1,98 @@ # History Sync Completion Signal (EOSE-equivalent) -> **One-sentence summary:** Add an explicit `HistorySyncComplete` message -> to `willow-transport` so clients can tell — per topic, per provider — -> when backfill has finished and the gossip feed is live, eliminating -> the "is history still loading?" ambiguity that today's protocol -> silently imposes on every freshly-joined client. +> **One-sentence summary:** Add an explicit `HistorySyncComplete` +> variant to `WireMessage` in `willow-common` so clients can tell — +> per topic, per provider — when backfill has finished and the gossip +> feed is live, eliminating the "is history still loading?" ambiguity +> that today's protocol silently imposes on every freshly-joined client. ## Motivation -Today a joining client subscribes to a server's gossip topic and starts -receiving events: some are historical (replayed by the replay worker -from its 1000-event ring buffer, or streamed by the storage worker from -SQLite), some are live. The client has no way to tell these apart. -The UI therefore has to guess — usually with a loading spinner that -stays up for a fixed debounce window, or that flips off the first time -the event stream goes quiet for N ms. Both heuristics are wrong on a -slow network and both waste milliseconds on a fast one. +Today a joining client picks up history through two different paths: + +1. A `WireMessage::SyncRequest` / `SyncBatch` exchange on + `_willow_server_ops` (the SERVER_OPS topic), where peers and the + replay worker stream missing events back to the joiner. +2. Targeted `WorkerRequest::Sync` / `WorkerRequest::History` requests + on `_willow_workers` (the WORKERS topic), answered with + `WorkerResponse::SyncBatch`, `Snapshot`, or `HistoryPage`. + +In both paths the events that arrive on `_willow_server_ops` (or are +re-broadcast there after a worker hands them over) are interleaved +with live events that ordinary peers are publishing right now. The +client has no way to tell historical from live. The UI therefore has +to guess — usually with a loading spinner that stays up for a fixed +debounce window, or that flips off the first time the event stream +goes quiet for N ms. Both heuristics are wrong on a slow network and +both waste milliseconds on a fast one. Nostr solved this cleanly with NIP-01's `EOSE` marker: `["EOSE", ]` is a zero-payload frame the relay emits after all stored events for a subscription have been flushed, drawing a bright line between backfill and live tail. We want the same property in Willow. The details differ — Willow has no subscription ids, no single -authoritative relay, and two distinct sync providers — but the user- +authoritative relay, and history is served by three concurrent +provider classes (see "Multiple providers" below) — but the user- visible invariant we need is identical: **after the client has seen one of these markers from a trusted provider, it knows the loading state has resolved and the UI can commit.** ## Wire format -Introduce a new `MessageType::HistorySyncComplete = 7` variant in -`crates/transport/src/lib.rs:64` alongside the existing seven tags. -The payload is a new struct: +The existing transport-layer enum `MessageType` in +`crates/transport/src/lib.rs` (variants `Chat=0` … `Ping=6`, declared +on line 64) is bypassed by every production code path: all current +gossipsub traffic is dispatched through the single +`MessageType::Channel` envelope and routed by the inner +`WireMessage` enum at `crates/common/src/wire.rs:13`. Adding a new +`MessageType` variant for this signal would diverge from that +established convention for no benefit, and adding a payload type at +the transport layer would create a `willow-transport → willow-state` +dependency cycle (transport currently has no `EventHash` access; the +dependency graph in `CLAUDE.md` runs `client → state` and +`state → transport`). + +We therefore add a new variant to `WireMessage` in `willow-common` +(which already depends on `willow-state`): ```rust -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct HistorySyncComplete { - /// The TopicId (blake3 of the canonical topic string) this marker applies to. - pub topic_id: [u8; 32], - /// Hash of the last event the provider streamed before emitting the marker, - /// or `None` if the provider had zero stored events for this topic. - pub last_event_hash: Option, +/// Signals that a sync provider has finished streaming the historical +/// portion of its store for a topic. Subsequent events on the same +/// topic from this provider are live, not backfill. +HistorySyncComplete { + /// The TopicId (blake3 of the canonical topic string) this marker + /// applies to. Carried explicitly — see rationale below. + topic_id: [u8; 32], + /// Hash of the last event the provider streamed before emitting the + /// marker, or `None` if the provider had zero stored events for + /// this topic. + last_event_hash: Option, /// Monotonically-increasing cursor scoped to (topic_id, provider). /// Incremented when the provider restarts and re-streams from zero. - pub stream_generation: u64, -} + stream_generation: u64, +}, ``` `topic_id` is carried explicitly (rather than implicit in the gossip topic) so that the marker survives relay-bridge forwarding and so a single audit log can correlate markers across topics. The provider's -identity is deliberately **not** carried in the payload: the marker -rides the same Ed25519-wrapped envelope as ordinary events -(`crates/identity/src/lib.rs`), so the receiver derives the provider's -`EndpointId` from the verified envelope signer at unpack time. This -forecloses a class of relay-rewrite / MITM attacks where a separate -`provider_peer` field could be edited to attribute the marker to a -different trusted provider, and saves the bytes besides. -`last_event_hash` lets the client detect truncation: if the last event -the client *received* from that provider does not match the hash in -the marker, the client MUST treat the sync as incomplete (see "Sharp -edges" below). `stream_generation` exists so a provider that crashes -and restarts mid-stream cannot confuse a client still holding a stale -marker from the previous run. The semantics are inspired by Nostr's -one-EOSE-per-REQ pattern; the actual contract is adapted to Willow's -subscription-less topic gossip and is intentionally stronger than -NIP-01's (see "Sharp edges"). +identity is deliberately **not** carried in the payload: like every +other `WireMessage`, the marker rides the same Ed25519-signed +envelope built by `pack_wire` / verified by `unpack_wire` +(`crates/common/src/wire.rs:104-119`), so the receiver derives the +provider's `EndpointId` from the verified envelope signer at unpack +time. This forecloses a class of relay-rewrite / MITM attacks where a +separate `provider_peer` field could be edited to attribute the +marker to a different trusted provider, and saves the bytes besides. +`last_event_hash` lets the client detect truncation: if the last +event the client *received* from that provider does not match the +hash in the marker, the client MUST treat the sync as incomplete (see +"Sharp edges" below). `stream_generation` exists so a provider that +crashes and restarts mid-stream cannot confuse a client still holding +a stale marker from the previous run. The semantics are inspired by +Nostr's one-EOSE-per-REQ pattern; the actual contract is adapted to +Willow's subscription-less topic gossip and is intentionally stronger +than NIP-01's (see "Sharp edges"). The struct serialises well under the existing 256 KB `MAX_DESER_SIZE` ceiling in `crates/transport/src/lib.rs:36` — it is @@ -89,15 +116,29 @@ unpack time: - **Stream generation** distinguishes fresh streams from a restarted provider. +### Which gossip topic carries the marker + +In current code the workers do **not** subscribe to per-channel +topics: `crates/worker/src/runtime.rs` only joins `WORKERS_TOPIC` +(`_willow_workers`) and `SERVER_OPS_TOPIC` (`_willow_server_ops`). +Per-channel topics are joined only by clients. The marker therefore +travels on `_willow_server_ops` for server-state backfill (where the +joining client is already subscribed alongside the workers) and on +the per-channel topic for any peer-to-peer history that streams +directly between clients on that channel topic. A worker that did not +previously broadcast on the channel topic is **not** required to do +so by this spec; if a future worker grows direct channel-topic +backfill it must subscribe to that topic first. + The marker is sent as a regular `TopicHandle::broadcast` frame on the -topic's gossip mesh — the same path ordinary events take. This costs -one mesh-wide message per `NeighborUp` on the provider, which is +chosen topic — the same path ordinary events take. This costs one +mesh-wide message per `NeighborUp` on the provider, which is acceptable overhead for a per-(topic, provider, generation) deduped marker; existing peers ignore markers whose `(provider, stream_generation)` they have already observed. Relays forward the marker by their existing topic-routing rules: see the -`Trust model` section in the module-level documentation of -`crates/relay/src/lib.rs`, which establishes the relay as +`Scope: transport only` section in the module-level documentation of +`crates/relay/src/lib.rs:9-22`, which establishes the relay as content-agnostic at the wire level. Using `broadcast` rather than `broadcast_neighbors` is a deliberate @@ -112,23 +153,27 @@ same effect with no new primitives. ## Multiple providers A single server has, in the current architecture, three candidate -providers for history: +provider classes for history: | Provider | Source | Completeness | |----------|--------|--------------| -| **Replay worker** | in-memory ring buffer, last 1000 events | lossy, recent | +| **Replay worker** | per-author DAG with `max_events_per_author` cap (default 1000, see `crates/replay/src/role.rs:64`); LRU-evicted at `MAX_SERVERS = 1000` per node (`crates/replay/src/role.rs:18`) | lossy, recent | | **Storage worker** | SQLite archival log | authoritative, long tail | | **Peer** | any peer with more state than us | opportunistic | -A joining client can receive up to three `HistorySyncComplete` markers -for the same topic. We define "caught up" as a **first-trusted-wins** -rule, with a pluggable policy: +A joining client can receive up to three `HistorySyncComplete` +markers for the same topic (one per provider class). We define +"caught up" as a **first-trusted-wins** rule, with a pluggable +policy: 1. **Default policy (fast UI):** the client considers itself caught up for a topic as soon as it has received a valid marker from **any - peer granted `SyncProvider` permission** (the relay-worker role in - the state machine). This minimises perceived load time and matches - the UX the feature exists to deliver. + peer granted `SyncProvider` permission** + (`crates/state/src/event.rs:23` — `SyncProvider` is a permission + that any peer can hold, granted by the owner via + `EventKind::GrantPermission`; it is not a role). This minimises + perceived load time and matches the UX the feature exists to + deliver. 2. **Strict policy (opt-in):** wait until markers have arrived from **a majority of currently-connected `SyncProvider` peers**, or from the storage worker specifically if one is in the trust list. This @@ -141,7 +186,7 @@ let any peer prematurely flip the UI's loading flag off. ## Client API Emit a new variant on `ClientEvent` in -`crates/client/src/events.rs:10`: +`crates/client/src/events.rs:19`: ```rust /// Backfill from at least one trusted SyncProvider has finished for this topic. @@ -155,9 +200,9 @@ HistorySynced { ``` The UI subscribes to this event exactly the way it already subscribes -to `MessageReceived`. The natural consumer is the +to `MessageReceived`. The natural counterpart is the `SyncCompleted { ops_applied }` variant already at -`crates/client/src/events.rs:48`: that existing event is session-wide +`crates/client/src/events.rs:57`: that existing event is session-wide and is emitted after a batch of operations lands, whereas `HistorySynced` is topic-scoped and marks the boundary, not the work. Both are kept — they answer different questions. @@ -165,17 +210,29 @@ Both are kept — they answer different questions. ## Provider-side emission Emission is triggered by `NeighborUp` events on the provider's -`TopicEvents` stream (`crates/network/src/traits.rs:52`): - -- **Replay worker**: after flushing its 1000-event ring buffer (or - zero events if empty), emit a marker with the hash of the last - flushed event, or `None` when empty. +`TopicEvents` stream (`GossipEvent::NeighborUp` variant at +`crates/network/src/traits.rs:56`; the `TopicEvents` trait at +`crates/network/src/traits.rs:84`): + +- **Replay worker**: after responding to a peer's + `WorkerRequest::Sync` with the computed delta (or `Snapshot`), + emit a marker on `_willow_server_ops` with the hash of the last + event in that response, or `None` when it had nothing to send. + The replay worker today uses a per-author DAG (max-events-per-author + default 1000) and an LRU server cap of 1000 — see + `crates/replay/src/role.rs:18,64`. - **Storage worker**: snapshot the watermark *first*, stream every - row up to that watermark via `SELECT ... ORDER BY seq ASC`, emit - the marker. Events that arrive at the worker after the watermark - was taken are forwarded as ordinary gossip after the marker — they - are live, not historical. -- **Peer-to-peer**: emit after the DAG-diff exchange from + row up to that watermark via the `sync_since` path + (`crates/storage/src/store.rs:289-347`, `ORDER BY seq ASC`) — not + the paginated `history()` path (`store.rs:184-238`, + `ORDER BY seq DESC`), which is for explicit user-initiated history + fetches. Emit the marker after the last row is sent. Events that + arrive at the worker after the watermark was taken are forwarded as + ordinary gossip after the marker — they are live, not historical. +- **Peer-to-peer**: emit after the + `SyncMessage::Advertise(HeadsSummary)` → + `SyncMessage::Request(Vec)` → + `SyncMessage::Response(Vec)` exchange from `2026-04-01-per-author-merkle-dag-state-design.md` completes and before relaying any live events to the new neighbour. @@ -207,15 +264,16 @@ in this `stream_generation` so a reconnect loop cannot spam the UI. reconnects requests history again; providers emit a new marker with a new `stream_generation`. The client SHOULD discard markers whose `(provider, stream_generation)` is older than one it has already - seen since the last `NeighborUp` for that provider — where - `provider` is the verified envelope signer. + seen since the last `NeighborUp` for that provider. ## Testing **State-level** (`crates/state/src/tests.rs`, `just test-state`): no -state-machine change — `HistorySyncComplete` is a transport-layer -message, not an `EventKind`. A single round-trip test in -`crates/transport/src/lib.rs` covers serde + size bounds. +state-machine change — `HistorySyncComplete` is a wire-layer +`WireMessage` variant, not an `EventKind`. A round-trip test added +to `crates/common/src/wire.rs`'s test module (alongside the existing +`pack_unpack_*_round_trip` cases) covers serde + envelope signing + +size bounds. **Client-level** (`crates/client/src/lib.rs` test module, `just test-client`): @@ -230,10 +288,10 @@ message, not an `EventKind`. A single round-trip test in with a warning logged but no false-positive completion. **Relay-level** (`crates/relay/tests/`, `just test-relay`): verify the -relay forwards `MessageType::HistorySyncComplete` bytes unchanged (it -should already — the relay is opaque at the message-type level — but -the test pins that contract so a future size-bounded filter does not -silently drop the marker). +relay forwards `WireMessage::HistorySyncComplete`-bearing envelopes +unchanged (it should already — the relay is opaque at the wire level, +see `crates/relay/src/lib.rs:9-22` — but the test pins that contract +so a future size-bounded filter does not silently drop the marker). **Browser-level** (`crates/web/tests/browser.rs`, `just test-browser`): the loading spinner component hides on receipt of `HistorySynced` @@ -242,10 +300,11 @@ live events. ## Interaction with existing specs -- `2026-04-01-per-author-merkle-dag-state-design.md` — the DAG-diff - protocol produces a natural "I have sent you everything in my - frontier" moment; `HistorySyncComplete` is the wire encoding of - that moment. +- `2026-04-01-per-author-merkle-dag-state-design.md` — the + `SyncMessage::Advertise` / `Request` / `Response` exchange (over + per-author `HeadsSummary`) produces a natural "I have sent you + everything in my frontier" moment; `HistorySyncComplete` is the + wire encoding of that moment. - `2026-04-12-state-authority-and-mutations.md` — markers do **not** flow through `apply_event`. They are not authority-bearing; they cannot grant, revoke, or mutate `ServerState`. A malicious marker From 775c7baa4e6baadb2ed54e4b318bc4d9a182820e Mon Sep 17 00:00:00 2001 From: Noah Date: Sat, 25 Apr 2026 02:15:43 -0700 Subject: [PATCH 4/5] spec(#214): align with current sync paths, add broadcast-handle prerequisite - round 2 - Motivation: drop the false claim that clients consume WorkerRequest/Response today - No re-broadcast on _willow_server_ops; corrected - Peer-to-peer provider class explicitly conditional on per-author DAG sync spec - Prerequisite: replay/storage workers must add SERVER_OPS broadcast handle - Storage worker marker transport pinned: (a) broadcast on SERVER_OPS, not (b) WorkerResponse - Minor line cites and wording (MessageType:64->66, wire.rs:105-120, relay scope:8-22) Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/specs/2026-04-24-history-sync-eose.md | 186 ++++++++++++++------- 1 file changed, 122 insertions(+), 64 deletions(-) diff --git a/docs/specs/2026-04-24-history-sync-eose.md b/docs/specs/2026-04-24-history-sync-eose.md index 41194228..b073cae7 100644 --- a/docs/specs/2026-04-24-history-sync-eose.md +++ b/docs/specs/2026-04-24-history-sync-eose.md @@ -8,30 +8,31 @@ ## Motivation -Today a joining client picks up history through two different paths: - -1. A `WireMessage::SyncRequest` / `SyncBatch` exchange on - `_willow_server_ops` (the SERVER_OPS topic), where peers and the - replay worker stream missing events back to the joiner. -2. Targeted `WorkerRequest::Sync` / `WorkerRequest::History` requests - on `_willow_workers` (the WORKERS topic), answered with - `WorkerResponse::SyncBatch`, `Snapshot`, or `HistoryPage`. - -In both paths the events that arrive on `_willow_server_ops` (or are -re-broadcast there after a worker hands them over) are interleaved -with live events that ordinary peers are publishing right now. The -client has no way to tell historical from live. The UI therefore has -to guess — usually with a loading spinner that stays up for a fixed -debounce window, or that flips off the first time the event stream -goes quiet for N ms. Both heuristics are wrong on a slow network and -both waste milliseconds on a fast one. +Today a joining client picks up history through a single client-visible +path: a `WireMessage::SyncRequest` / `SyncBatch` exchange on +`_willow_server_ops` (the SERVER_OPS topic), where peers and the replay +worker stream missing events back to the joiner. (A second path exists +strictly between workers — `WorkerRequest::Sync` / `WorkerResponse::SyncBatch` +on `_willow_workers` — but clients neither send nor subscribe to those +messages today; see the explicit TODO at +`crates/client/src/listeners.rs:294-295` to migrate clients onto the +worker heads-based protocol. Until that lands, history reaches the +client only via SERVER_OPS.) + +Events that arrive on `_willow_server_ops` are interleaved with live +events that ordinary peers are publishing right now. The client has no +way to tell historical from live. The UI therefore has to guess — +usually with a loading spinner that stays up for a fixed debounce +window, or that flips off the first time the event stream goes quiet +for N ms. Both heuristics are wrong on a slow network and both waste +milliseconds on a fast one. Nostr solved this cleanly with NIP-01's `EOSE` marker: `["EOSE", ]` is a zero-payload frame the relay emits after all stored events for a subscription have been flushed, drawing a bright line between backfill and live tail. We want the same property in Willow. The details differ — Willow has no subscription ids, no single -authoritative relay, and history is served by three concurrent +authoritative relay, and history is (or will be) served by multiple provider classes (see "Multiple providers" below) — but the user- visible invariant we need is identical: **after the client has seen one of these markers from a trusted provider, it knows the loading @@ -40,8 +41,9 @@ state has resolved and the UI can commit.** ## Wire format The existing transport-layer enum `MessageType` in -`crates/transport/src/lib.rs` (variants `Chat=0` … `Ping=6`, declared -on line 64) is bypassed by every production code path: all current +`crates/transport/src/lib.rs` (declared on line 64; variants `Chat=0` +… `Ping=6` on lines 66-78) is bypassed by every production code path: +all current gossipsub traffic is dispatched through the single `MessageType::Channel` envelope and routed by the inner `WireMessage` enum at `crates/common/src/wire.rs:13`. Adding a new @@ -79,7 +81,7 @@ single audit log can correlate markers across topics. The provider's identity is deliberately **not** carried in the payload: like every other `WireMessage`, the marker rides the same Ed25519-signed envelope built by `pack_wire` / verified by `unpack_wire` -(`crates/common/src/wire.rs:104-119`), so the receiver derives the +(`crates/common/src/wire.rs:105-120`), so the receiver derives the provider's `EndpointId` from the verified envelope signer at unpack time. This forecloses a class of relay-rewrite / MITM attacks where a separate `provider_peer` field could be edited to attribute the @@ -119,13 +121,42 @@ unpack time: ### Which gossip topic carries the marker In current code the workers do **not** subscribe to per-channel -topics: `crates/worker/src/runtime.rs` only joins `WORKERS_TOPIC` -(`_willow_workers`) and `SERVER_OPS_TOPIC` (`_willow_server_ops`). -Per-channel topics are joined only by clients. The marker therefore -travels on `_willow_server_ops` for server-state backfill (where the -joining client is already subscribed alongside the workers) and on -the per-channel topic for any peer-to-peer history that streams -directly between clients on that channel topic. A worker that did not +topics: `crates/worker/src/runtime.rs:30-34` joins only +`WORKERS_TOPIC` (`_willow_workers`) and `SERVER_OPS_TOPIC` +(`_willow_server_ops`). Per-channel topics are joined only by +clients. **Worker → SERVER_OPS is currently ingest-only.** The +runtime binds the SERVER_OPS subscription handle as +`let (_ops_sender, ops_events) = ...` (note the underscore), and all +worker outbound traffic (`HeartbeatActor`, `SyncActor`, the request +reply path in `crates/worker/src/actors/network.rs:141-149`) goes out +on `workers_sender`, the WORKERS topic. Worker replies are +`WorkerWireMessage::Response` frames on WORKERS — they are not +re-broadcast onto SERVER_OPS today. + +This spec selects, of the two transports a worker-emitted marker +could ride, **option (a): extend the worker runtime to broadcast on +SERVER_OPS**. The alternative (b) — ride the existing +`WorkerWireMessage::Response` path on WORKERS and require clients to +subscribe to that topic — was rejected because it inverts the topic's +semantics (WORKERS today carries worker-to-worker control traffic +only; clients are intentionally absent), it forces every client onto +a mesh sized for workers, and it couples the marker's lifetime to a +specific `WorkerRequest`/`Response` correlation rather than a +broadcast event any subscriber on the channel/server topic can see. + +**Prerequisite (in scope of the implementing PR, not this spec):** +both the replay worker and the storage worker must add an outbound +broadcast handle for `_willow_server_ops`. Concretely, change the +runtime binding from `let (_ops_sender, ops_events) = ...` to a named +sender and thread it into whichever actor emits the marker. No new +network primitive is required — `TopicHandle::broadcast` already +exists; only the binding does not. + +With that prerequisite in place, the marker travels on +`_willow_server_ops` for server-state backfill (where the joining +client is already subscribed alongside the workers) and on the +per-channel topic for any peer-to-peer history that streams directly +between clients on that channel topic. A worker that did not previously broadcast on the channel topic is **not** required to do so by this spec; if a future worker grows direct channel-topic backfill it must subscribe to that topic first. @@ -138,7 +169,7 @@ marker; existing peers ignore markers whose `(provider, stream_generation)` they have already observed. Relays forward the marker by their existing topic-routing rules: see the `Scope: transport only` section in the module-level documentation of -`crates/relay/src/lib.rs:9-22`, which establishes the relay as +`crates/relay/src/lib.rs:8-22`, which establishes the relay as content-agnostic at the wire level. Using `broadcast` rather than `broadcast_neighbors` is a deliberate @@ -152,14 +183,16 @@ same effect with no new primitives. ## Multiple providers -A single server has, in the current architecture, three candidate -provider classes for history: +A single server has, in the architecture this spec targets, three +candidate provider classes for history. The first two exist today; +the third is **conditional on the per-author Merkle-DAG sync protocol +landing** (see "Interaction with existing specs"): -| Provider | Source | Completeness | -|----------|--------|--------------| -| **Replay worker** | per-author DAG with `max_events_per_author` cap (default 1000, see `crates/replay/src/role.rs:64`); LRU-evicted at `MAX_SERVERS = 1000` per node (`crates/replay/src/role.rs:18`) | lossy, recent | -| **Storage worker** | SQLite archival log | authoritative, long tail | -| **Peer** | any peer with more state than us | opportunistic | +| Provider | Source | Completeness | Status | +|----------|--------|--------------|--------| +| **Replay worker** | per-author DAG with `max_events_per_author` cap (default 1000, see `crates/replay/src/role.rs:64`); LRU-evicted at `MAX_SERVERS = 1000` per node (`crates/replay/src/role.rs:18`) | lossy, recent | exists today | +| **Storage worker** | SQLite archival log | authoritative, long tail | exists today | +| **Peer** | any peer with more state than us | opportunistic | **conditional** — requires `SyncMessage` wire plumbing (see below) | A joining client can receive up to three `HistorySyncComplete` markers for the same topic (one per provider class). We define @@ -212,29 +245,48 @@ Both are kept — they answer different questions. Emission is triggered by `NeighborUp` events on the provider's `TopicEvents` stream (`GossipEvent::NeighborUp` variant at `crates/network/src/traits.rs:56`; the `TopicEvents` trait at -`crates/network/src/traits.rs:84`): - -- **Replay worker**: after responding to a peer's - `WorkerRequest::Sync` with the computed delta (or `Snapshot`), - emit a marker on `_willow_server_ops` with the hash of the last - event in that response, or `None` when it had nothing to send. - The replay worker today uses a per-author DAG (max-events-per-author - default 1000) and an LRU server cap of 1000 — see +`crates/network/src/traits.rs:84`). All worker-side emissions +described below are gated on the broadcast-handle prerequisite +recorded in "Which gossip topic carries the marker" — neither the +replay nor the storage worker can broadcast on `_willow_server_ops` +in the current code. + +- **Replay worker** (requires SERVER_OPS broadcast handle): after + responding to a peer's `WorkerRequest::Sync` with the computed + delta (or `Snapshot`), emit a marker on `_willow_server_ops` with + the hash of the last event in that response, or `None` when it had + nothing to send. The reply itself still goes back as + `WorkerWireMessage::Response` on `_willow_workers` (worker-to-worker + RPC); only the `HistorySyncComplete` marker is broadcast on + SERVER_OPS, where the joining client is subscribed. The replay + worker today uses a per-author DAG (max-events-per-author default + 1000) and an LRU server cap of 1000 — see `crates/replay/src/role.rs:18,64`. -- **Storage worker**: snapshot the watermark *first*, stream every - row up to that watermark via the `sync_since` path - (`crates/storage/src/store.rs:289-347`, `ORDER BY seq ASC`) — not - the paginated `history()` path (`store.rs:184-238`, - `ORDER BY seq DESC`), which is for explicit user-initiated history - fetches. Emit the marker after the last row is sent. Events that - arrive at the worker after the watermark was taken are forwarded as - ordinary gossip after the marker — they are live, not historical. -- **Peer-to-peer**: emit after the +- **Storage worker** (requires SERVER_OPS broadcast handle): + snapshot the watermark *first*, stream every row up to that + watermark via the `sync_since` path (`crates/storage/src/store.rs:289-347`, + `ORDER BY seq ASC`) — not the paginated `history()` path + (`store.rs:184-238`, `ORDER BY seq DESC`), which is for explicit + user-initiated history fetches. The streamed rows themselves still + ride the existing `WorkerResponse::SyncBatch` reply on + `_willow_workers`; the marker is emitted on `_willow_server_ops` + after the last row is sent so any client subscribed to SERVER_OPS + sees it. Events that arrive at the worker after the watermark was + taken are forwarded as ordinary gossip after the marker — they are + live, not historical. +- **Peer-to-peer** (**conditional** on the per-author DAG sync wire + protocol landing): `SyncMessage` is defined in + `crates/state/src/sync.rs:37-44` but has zero producers or + consumers anywhere on the network today — there is no + `WireMessage::Advertise`/`Request`/`Response` variant, no handler. + When that protocol is wired up per + `2026-04-01-per-author-merkle-dag-state-design.md`, peer providers + emit a marker after the `SyncMessage::Advertise(HeadsSummary)` → `SyncMessage::Request(Vec)` → - `SyncMessage::Response(Vec)` exchange from - `2026-04-01-per-author-merkle-dag-state-design.md` completes and - before relaying any live events to the new neighbour. + `SyncMessage::Response(Vec)` exchange completes and before + relaying any live events to the new neighbour. Until then, only + the worker-emitted markers exist. The provider tracks which neighbours it has already sent a marker to in this `stream_generation` so a reconnect loop cannot spam the UI. @@ -245,7 +297,8 @@ in this `stream_generation` so a reconnect loop cannot spam the UI. completeness against `max_limit` truncation; clients are expected to consume a partial reply silently. Willow's contract is intentionally stronger: `last_event_hash` lets clients detect truncation against - the 1000-event replay buffer or any other ring-buffered source. If + the per-author 1000-event chain cap or any other ring-buffered + source. If the DAG cursor the client was tracking does not link to `last_event_hash` through a known parent chain, the client SHOULD fall back to the storage worker or a peer with a deeper history @@ -290,7 +343,7 @@ size bounds. **Relay-level** (`crates/relay/tests/`, `just test-relay`): verify the relay forwards `WireMessage::HistorySyncComplete`-bearing envelopes unchanged (it should already — the relay is opaque at the wire level, -see `crates/relay/src/lib.rs:9-22` — but the test pins that contract +see `crates/relay/src/lib.rs:8-22` — but the test pins that contract so a future size-bounded filter does not silently drop the marker). **Browser-level** (`crates/web/tests/browser.rs`, `just test-browser`): @@ -304,14 +357,19 @@ live events. `SyncMessage::Advertise` / `Request` / `Response` exchange (over per-author `HeadsSummary`) produces a natural "I have sent you everything in my frontier" moment; `HistorySyncComplete` is the - wire encoding of that moment. + wire encoding of that moment. **Blocks-on:** the peer-to-peer + provider class enumerated in "Multiple providers" cannot ship until + this spec lands a `WireMessage` variant for `SyncMessage` and the + per-channel-topic handshake to drive it. The replay/storage worker + provider classes do **not** depend on this spec and can ship first. - `2026-04-12-state-authority-and-mutations.md` — markers do **not** flow through `apply_event`. They are not authority-bearing; they cannot grant, revoke, or mutate `ServerState`. A malicious marker is a UX bug at worst, never a state bug. - `2026-03-27-worker-nodes-design.md` — replay and storage workers - gain a new emission obligation documented above; no change to their - trust status. + gain a new emission obligation documented above plus a new outbound + broadcast handle for `_willow_server_ops` (see "Which gossip topic + carries the marker"); no change to their trust status. ## Open questions @@ -330,9 +388,9 @@ live events. 4. Per-channel vs per-server scope: today `TopicId` is per-channel, but a server-wide marker ("all channels have caught up") would simplify the join flow. Do we want both? -5. Is `stream_generation: u64` overkill? A simple `ChaCha20`-derived - nonce would avoid the "did I remember to bump it?" bug class, at - the cost of not being orderable. +5. Is `stream_generation: u64` overkill? A random `u64` (e.g. from + `ChaCha20Rng`) would avoid the "did I remember to bump it?" bug + class, at the cost of not being orderable. 6. Should markers be rate-limited at the relay? Today the relay is content-agnostic, but a compromised provider could emit markers in a tight loop and force every subscribed UI to re-render. From 51514bc1b51006644bf96889c2dceb64cf0fc843 Mon Sep 17 00:00:00 2001 From: Noah Date: Sat, 25 Apr 2026 02:24:35 -0700 Subject: [PATCH 5/5] spec(#214): fix peer-to-peer blocks-on attribution + minor wording - round 4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Blocks-on attribution: per-author DAG spec, not this spec, lands SyncMessage wire variant - Dep graph edge: state → identity → transport (transitive) - Storage worker: one SyncBatch reply on _willow_workers (not a stream) - Motivation: clarify SERVER_OPS carries state-sync, not live chat - TopicId: shared type, not "per-channel" Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/specs/2026-04-24-history-sync-eose.md | 62 +++++++++++++--------- 1 file changed, 38 insertions(+), 24 deletions(-) diff --git a/docs/specs/2026-04-24-history-sync-eose.md b/docs/specs/2026-04-24-history-sync-eose.md index b073cae7..2b1e40e1 100644 --- a/docs/specs/2026-04-24-history-sync-eose.md +++ b/docs/specs/2026-04-24-history-sync-eose.md @@ -19,13 +19,18 @@ messages today; see the explicit TODO at worker heads-based protocol. Until that lands, history reaches the client only via SERVER_OPS.) -Events that arrive on `_willow_server_ops` are interleaved with live -events that ordinary peers are publishing right now. The client has no -way to tell historical from live. The UI therefore has to guess — -usually with a loading spinner that stays up for a fixed debounce -window, or that flips off the first time the event stream goes quiet -for N ms. Both heuristics are wrong on a slow network and both waste -milliseconds on a fast one. +Today's history path piggybacks on the SERVER_OPS state-sync exchange +(SERVER_OPS carries server-level mutations + `SyncRequest`/`SyncBatch`, +not live chat — chat events flow on per-channel topics). Because chat +events live in the same DAG as those server mutations, they ride back +to the joiner inside the same `SyncBatch` payload as the historical +state — interleaved with whatever new server-state events ordinary +peers happen to be publishing on `_willow_server_ops` right now. The +client has no way to tell historical from live. The UI therefore has +to guess — usually with a loading spinner that stays up for a fixed +debounce window, or that flips off the first time the event stream +goes quiet for N ms. Both heuristics are wrong on a slow network and +both waste milliseconds on a fast one. Nostr solved this cleanly with NIP-01's `EOSE` marker: `["EOSE", ]` is a zero-payload frame the relay emits after all stored @@ -52,7 +57,7 @@ established convention for no benefit, and adding a payload type at the transport layer would create a `willow-transport → willow-state` dependency cycle (transport currently has no `EventHash` access; the dependency graph in `CLAUDE.md` runs `client → state` and -`state → transport`). +`state → identity → transport` (via willow-identity)). We therefore add a new variant to `WireMessage` in `willow-common` (which already depends on `willow-state`): @@ -263,17 +268,19 @@ in the current code. 1000) and an LRU server cap of 1000 — see `crates/replay/src/role.rs:18,64`. - **Storage worker** (requires SERVER_OPS broadcast handle): - snapshot the watermark *first*, stream every row up to that - watermark via the `sync_since` path (`crates/storage/src/store.rs:289-347`, - `ORDER BY seq ASC`) — not the paginated `history()` path + snapshot the watermark *first*, then send every row matching the + watermark in one `WorkerResponse::SyncBatch` reply on `_willow_workers` + via the `sync_since` path (`crates/storage/src/store.rs:289-347`, + `ORDER BY seq ASC`, capped at `SYNC_BATCH_LIMIT = 10_000` rows per + reply) — not the paginated `history()` path (`store.rs:184-238`, `ORDER BY seq DESC`), which is for explicit - user-initiated history fetches. The streamed rows themselves still - ride the existing `WorkerResponse::SyncBatch` reply on - `_willow_workers`; the marker is emitted on `_willow_server_ops` - after the last row is sent so any client subscribed to SERVER_OPS - sees it. Events that arrive at the worker after the watermark was - taken are forwarded as ordinary gossip after the marker — they are - live, not historical. + user-initiated history fetches. The marker is emitted on + `_willow_server_ops` after the SyncBatch reply is sent so any client + subscribed to SERVER_OPS sees it. Events that arrive at the worker + after the watermark was taken are not part of this reply; once the + worker grows a path to forward them as ordinary gossip after the + marker (forward-looking — no such path exists in storage today), + they will be live, not historical. - **Peer-to-peer** (**conditional** on the per-author DAG sync wire protocol landing): `SyncMessage` is defined in `crates/state/src/sync.rs:37-44` but has zero producers or @@ -359,9 +366,12 @@ live events. everything in my frontier" moment; `HistorySyncComplete` is the wire encoding of that moment. **Blocks-on:** the peer-to-peer provider class enumerated in "Multiple providers" cannot ship until - this spec lands a `WireMessage` variant for `SyncMessage` and the - per-channel-topic handshake to drive it. The replay/storage worker - provider classes do **not** depend on this spec and can ship first. + `2026-04-01-per-author-merkle-dag-state-design.md` lands a + `WireMessage` variant for `SyncMessage` and the per-channel-topic + handshake to drive it. (This spec lands only + `WireMessage::HistorySyncComplete`; the `SyncMessage` wire-plumbing + is owned by the per-author DAG spec.) The replay/storage worker + provider classes do **not** depend on that work and can ship first. - `2026-04-12-state-authority-and-mutations.md` — markers do **not** flow through `apply_event`. They are not authority-bearing; they cannot grant, revoke, or mutate `ServerState`. A malicious marker @@ -385,9 +395,13 @@ live events. clients can cross-check against the "majority-agreed state" mechanism from `CLAUDE.md`'s trust model? This would unify history completion and state convergence into one signal. -4. Per-channel vs per-server scope: today `TopicId` is per-channel, - but a server-wide marker ("all channels have caught up") would - simplify the join flow. Do we want both? +4. Per-channel vs per-server scope: today the per-channel `TopicId` + is the unit of subscription a joining client sees historical chat + on (the same `TopicId` type is also used for SERVER_OPS, WORKERS, + PROFILES, and voice topics, but those are not what the UI's + "loading channel history" spinner gates on). A server-wide marker + ("all channels have caught up") would simplify the join flow. Do + we want both? 5. Is `stream_generation: u64` overkill? A random `u64` (e.g. from `ChaCha20Rng`) would avoid the "did I remember to bump it?" bug class, at the cost of not being orderable.