Repository navigation
Adopt the dual-clock record model on RocksDB: first word = transaction timestamp/log key, source/origin version in a distinct second word (stages 0b and 2) #2412
Description
Direction (rewritten 2026-09-02)
Continues from #2409 (stage 0a, merged). The model, settled after rocksdb-js#813 (native commit-time stamping) was closed unmerged:
| Word | Meaning | Assigned by | Uniqueness |
|---|---|---|---|
First word (localTime) |
Write identity: verification-table freshness, record→log-entry lookup, subscription staleness, transaction grouping | rocksdb-js startTimestamp at transaction construction. A replication receiver or crash replay adopts the origin's first word via setTimestamp; no other caller may. |
Unique within one origin's log. Identity is (nodeId, first word); records already carry nodeId. |
Second word (version, under HAS_DISTINCT_VERSION_FLAG) |
Record version: LWW in precedesExistingVersion, CRDT ordering, @updatedTime/@createdTime, ETag/Last-Modified, replication |
Source lastModified (capped per #2065) or the origin's version. Absent ⇒ version == first word (today's encoding). |
Legitimately non-unique |
| Log batch key | Resume cursor in the per-origin log | Equals the first word of every record in the batch | Per-origin log |
Invariant: a record's first word equals the batch key in the origin node's log. auditStore.get(firstWord, tableId, id, nodeId) (Table.ts ~2737/2801/2822 → RocksTransactionLogStore.ts ~171) depends on it. No native commit-time stamping (rocksdb-js#811 is rescoped to a read surface, setTimestamp hardening and a clock-floor seed). Per-node origin-clock logs and the resume-cursor model are unchanged; the previous plan's log-key-domain flip and per-hop cursors are gone.
Stage 0b — normalize to LMDB semantics (no format change)
- On RocksDB,
AuditRecord.version= record version andlocalTime= log key, the roles LMDB already has; the Deliver RocksDB subscription events for source fills whose version differs from the log key #2409recordVersionalias gets absorbed. - Coordinated harper-pro PR (harper-pro#790): the receiver sets the apply transaction's timestamp from
localTime(the origin's first word) and appliesversionper write; cursor sites move tolocalTime. - Trap to pin: under normalization
removeAuditEntry's tombstone compare and the blob orphan scan would start matching record versions. A version match is not write identity; both must be pinned to alocalTimecompare in the same change. event.versiondelivered to subscribers:version= record version,localTime= log position, pinned by a transport-level test (ruling D3).
Stage 2 — distinct-version format
- Encoder writes the first word =
transaction.timestampalways (restoring first word == log key for fills, which Fix sourcedFrom cache-fill conflict convergence #2065 broke and Deliver RocksDB subscription events for source fills whose version differs from the log key #2409 papered over on the read side), and an 8-byte second word underHAS_DISTINCT_VERSION_FLAGonly when version ≠ first word. Old records decode as version == first word; lazy migration, no bulk rewrite. - Call-site audit: every
precedesExistingVersion(...)andauditStore.get(...)argument sorted into identity (first word) vs ordering (version). Known so far:previousVersionis stored fromexistingEntry.versionon RocksDB (RecordEncoder.ts~1019) and must become the first word or the out-of-order walk (Table.ts~2836) breaks as soon as a distinct version exists;additionalAuditRefs[].versionstays a version; the incoming write's ordering value is its version (its transaction timestamp unless a source/origin supplied one), compared against the existingversion. @updatedTime(Table.ts~2537) is stamped from the version, not the transaction timestamp. ETag/Last-Modified already readentry.versionviacontext.lastModified; verify, don't change.VERSION_NOT_UNIQUE_FLAGstays (Test #766 is not retired). Re-key its producer (RecordEncoder.ts~886, which compares versions today) to first-word equality: set when the new first word ≤ the existing first word, which is the cross-origin same-millisecond tie-break rewrite. Deletes stay excluded as today.- Remove
timestampfrom the publicContext(ResourceInterface.ts~127, undocumented). Keep an internal-only carrier for the receiver (Table.ts~1046) and replay (replayLogs.ts~235); the fill token (Table.ts~6221) then only ever sees the transaction's own timestamp, closing Fix sourcedFrom cache-fill conflict convergence #2065's open item on unvalidated caller timestamps. - Fix sourcedFrom cache-fill conflict convergence #2065's
max(token, now)cap on source versions stays: a future-dated version still wins LWW against every later local write. - Two REQUIRED gates, format-only now: a peer gate (second-word bytes must not reach peers below the format floor; no protocol negotiation exists today) and a local downgrade gate (a durable in-store format floor written before or atomically with the first two-word record; package-version prompting cannot teach an already-released binary to refuse the store).
- Write-identity blocker unchanged: same-key writes chained in one transaction share a first word. Prove they collapse to one durable mutation or add an intra-transaction ordinal before any cleanup migrates to first-word identity.
- Corrupt-value outcomes are context-specific (drop/disconnect for peer input; skip publication where safe; controlled recovery failure for WAL-only state), validated per identity at its true ingress point.
Dependencies
rocksdb-js#811 (rescoped), harper-pro#790, harper#2411 (replay applies version per write; first word via setTimestamp).
Activity
kriszyp commentedon Aug 31, 2026
Companions: rocksdb-js stamping (stage 1) HarperFast/rocksdb-js#811; harper-pro fill-version divergence (stage 0b coordinated PR) HarperFast/harper-pro#790; replay version fidelity #2411.
[-]Adopt the dual-clock record model on RocksDB: version/localTime normalization (stage 0b) and the distinct-version format (stage 2)[/-][+]Adopt the dual-clock record model on RocksDB: first word = transaction timestamp/log key, source/origin version in a distinct second word (stages 0b and 2)[/+]kriszyp commentedon Sep 3, 2026
Flagging a dependent that has just been sequenced behind this work, so it can be weighed while the
identity model is still being settled: #2485 (concurrent multi-path deliveries of one write each
persist a transaction-log entry).
Its fix needs exactly the pair this issue makes authoritative — origin nodeId plus the origin
transaction's first word — as a claim key, so that N concurrent deliveries of one origin transaction
are single-flighted before any of them stages an append. A repro is committed on
fix/concurrent-duplicate-apply-audit-entries: 15 concurrent identical applies leave 15 entries, andthree concurrent deliveries of a two-write same-key transaction leave 6 where one delivery leaves 2.
The full spec, including two shapes that look simpler and are wrong, is in
this comment.
Two things from tracing #2485 that bear on decisions here:
transaction share the first word, and a duplicated delivery of such a transaction is the case that
breaks any per-write identity guard in both directions. The repro's third case pins it.
promise. In rocksdb-js 2.8 the log batch is written in the log lane before the RocksDB commit and
is published by whichever commit attempt succeeds;
committedPositionsurvivesresetTransaction()and
addLogEntryis write-once per handle. So a losing attempt's append is durable, and a processthat dies between a winner's append and its commit leaves an orphan that a post-restart redelivery
reads as proof of application. If a crash-safe "one entry per identity" is wanted, it needs the log
batch to participate in the commit outcome, which is a rocksdb-js change and would belong with K6 load test scenarios for blob storage (insert, update, read, expiry, concurrency) #811
rather than being bolted on later.
No action needed on this issue's own scope; raising it so the sequencing is visible from this side too.
— Claude Fable 5.1
Report the record version and the log key as separate clocks on Rocks…
Expose record version and txnLogKey as separate audit clocks (#2497)
Start the CRDT reverse walk from the record's audit-log key, not its …
Merge main; read the floor's cursor from txnLogKey, not localTime
kriszyp commentedon Oct 7, 2026
Confirmed on current main (4cceb72): a RocksDB transaction's writes are ordered by its start time, LMDB's by commit time, so the two engines disagree on which write wins when a transaction stays open across another commit.
resources/DatabaseTransaction.ts:1489stampsthis.timestampfromtransaction.getTimestamp(), rocksdb-js'sstartTimestamp(set at construction).resources/LMDBTransaction.ts:171stamps insidecommit().precedesExistingVersion(resources/Table.ts, write paths at ~3760/3824/4718) compares that stamp directly againstexistingEntry.version, so on RocksDB the held-open transaction's writes lose to any write that commits while it is open, although its request returned 200.Repro is on main: #2820's arm 3 (
integrationTests/database/same-txn-write-chain.test.ts). LMDB commits{status: DONE, seq: 2}3/3; RocksDB commits{status: WRITER_0, seq: 1}3/3. That test's comment already names this as engine ordering and accepts either winner.This issue's plan as written does not change it: the second word is set from a source
lastModifiedor the origin's version, and "absent ⇒ version == first word", so a plain local write keeps the construction-time stamp as its LWW version.recordLock.tsnextHolderVersion()carries aTODO(#2412)for the same gap on lock-ordered stamps.Question: is start-time ordering the intended end state for local RocksDB writes? If yes, it should be stated here and in the docs, since nothing currently says the engines can pick different winners. If not, the fix probably belongs in this issue as a local commit-time second word.
— Claude Opus 5.5 (finding-triage)