diff --git a/pages/fundamentals/data-durability.mdx b/pages/fundamentals/data-durability.mdx index 449dfc644..46a8410dc 100644 --- a/pages/fundamentals/data-durability.mdx +++ b/pages/fundamentals/data-durability.mdx @@ -103,6 +103,11 @@ Per-transaction WAL checksums were introduced in Memgraph v3.12. WAL files written by older versions do not contain checksums and are recovered without integrity verification. Checksum verification applies only to WAL files written by Memgraph v3.12 or newer. + +Before Memgraph v3.13, a checksum mismatch anywhere in the WAL chain recovered +the readable prefix of that file and then continued with the files after it, +which could bring the database up with an incomplete dataset without reporting +it. Since v3.13, damage in a completed WAL file fails recovery instead. To guard against silent on-disk corruption, Memgraph protects WAL files with @@ -113,16 +118,37 @@ To guard against silent on-disk corruption, Memgraph protects WAL files with - **Each transaction** is protected by a 4-byte checksum covering the transaction's bytes (transaction start, deltas and transaction end). -Checksums are verified automatically during recovery. If a transaction's stored -checksum does not match the recomputed value, the WAL is considered corrupted at -that point and recovery stops, so corrupted data is never applied to the -database. A mismatch in the WAL header causes recovery from that file to fail. +Checksums are verified automatically during recovery, so corrupted data is never +applied to the database. A mismatch in the WAL header causes recovery from that +file to fail. What happens on a transaction checksum mismatch depends on whether +the file was completed: + +- A **completed (finalized) WAL file** — one Memgraph finished writing and + rotated away from — was flushed to disk before it was closed, so every + transaction in it is durable and was already acknowledged. A mismatch there + means the bytes on disk rotted, and recovery fails rather than continuing with + part of the file. Recovering only a prefix would be unsound, because the WAL + files that follow were written on top of the transactions that went missing. +- A **WAL file that was still being written** when the process stopped may + legitimately have a torn tail. Recovery applies its transactions up to the + last whole one and stops there, which is how an interrupted write degrades + gracefully. + +Files that the newest snapshot already covers are skipped without being read, so +corruption in those is harmless. If a WAL file written after the newest snapshot +is damaged, the database fails to recover; use +[`--storage-allow-recovery-failure`](/database-management/configuration) together +with `RECOVER SNAPSHOT` (see [recovery failure +handling](#recovery-failure-handling)), or restore from a backup. The same checksums protect WAL files that are buffered on disk on a replica before being applied, so corruption introduced between the main and the replica -is detected before the data is committed. Deltas streamed during the commit -(`PrepareCommitRpc`) are not checksummed because the TCP transport already -provides integrity guarantees. +is detected before the data is committed. When a replica hits damage in a +finalized file it reports the failure and stops applying the rest of the batch +instead of skipping ahead; the in-flight transaction is aborted, main does not +advance its view of the replica, and the transfer is retried later. Deltas +streamed during the commit (`PrepareCommitRpc`) are not checksummed because the +TCP transport already provides integrity guarantees. Snapshots are not yet protected by checksums. diff --git a/pages/release-notes.mdx b/pages/release-notes.mdx index fdb026d2c..43dacaa9b 100644 --- a/pages/release-notes.mdx +++ b/pages/release-notes.mdx @@ -133,6 +133,19 @@ guide. - Fixed a crash when multiple sessions printed query plans at the same time (for example concurrent `EXPLAIN` / `PROFILE`, or query-plan logging). [#4489](https://github.com/memgraph/memgraph/pull/4489) +- Fixed silent data loss when a WAL file other than the last one in the chain + was damaged. Recovery used to apply the readable prefix of that file and then + apply the following files in full, coming up with an incomplete dataset + without reporting anything; on a replica the divergence never healed because + the replica reported the later timestamp and main considered it caught up. + Damage in a completed (finalized) WAL file is now fatal: the database fails + recovery instead of starting stale, and a replica stops applying the rest of + the chain. A WAL file that was still being written when the process crashed + is unaffected and is still truncated to its last whole transaction. Recover + such an instance with + [`--storage-allow-recovery-failure`](/database-management/configuration) plus + `RECOVER SNAPSHOT`, or from a backup. + [#4528](https://github.com/memgraph/memgraph/pull/4528) - `TERMINATE TRANSACTIONS` no longer reports `killed: true` for a transaction the caller is not authorized to terminate. Such a match now reports `killed: false`, indistinguishable from an id that does not exist. @@ -163,6 +176,12 @@ guide. statements). Under index-heavy workloads this removes GC-correlated latency spikes on index and constraint creation. [#4468](https://github.com/memgraph/memgraph/pull/4468) +- Recovery from WAL files is roughly 35% faster. Each WAL file now records its + timestamp range and transaction count in its header, so deciding which files + to recover from no longer requires parsing every file end to end, and + replaying a file parses it once instead of twice. This also speeds up replica + recovery and the WAL cleanup that runs after every snapshot. + [#4528](https://github.com/memgraph/memgraph/pull/4528) - Cluster management queries forwarded from a follower coordinator to the leader now have [explicit RPC timeouts](/clustering/high-availability/how-high-availability-works#rpc-timeouts),