Repository navigation
Conversation
Summary: This commit is Peter Dillinger's change from meta#14675, carried as part 1/7 of the series adding the blog file format for WAL and blob files; see "Read blog-format WALs (7/7)" for the series overview. The description below is Peter's, from that pull request. Introduces the "blog" file format (portmanteau of "blob" + "log"), a new unified file format for WAL and blob files in RocksDB. This change makes the new format an opt-in option for blob files ONLY. An immediate follow-up will add WAL support. This format is intended to be the future default for both WAL and blob files, and likely also manifest files. The impetus for this new file format was an apparent convergence in requirements for interesting and useful future directions for RocksDB, along with some tech debt: * Supporting blob "direct write" (key-value separation in the memtable) with WAL enabled and at least the option to have all the WAL+blob data go into one file to reduce overheads in some cases like WAL sync write with blob direct write. (In other cases, separating WAL WriteBatches and blobs into distinct files would likely be the better choice.) The "preamble start" marker record is intended to support this case so that WriteBatches can carry external values in a "preamble" in memory and the WriteBatch doesn't need to be rewritten on storage to a single blog file serving both WAL and blob functions. (Details in later work.) * Preserve the continuity of each blob value for efficient reads (NOTE: WAL/Manifest format often breaks up payloads), and extend this continuity to WriteBatches so that keys/values with known checksums could be carried and extended to the WriteBatch and its contiguous encoding in the blog-as-WAL file. (The goal is to leverage checksums across layers as much as possible rather than computing new ones at each layer; only CRC checksums are "extendable.") * Support some "linear log" workloads with monotonically increasing keys and FIFO pruning of old data. A CF could be configured to use its own blog-as-WAL files writing this data, and those files could get indexing information written to them as each file is sealed. This would enable moderately efficient read queries that process WriteBatch records for results, and no WAL->SST write amplification. * Modernize blog and WAL formats with features like explicit versioning and extensibility, configurable and context-aware checksums, debugging and statistical information, customizable compression (CompressionManager aware), and more. New DB options: use_blog_format_for_blobs, blog_checksum. Other public API changes: * ChecksumType moved to include/rocksdb/checksum_type.h. * kStreamingCompressionSentinel (0x7F) added to CompressionType enum. Some included refactoring: * BlobLogWriter::log_number_ removed (was unused). * BlobLogWriter::AppendFooter renamed to LegacyAppendFooterAndClose. Other notable behavior changes to support the implementation: * Extend blob metadata in MANIFEST: per-file schema version, file identity, and file size, used for enhanced integrity checking and distinguishing blog-as-blob files from legacy blob files for accurate accounting purposes such as GC. (The file size is now generated for all blob files, in an ignorable forward-compatible way; other info is only present for blog-as-blob files.) * Partial blob_dump / ldb dump --path=... support for blog-format blob files. (Ideally we would move away from distinct binaries such as blob_dump and consolidate under 1 or 2 data inspection tools.) Differential Revision: D122185433
Summary: Part of a series adding the blog file format for WAL and blob files; see "Read blog-format WALs (7/7)" for the motivation and overview. Blob files are now verified against their MANIFEST metadata (size, format, schema version, and file identity) when opened, and blob reads check that a reference falls within the file before issuing I/O. Fixes several issues in blog blob files from 1/7, including `use_fsync` being ignored when sealing, missing whole-file checksums on direct-write files, and lazy reads ignoring a column family's custom compression manager, plus a race between iterator creation and `SetOptions`. Blog WAL escape sequences are bound to the WAL's log number so a recycled WAL can be recognized from its header, and the spec is updated to match. Differential Revision: D122185434
|
@kl-chou has exported this pull request. If you are a Meta employee, you can view the originating Diff in D122185434. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Part of a series adding the blog file format for WAL and blob files; see "Read blog-format WALs (7/7)" for the motivation and overview.
Blob files are now verified against their MANIFEST metadata (size, format, schema version, and file identity) when opened, and blob reads check that a reference falls within the file before issuing I/O. Fixes several issues in blog blob files from 1/7, including
use_fsyncbeing ignored when sealing, missing whole-file checksums on direct-write files, and lazy reads ignoring a column family's custom compression manager, plus a race between iterator creation andSetOptions. Blog WAL escape sequences are bound to the WAL's log number so a recycled WAL can be recognized from its header, and the spec is updated to match.Differential Revision: D122185434