HDDS-16354. Make the dfsrw read/write ratio configurable - #11219
HDDS-16354. Make the dfsrw read/write ratio configurable#11219ermahesh wants to merge 2 commits into
Conversation
The dfsrw workload paired one read with every write, pinning the mix at 1:1. Real workloads are rarely balanced, so --read-write-ratio now sets how many reads a write is followed by: 4 reads back each write four times, 0.25 reads back every fourth write, and the default of 1 leaves the workload as it was. Reads stay inside the thread that wrote the file, so a read still validates the CRC32 of the latest write of the path it reads, and an overwritten path returning older bytes is still detected. A ratio that is not a whole number is accumulated per thread rather than rounded on every write, otherwise every ratio below 0.5 would read back nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| defaultValue = "10000") | ||
| private int maxFilesPerThread; | ||
|
|
||
| @Option(names = {"--read-write-ratio"}, |
There was a problem hiding this comment.
Would --reads-per-write be clearer here? Also, "four read-backs of each write" suggests the newly written file is read four times, while each read actually selects a random file from the thread’s history.
Perhaps we could describe this as "four validation reads per write". What do you think?
| * last of the three reads. | ||
| */ | ||
| @Test | ||
| void everyReadOfATaskValidatesContent() { |
There was a problem hiding this comment.
Currently, I think this only validates the last read. Could we parameterize the corruption point to 1, 2, and 3, and verify the checksum-mismatch failure?
| /** | ||
| * Passes the wrapped stream through, with the first byte of the file flipped. | ||
| */ | ||
| private static final class FlippingInputStream extends InputStream |
There was a problem hiding this comment.
nit: Could FlippingInputStream extend FSInputStream? It already provides the positioned-read plumbing, so the fixture could avoid several forwarding methods.
| * n-th write rather than never reading at all. | ||
| */ | ||
| private int readsDue(double readsPerWrite) { | ||
| readCredit += readsPerWrite; |
There was a problem hiding this comment.
Could we avoid accumulating the ratio as a double? For example, adding 0.1 ten times produces 0.9999999999999999, so the expected read is delayed until the eleventh write. An exact decimal or fixed-point representation would avoid this per-thread drift.
|
cc @yandrey321 if you’d like to take a look. This follow-up implements the configurable write/read ratio you suggested here #10651 (comment). Thanks! |
…ctly - Rename --read-write-ratio to --reads-per-write, and describe the reads as validation reads that each pick a file the thread wrote at random, rather than as read-backs of the write just made. - Carry the ratio in whole millionths of a read instead of accumulating it as a double, which drifted: adding 0.1 ten times gives 0.9999999999999999, so the read the tenth write was due slipped to an eleventh. The option is now a BigDecimal, which also removes the NaN and infinity checks. - Parameterize the corruption point of everyReadOfATaskValidatesContent over the first, second and third read, so it no longer covers only the last. - Extend FlippingInputStream from FSInputStream, which supplies the positioned reads and drops three forwarding methods. Adds carriesFractionalRatioWithoutDrift, which fails on the old accumulator.
| + "files a task validates, not which write it validates. A value that is not a whole number is spread " | ||
| + "over the writes of a thread rather than rounded on every one of them.", | ||
| defaultValue = "1.0") | ||
| private BigDecimal readsPerWrite; |
There was a problem hiding this comment.
why not read percent as a param? And then get a random number between 0 and 1 and if its less then read percent do the read, otherwise do the write.
3 participants
What changes were proposed in this pull request?
Follow-up from HDDS-14524.
The
dfsrw(dfs-read-write-validator) freon workload issues exactly one readafter every write, so its read/write mix is pinned at 1:1. Real workloads are
rarely balanced, and the benchmark is more useful when the mix can be tuned
toward read-heavy or write-heavy.
This PR adds a
--read-write-ratiooption that sets how many reads each write isfollowed by:
--read-write-ratio1.0(default)40.25The existing per-path CRC32 validation and stale-read detection are unaffected. A
read still picks a random file from the history of the thread that wrote it and
compares against the checksum of the most recent write of that path, so both
corruption and an overwritten path returning older bytes are still detected.
Why a ratio rather than a
--read-threads/--write-threadssplitA read-only thread would have no write history of its own, so it would have to
read paths owned by other threads. Those paths are overwritten concurrently,
which is precisely the situation the stale-read check treats as a failure.
Splitting threads would therefore have meant weakening the validation, so every
read is kept inside the thread that wrote the file.
Fractional ratios
A ratio that is not a whole number is accumulated per thread rather than rounded
on each write:
Rounding per write would collapse every ratio below 0.5 to zero reads, leaving a
"validator" that never validates. Carrying the remainder makes
0.25read backevery fourth write and
1.5alternate between one and two reads. At the defaultof
1.0the credit lands on exactly 1.0 every time, so the default path isunchanged.
Notes
-nkeeps its meaning: one task is one write, and the number of files writtenfor a given
-ndoes not change with the ratio.file-writeandfile-read-validatetimers are unchanged, so the twosides of the mix stay separately measurable.
the existing
--size,--bufferand--max-files-per-threadchecks.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16354
How was this patch tested?
New unit test
TestHadoopFsReadWriteRatioinhadoop-ozone/freon.The existing integration test
TestHadoopFsReadWriteValidatoris unchanged andcontinues to cover the default behaviour against a real cluster.