Split Components iterator to prefixed and non-prefixed versions and optimize Components for non-prefix path based platforms#156496
Conversation
rustbot
added
S-waiting-on-review
|
rustbot has assigned @Mark-Simulacrum. Use Why was this reviewer chosen?The reviewer was selected based on:
|
This comment has been minimized.
This comment has been minimized.
1627e2f to
33e69e1
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
33e69e1 to
ed9d33d
Compare
This comment has been minimized.
This comment has been minimized.
ed9d33d to
0b0f84c
Compare
This comment has been minimized.
This comment has been minimized.
0b0f84c to
8ed33ea
Compare
This comment was marked as outdated.
This comment was marked as outdated.
2151b8f to
83cdbed
Compare
This comment was marked as outdated.
This comment was marked as outdated.
83cdbed to
3921fff
Compare
rustbot
added
S-waiting-on-author
0a25dda to
92e0132
Compare
|
New benchmarking results. You can see what the benchmark code looks like here and run it yourself to see if there are any difference in measurements on your end: This is the measurement of the current implementation of This is the measurement of the new implementation of Edit: Updated |
|
Here are the benchmark results with black box: From current From this Edit: Updated Edit 2: Took off Path ordering benchmark here since it was incorrect see below to see corrected path ordering benchmarks. |
92e0132 to
574d7f2
Compare
rustbot
added
S-waiting-on-review
|
I'm confident this code works (passed CI in previous run, the current amended commit change I made doesn't change logic, but makes the code written in a more idiomatic way). In my opinion, the logic in this code should look more readable than how |
|
@rustbot label +I-libs-nominated Since |
rustbot
added
the
I-libs-nominated
I think I'm having a hard time seeing how this would work with the impl PartialEq for Path {
#[inline]
fn eq(&self, other: &Path) -> bool {
// Identical byte sequences always produce identical component sequences,
// so skip the more expensive component-wise comparison in that case.
self.as_os_str() == other.as_os_str() || Iterator::eq(self.components().rev(), other.components().rev())
}
}
(same thing for PathBuf)Because That kinda also means that the current |
…yte equality and Iterator::eq directly. Co-authored-by: =?UTF-8?q?=E7=A6=BE=E5=8F=AF?= <chengkelfan@qq.com>
c10aaf6 to
56a6c63
Compare
|
This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
|
@clarfonthey, there's something else I realized with this PR This would reduce the What do you think about this? |
This comment has been minimized.
This comment has been minimized.
…path check to Components/Path equality by finding first mismatched byte, separated Components iterator into different versions and re-exported them accordingly based on target os. This also ends up removing HAS_PREFIXES because it becomes unused.
2c07391 to
1153b10
Compare
|
@clarfonthey You might like the results of this And with bringing this back to doing the same subslicing approach, but written in a more simplified way and normalizing bytes upfront, as far as I can tell, we pretty much have the same performance for |
This comment has been minimized.
This comment has been minimized.
Redesigned `Components` iterator to use front and back indexing instead mutating and subslicing path field
|
@bors try jobs=dist-* Just in case any other errors pop up on other platforms. |
This comment has been minimized.
This comment has been minimized.
Redesigned `Components` iterator to use front and back indexing instead mutating and subslicing path field try-job: dist-*
rust-bors
Bot
added
S-waiting-on-author
|
A job failed! Check out the build log: (web) (plain enhanced) (plain) Click to see the possible cause of the failure (guessed by this bot) |
|
💔 Test for 8b548fb failed: CI. Failed job:
|
|
@bors try jobs=dist-* I'll try running it again, I didn't see the bors message on phone when I ran the try command without the jobs parameter, so I thought I ran the incorrect command without specifying a jobs parameter. |
This comment has been minimized.
This comment has been minimized.
Redesigned `Components` iterator to use front and back indexing instead mutating and subslicing path field try-job: dist-*
|
💔 Test for 9f2eefd failed: CI. Failed job:
|
|
A job failed! Check out the build log: (web) (plain enhanced) (plain) Click to see the possible cause of the failure (guessed by this bot) |
|
@bors try Let's try this. |
This comment has been minimized.
This comment has been minimized.
Redesigned `Components` iterator to use front and back indexing instead mutating and subslicing path field
asder8215
changed the title
Components iterator to use front and back indexing instead mutating and subslicing path fieldComponents iterator to prefixed and non-prefixed versions and optimize Components for non-prefix path based platforms
8 participants
View all comments
This PR entirely changes how
Components<'_>is implemented.Currently, theThisComponents<'_>iterator 'consumes' components through mutating its path field to a subslice that presents the left over unconsumed path components (this consumed path component is what's returned inComponents::nextorComponents::next_back). However, this PR keeps the path field alive/unmodified and uses front and back indexing strategy to extract consumed/unconsumed components.Components<'_>implementation still uses a subslicing approach similar to the original implementation, but it's highly optimized for non-Windows platform, given that there's noPrefix<'_>components to check with.Ideally, at least from a few benchmarking test I've done locally, this
Components<'_>implementation should perform similarly with the original one when it comes toComponents::next/Componentsordering comparisons (the original one was already highly optimized as is). Where this non-Windows implementation shine though is improvingComponents::as_path(as a result certain things benefit from this likeAncestorsiterator,std::fs::create_dir_all, etc.) andComponents::next_back(and as a resultPathequality).There are a lot of different things I've done to make sure that
Componentsequality is improved dramatically. For example, just like whatComponentscomparison does with checking if we see a mismatched byte in the forward direction and setting ourComponentsiterator to the nearest separator, we do the same thing withComponentsequality in the opposite direction (this actually is one of the significant reason why this implementation ofComponentsequality is really, really faster now). Additionally, there are other optimizations I do here, like doing less boolean checks or usize comparisons withinComponents::next_backand normalizing any separator/curr directory bytes upfront (the original implementation had a lot of conditional checks that could be way more simplified; the prefix checking also didn't benefit non-Windows platform). With the way I haveStateenum implemented in thisComponentsiterator, I can do other small optimizations like checking for empty paths, so that I can returntrueorfalseimmediately.Because the implementation I'm going for is specific to non-Windows/non-prefix-supporting platforms, I chose to split the
Componentsiterator implementation to two different version that is re-exported accordingly to individual platforms (this leads toHAS_PREFIXESconstant being redundant and unnecessary). I'm open to discussion on how to organize the code here better, naming stuff, etc. Some prefix-specific things have been refactored to be exclusively on platforms that utilize prefix components.So far, I've been benchmarking it locally. I formerly did it with Criterion, but now I've just been benchmarking manually through hyperfine. What I've been testing was how fast can this implementation of
Components::next/Components::next_backrun paths like "a1..aN/a1..aN" vs "b/a1..aN/a1..aN/" (where N is some number of a's I put for each component, and you can assume that there are some X path components in the path); I also played around with the mismatching component placed at the end of the path versus beginning, and benchmarked running the same path equality/comparisons 10000 times to see the result cumulatively. You can try hyperfining the code in this repo; I'll get around to consolidating those benchmarks in Criterion a bit later and then posting results in the comments below.