Skip to content

regex: honor custom line terminators in the regex engine - #3542

Open
ArjunCodess wants to merge 1 commit into
BurntSushi:masterfrom
ArjunCodess:fix/null-data-fast-path
Open

ArjunCodess wants to merge 1 commit into
BurntSushi:masterfrom
ArjunCodess:fix/null-data-fast-path

Conversation

@ArjunCodess

Copy link
Copy Markdown

--null-data --no-mmap was going through the slow line-by-line searcher. therefore a file of nuls with one needle in the middle was walking every nul as its own empty record.

grep-regex never told the regex engine about the configured terminator, and hence (?m:^) / (?m:$) only understood \n. the searcher worked around that by disabling the fast path when the terminator was nul. actually that workaround is what made the search so slow.

i passed the terminator through to regex-automata. --null-data still means nul-separated records, and --line-regexp still matches a whole record. a grep-searcher test counts matcher calls on a nul-heavy haystack so we can see the fast path is in use.

Fixes #2832

The searcher's fast path searches a whole buffer at once, so `(?m:^)`
and `(?m:$)` need the regex engine to treat NUL as a line boundary when
`--null-data` is used. Without that, the searcher fell back to walking
every NUL as its own empty record.

Fixes BurntSushi#2832
@ArjunCodess

Copy link
Copy Markdown
Author

@BurntSushi the custom line terminator change is ready whenever you want to look.

@ArjunCodess

Copy link
Copy Markdown
Author

@BurntSushi still happy to adjust the custom line terminator change if you want it shaped differently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--null-data seems to inhibit some optimizations

1 participant