fix(csv): sniff dialect after leading empty lines - #4467
Merged
PeterStaar-IBM merged 1 commit intoOct 1, 2026
Merged
PeterStaar-IBM merged 1 commit into
PeterStaar-IBM merged 1 commit into
Conversation
Start the bounded fallback sample at the first nonempty physical line without changing the input passed to csv.reader. Assisted-by: OpenAI GPT-6.1-Sol <noreply@openai.com> Signed-off-by: sergioperezcheco <checo520@outlook.com>
Contributor
|
✅ DCO Check Passed Thanks @sergioperezcheco, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4466.
Delimiter detection
An empty first physical line sends detection straight to the larger sample. For ragged records,
csv.Sniffercannot find a consistent delimiter count in that sample and the backend falls back to comma. The same file without the leading empty line is correctly detected from its header.Skip genuinely empty physical lines before sniffing, and start the lazy 4096-character fallback window at the first nonempty line. Parsing still starts at offset zero. Empty-field records, quoted empty fields and whitespace-only fields are not skipped, and existing blank-record filtering remains unchanged. Starting the fallback window at the header also prevents a long prefix of empty lines from exhausting it before a multiline quoted field is reached.
This complements #4316, which filters empty records after parsing, and retains the first-line-first detection strategy introduced in #3985.
Regression coverage
35 added public
DocumentConvertercases cover Path and stream inputs, all five supported delimiters, LF/CRLF prefixes, multiline quoted headers after 4096 empty lines, and preservation of nonempty first records. On the original implementation, these cases produce 25 failures and 10 passes. The complete CSV suite now passes all 52 cases; existing reference outputs are unchanged.make validatepasses all applicable project hooks, including Ruff, ty, Tach and module coverage. No full non-CSV test suite was run.Implementation, investigation and local validation used AI assistance through Hermes with OpenAI GPT-6.1-Sol.