Skip to content

perf(parser): reuse DAT line buffers and avoid duplicate prefix allocations - #72

Merged
tgies merged 6 commits into
mainfrom
perf/weekly-import
Sep 11, 2026
Merged

tgies merged 6 commits into
mainfrom
perf/weekly-import

Conversation

@tgies

@tgies tgies commented Sep 11, 2026 •

Copy link
Copy Markdown
Owner

DAT parsing currently allocates every field just to recognize a record prefix and copies every raw line before parsing it. Inspect the borrowed prefix and reuse the line buffer, preserving the public parsed-line API, continuation handling, raw records and error behavior.

Three full-size comparisons reduced median fresh-import time from 279.90 to 256.28 seconds (8.4%). Including the previously merged removal of the counting pass, the controlled reduction is 14.5%. The production importer remains serial: both threaded prototypes regressed repeated fresh imports and spent more CPU. A single existing-database refresh pair favored threading, so that workload remains a candidate for repeated qualification. The receipts preserve both outcomes and reconstruct the prototypes without shipping their API.

The independent verifier accepted all 21 isolated fresh comparisons and five seeded refreshes, checking every stored field, primary key, sequence and schema object, with generated import timestamps validated separately. The exact final binary also matches both isolated references in full-sized fresh and seeded checks. The combined-index experiment saved another 61 seconds against its matching control; production index ownership remains separate work.

Validation: 697 workspace tests and three doctests on Rust 1.88, current-stable workspace Clippy, formatting, three comparison-oracle tests, and all code-checkpoint hosted checks pass. Regressions cover Unicode/CRLF/continuations, duplicate-record ordering, late malformed input, and callback-panic rollback/restoration. The existing transitive unty-next benchmark dependency moves to compatible 0.1.2 so the locked all-feature build works on Rust 1.88.

Full methodology, per-run measurements, source/binary hashes and limits are in docs/import-performance.md. This PR does not release a CLI version or deploy the service.

@codecov

codecov Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.43%. Comparing base (a47e551) to head (7b7d5a4).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #72      +/-   ##
==========================================
+ Coverage   97.35%   97.43%   +0.08%     
==========================================
  Files          58       58              
  Lines       13403    13406       +3     
==========================================
+ Hits        13048    13062      +14     
+ Misses        355      344      -11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tgies tgies changed the title perf(import): reduce parser allocations and qualify bounded parsing perf(parser): reuse DAT line buffers and avoid duplicate prefix allocations Sep 11, 2026
@tgies
tgies marked this pull request as ready for review September 11, 2026 21:03
@tgies
tgies merged commit d3571df into main Sep 11, 2026
11 checks passed
@tgies
tgies deleted the perf/weekly-import branch September 11, 2026 21:06
@tgies tgies mentioned this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant