Skip to content

[OOC] IO Handler Performance Improvements and Generalization#2532

Open
janniklinde wants to merge 1 commit into
apache:mainfrom
janniklinde:ooc/planner-merge-3
Open

[OOC] IO Handler Performance Improvements and Generalization#2532
janniklinde wants to merge 1 commit into
apache:mainfrom
janniklinde:ooc/planner-merge-3

Conversation

@janniklinde

Copy link
Copy Markdown
Contributor

This PR improves performance and generality of OOCMatrixIOHandler. We introduce experimental input/output streams that worked better in practice for OOC spilling in local experiments. We also introduce a lightweight OOCFuture to replace CompletableFuture in hot paths.

@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 53.49794% with 226 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.60%. Comparing base (85ea875) to head (a0cbd53).

Files with missing lines Patch % Lines
...time/ooc/cache/io/OOCBufferedDataOutputStream.java 46.30% 73 Missing and 7 partials ⚠️
.../org/apache/sysds/runtime/ooc/cache/OOCFuture.java 33.89% 72 Missing and 6 partials ⚠️
...ntime/ooc/cache/io/OOCBufferedDataInputStream.java 72.72% 28 Missing and 11 partials ⚠️
...sysds/runtime/ooc/cache/io/OOCMatrixIOHandler.java 57.44% 13 Missing and 7 partials ⚠️
...me/instructions/spark/data/IndexedMatrixValue.java 66.66% 4 Missing and 1 partial ⚠️
.../runtime/ooc/cache/io/SpillableObjectRegistry.java 75.00% 1 Missing and 2 partials ⚠️
...he/sysds/runtime/ooc/cache/io/SpillableObject.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2532      +/-   ##
============================================
- Coverage     71.65%   71.60%   -0.05%     
- Complexity    49328    49395      +67     
============================================
  Files          1580     1585       +5     
  Lines        190516   190967     +451     
  Branches      37364    37424      +60     
============================================
+ Hits         136507   136746     +239     
- Misses        43486    43660     +174     
- Partials      10523    10561      +38     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant