Skip to content

TIKA-4808 - improve in-memory pipes handling - #3028

Merged
tballison merged 2 commits into
mainfrom
TIKA-4808-improve-in-memory-inputstream-handling
Aug 17, 2026
Merged

TIKA-4808 - improve in-memory pipes handling#3028
tballison merged 2 commits into
mainfrom
TIKA-4808-improve-in-memory-inputstream-handling

Conversation

@tballison

Copy link
Copy Markdown
Contributor

This adds a helper fetcher that allows for direct passing of bytes from pipes client to pipes server without the usual fetcher from a static source.

This breaks the point of pipes, but it has a use, and it allows users with byte[] to still get the process isolation of pipes.

@tballison

Copy link
Copy Markdown
Contributor Author

cc @dpol1 this will be useful in grpc for stormcrawler.

@dpol1

dpol1 commented Aug 17, 2026

Copy link
Copy Markdown
Member

oh nice - this is the seam I was faking on the 4795 branch with a spool file and a fake fetcher. with this in core I can drop that bridge entirely and the ticket gets to be about the thing that actually matters: same bytes, parse profile and semantic hints -> same canonical Document, whichever door the bytes come in through. thanks for the cc - I'll rebase onto this once it lands.

three small edges flagged by claude:

  • maxInlineBytes/maxIpcPayloadBytes: the pair check lives in the setter, so lowering only maxIpcPayloadBytes (say 5m) leaves the 10m inline default unvalidated, and {inline: 95m, ipc: 200m} passes or fails depending on json key order. could the pair be checked after deserialization? server and grpc deserialize PipesConfig directly, so load() alone wouldn't cover them.
  • PayloadRouter.route(): if the source throws after the spool file is created, no Routed exists yet, so nobody deletes the partial file - I reproduced a leftover this way.
  • PipesForkParser.parse(): the inline body goes into the caller's ParseContext and is never restored, so a later file-backed parse retains and re-serializes the old body. should it be restored in finally, making the payload request-owned?

@dpol1

dpol1 commented Aug 17, 2026

Copy link
Copy Markdown
Member

👍

@tballison
tballison merged commit 333dc48 into main Aug 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants