Skip to content

feat: add udf executor using Row/RowBinary decoding and encoding - #494

Open
theelderbeever wants to merge 2 commits into
ClickHouse:mainfrom
theelderbeever:add-public-unvalidated-rowbinary-serde-helpers
Open

theelderbeever wants to merge 2 commits into
ClickHouse:mainfrom
theelderbeever:add-public-unvalidated-rowbinary-serde-helpers

Conversation

@theelderbeever

@theelderbeever theelderbeever commented Oct 17, 2026 •

Copy link
Copy Markdown

Summary

Adds a small isolated Executor and example which can be used to develop Native executable udf functions using the serde implementations for Row over RowBinary. Heavy inspiration taken from the Clickhouse blog.

Main motivation is additive functionality. Most of the runner could be written by hand case by case however the primary value is being able to reuse the RowBinary serde implementations for decoding the byte stream rather than needing to hand roll that portion. Alternatively, this could be satisfied by loosening the privacy bounds on the serialization and deserialization functions.

Regression test

Shown here using the executor vs the blog's count_token_rs example the data results were identical and the performance results were within a few percentage points. Sometimes faster and sometimes slower so I would say noise. I can provide the comparison results if desired.

Possible concerns

  • The serde parser may rescan the entire input buffer if the buffer wasn't full enough to parse a row fully. Worst case is when row sizes are very large or larger than the configure buffer around stdin. The docstring suggests increasing the buffer size. The blogpost also recommends this. This is mostly a product of trying to exclusively do borrowing. I wanted to try doing a serde streaming parser but it seemed like that would require significantly deeper changes to how the deserializer uses the buffer and would necessitate owned data rather than borrowed.
  • If a user configures their xml file saying the function takes (String, String) but the UDF input is Input { String } then the UDF will parse each argument as sequential rows. There is an error check for trailing bytes in the buffer and it will throw if this happens. But it only surfaces after the udf runs not at initial execution.

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided so that we can include it in CHANGELOG later
  • For significant changes, documentation in README and https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant