ffi: create data source from memory buffer#8426
Conversation
Merging this PR will improve performance by 12.48%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | encode_varbin[(1000, 2)] |
176.7 µs | 157.1 µs | +12.48% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/ffi-read-from-buffer (8429479) with develop (2528dca)
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
AdamGS
left a comment
There was a problem hiding this comment.
Seems reasonable to me, but maybe needs someone with more C/FFI opinions.
0ax1
left a comment
There was a problem hiding this comment.
Couple of nits and questions.
a219a48 to
268bb06
Compare
268bb06 to
8429479
Compare
| vortex_ensure!(!buffer.is_null()); | ||
|
|
||
| let session = vx_session::as_ref(session); | ||
| let bytes: &'static [u8] = |
There was a problem hiding this comment.
this approach implies that the data is already in ram, but in ch read buffer is a streaming abstraction, including positional read api over s3/ssd etc. wouldn't it be more reasonable to exposeVortexReadAt to ffi?
There was a problem hiding this comment.
I agree, but this is a separate work which is non-trivial as opposed to this function. I have it on my TODO list. The issue with this is that we already have an inverse abstraction (duckdbfs) which we probably want to pull out.
Match rust API: buffer is borrowed under the condition it won't be modified
until data source is freed.
We need this for ClickHouse integration as it hands us memory buffers.