Real-time streaming support

[maronavenue]

Is your feature request related to a problem or challenge?

Good day. Are there any plans to support real-time streaming of Arrow record batches? The use case I imagine would be that we could setup an Arrow Flight client on our side that will receive and process the stream from the Session Context upon executing the query. Otherwise, are there any recommendations that could enable this workflow?

Thanks as always.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

Hi @maronavenue --

I think several people use DataFusion for this - I think https://docs.rs/datafusion/latest/datafusion/datasource/streaming/struct.StreamingTable.html gives

@matthewmturner recently added an example in https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/file_stream_provider.rs in #10600

I think the documentation could still be improved, so if you have any time to help out that would be most appreciated 🙏

[maronavenue]

@alamb - Sorry it took a while, but I wanted to say thanks for sharing the above code references. I am still in the early stages of learning and discovering more things with DataFusion, but I'm more than happy to contribute.

[alamb]

Thanks @maronavenue -- all improvements to the documentation would be most welcome. I personally find when I try to explain something makes me undertand it.

Also, if you make a draft PR, even if it has gaps / TODOs, we can often find time to help fill them in.

[maronavenue]

@alamb - Took my first stab at #12379. I have started looking at the docs setup. Hoping to contribute meaningfully on that front soon.

Will keep this open for potential documentation.

[alamb]

We are now tracking streaming in a larger epic:

• [EPIC] Better support / documentation / tests for Streaming SQL Engines #24265