arrow-ipc StreamWriter better ergonomics with async

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'd like to use https://docs.rs/arrow-ipc/latest/arrow_ipc/writer/struct.StreamWriter.html in an asynchronous context. I need to implement std::io::Writer for it but it would be nice if I could give it something that e.g., implements the AsyncWrite trait

Describe the solution you'd like

Constructor with trait bounds W: AsyncWrite or StreamWriterAsync struct

Maybe we can follow a similar pattern to parquet where there is a separate async API

Note that you could also write to a buffer (Vec<u8> for example) and then send that buffer with async write

Definitely not as ergonomic as something built in

This came up in Datafusion: apache/datafusion#23247
I would actually like to propose a slightly different API here: a "sans IO" encoder -- specfically an encoder that does no IO

I recommend something like the BatchCoalecer API: https://docs.rs/arrow/latest/arrow/compute/struct.BatchCoalescer.html where the output is written to buffers

// Not there is no IO / stream:
let mut encoder = StreamEncoder::try_new_with_options(&schema, options).unwrap();
// Encode a batch to buffers
writer.encode(&batch1).unwrap();
// 
for buffer in writer.take_buffers() {
  // bufers is Vec<Vec<u8>> that has encoded IPC data and can be
  // written to a stream, etc
}

I think this could be a pretty thin wrapper around https://docs.rs/arrow-ipc/latest/arrow_ipc/writer/struct.IpcDataGenerator.html

@alamb

Hi @alamb . I open a PR to add the sans-IO stream encoder. Could you take a look when you have time? Thanks.

test(arrow-ipc): cover async stream encoding

feat(arrow-ipc): add sans-IO stream encoder

test(arrow-ipc): cover async stream encoding

feat(arrow-ipc): add sans-IO stream encoder

test(arrow-ipc): cover async stream encoding

perf(ipc): specialize message sinks

refactor(ipc): simplify message sink

feat(arrow-ipc): add sans-IO stream encoder

test(arrow-ipc): cover async stream encoding

perf(ipc): specialize message sinks

refactor(ipc): simplify message sink

feat(arrow-ipc): add sans-IO stream encoder (#10277)

Navigation Menu

Repository navigation

arrow-ipc StreamWriter better ergonomics with async #7812

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

arrow-ipc StreamWriter better ergonomics with async #7812

Description

Activity

alamb commented on Jun 28, 2025

alamb commented on Jul 25, 2025

alamb commented on Jul 1, 2026

Phoenix500526 commented on Jul 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions