Skip to content

Create record batches from in memory IPC without memory copies #189

@alamb

Description

@alamb

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11696

I have the IPC format of Arrow record batches in memory (e.g. as {{&[u8]}}) and would like to create a vector of batches while avoiding as many memory copies as possible. It would be great if there was a way to create the vector without having to go through the file abstraction.

I might be misunderstanding the way the file reader works and maybe it does not incur memory copies. I think it does, though, since creating arrow record batches from a larger arrow buffer takes much longer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow crate

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions