Skip to content

Avoid unecessary copies in Arrow IPC reader #2437

@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The Arrow IPC format is designed to avoid memory copies when moving data from one implementation to another. However, as noted by @tustvold on #2369 (comment), the arrow-rs ipc reader implementation is actually copying data unnecessarily

Describe the solution you'd like
In the ipc code, create a Buffer initially and rewrite the ipc implementation in terms of Buffer rather than &[u8] (as the final output needs to be in a Buffer

Describe alternatives you've considered
N/A

Additional context

Came up in the context of #2369

Possibly also related to #189

Metadata

Metadata

Assignees

Labels

arrowChanges to the arrow crateenhancementAny new improvement worthy of a entry in the changeloggood first issueGood for newcomersperformance

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions