Skip to content

Implement zero-copy FFI transfer of batches from JVM to Native #2171

Closed

Description

@andygrove

What is the problem the feature request solves?

Comet doesn't currently follow Arrow best practice when transferring batches from JVM to native code. Ownership is retained by the JVM after batches have been made available in FFI, requiring a copy to be made on the native side.

It would be better to pass ownership of batches to native rather than make copies.

This design was necessary originally due to the use of mutable Parquet buffers in the native_comet Parquet scan.

Describe the potential solution

Once #2152 is implemented, and once we have removed the native_comet scan (#2177) and all uses of mutable Parquet buffers, we should be able to implement zero-copy FFI.

Additional context

No response

Activity

  1. jatin510 commented on Aug 17, 2025

    @jatin510
    Contributor

    i would love to work on this issue @andygrove

  2. andygrove commented on Aug 17, 2025

    @andygrove
    MemberAuthor
  3. added this to the 1.0.0 milestone on Sep 8, 2025
  4. self-assigned this
    on Mar 20, 2026
  5. modified the milestones: 1.0.0, 0.15.0 on Mar 20, 2026
  6. modified the milestones: 0.15.0, 0.16.0 on Apr 25, 2026
  7. andygrove commented on May 26, 2026

    @andygrove
    MemberAuthor

    this is implemented now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions