Skip to content

Unify Python table serialization through Rust table state #3472

Description

@Xuanwo

Follow-up from #3432.

Python currently reconstructs local / namespace / remote connections using Python-side connection state. This is enough to make remote tables usable in PyTorch multiprocessing, but table reopen state is now split across Python wrappers and connection classes.

It would be cleaner to expose table / connection serialization primitives from the Rust layer and have Table, LanceTable, and RemoteTable use the same state format for pickle reopen. This would reduce duplicated Python logic and make native / remote table behavior easier to keep consistent.

Suggested scope:

  • Define the minimal Rust-side table reopen state needed for local and remote tables.
  • Expose it through the Python binding.
  • Make Python table pickle support delegate to this shared primitive where possible.
  • Keep non-serializable Python-only hooks such as custom header providers as explicit unsupported cases or worker-side factory hooks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions