Follow-up from #3432.
Python currently reconstructs local / namespace / remote connections using Python-side connection state. This is enough to make remote tables usable in PyTorch multiprocessing, but table reopen state is now split across Python wrappers and connection classes.
It would be cleaner to expose table / connection serialization primitives from the Rust layer and have Table, LanceTable, and RemoteTable use the same state format for pickle reopen. This would reduce duplicated Python logic and make native / remote table behavior easier to keep consistent.
Suggested scope:
- Define the minimal Rust-side table reopen state needed for local and remote tables.
- Expose it through the Python binding.
- Make Python table pickle support delegate to this shared primitive where possible.
- Keep non-serializable Python-only hooks such as custom header providers as explicit unsupported cases or worker-side factory hooks.
Follow-up from #3432.
Python currently reconstructs local / namespace / remote connections using Python-side connection state. This is enough to make remote tables usable in PyTorch multiprocessing, but table reopen state is now split across Python wrappers and connection classes.
It would be cleaner to expose table / connection serialization primitives from the Rust layer and have
Table,LanceTable, andRemoteTableuse the same state format for pickle reopen. This would reduce duplicated Python logic and make native / remote table behavior easier to keep consistent.Suggested scope: