Follow-up from #3432.
RemoteTable now carries __getstate__ / __setstate__ so it can be reopened after pickle/spawn and fork. Permutation still contains local LanceTable serialization helpers for native and in-memory tables, which makes the ownership of table pickling asymmetric.
We should move local table pickle support onto LanceTable itself so code that needs a picklable table can rely on the table abstraction instead of duplicating table-specific reopen logic.
Suggested scope:
- Add
__getstate__ / __setstate__ to LanceTable for persisted local tables.
- Decide whether in-memory tables should inline Arrow data or remain unsupported by default.
- Update
Permutation to rely on table pickle support instead of local helper functions.
- Preserve existing
with_connection_factory as the escape hatch for custom worker-side connection setup.
Follow-up from #3432.
RemoteTablenow carries__getstate__/__setstate__so it can be reopened after pickle/spawn and fork.Permutationstill contains localLanceTableserialization helpers for native and in-memory tables, which makes the ownership of table pickling asymmetric.We should move local table pickle support onto
LanceTableitself so code that needs a picklable table can rely on the table abstraction instead of duplicating table-specific reopen logic.Suggested scope:
__getstate__/__setstate__toLanceTablefor persisted local tables.Permutationto rely on table pickle support instead of local helper functions.with_connection_factoryas the escape hatch for custom worker-side connection setup.