fix(python): push down namespace full reads#3516
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
Xuanwo
left a comment
Xuanwo
left a comment
There was a problem hiding this comment.
This needs one correctness fix before merging.
| if _should_push_down_query_table( | ||
| self._namespace_client, self._pushdown_operations | ||
| ): | ||
| return self._execute_query(Query()).read_all() |
There was a problem hiding this comment.
This changes full-table reads into a QueryTable request built from an empty Query(), but _query_to_namespace_request() maps limit=None to k=10. That means namespace-backed to_arrow() / non-blob to_pandas() can silently return only 10 rows instead of the full table. We need a request/path that preserves unbounded plain-scan semantics, plus a >10 row regression test through the real request conversion layer.
There was a problem hiding this comment.
Fixed in a041f48. _query_to_namespace_request() now maps plain non-vector/non-FTS queries with limit=None to sys.maxsize, matching the existing remote-table unbounded sentinel, while vector/FTS queries keep the existing default k=10. The namespace regression now goes through the real _execute_server_side_query() / request conversion layer with 12 rows and asserts both to_arrow() and to_pandas() send k == sys.maxsize.
e3a2628 to
a041f48
Compare
2 participants
Bug Fix
What is the bug?
Namespace-backed
LanceTable.to_arrow()full-table reads bypassed the existingQueryTableserver-side query path and called the lower-level tableto_arrow()implementation directly. In Geneva/Sophon this could fail while parsing the Arrow IPC response forhist.get_table().to_arrow()/to_pandas(), even thoughhist.get_table().search().to_arrow()worked.What issues or incorrect behavior does the bug cause?
Full-table reads on namespace-backed tables with
QueryTablepushdown could fail with Arrow IPC parse errors, while query/search reads on the same table succeeded. Sinceto_pandas()delegates throughto_arrow()for non-blob/native cases, pandas export was affected too.How does this PR fix the problem?
When
QueryTablepushdown is enabled, sync and async tableto_arrow()now construct a plain no-filter, no-limit, all-columns query and execute it through the table-level_execute_query()path.AsyncTablenow preserves namespace context from async namespace connections so async full reads can make the same pushdown decision. Non-namespace tables and namespace tables withoutQueryTablepushdown keep their existing behavior.Tests
uv run --extra tests --extra dev --no-sync ruff check python/lancedb/table.py python/lancedb/namespace.py python/tests/test_namespace.pyuv run --extra tests --extra dev --no-sync ruff format python/lancedb/table.py python/lancedb/namespace.py python/tests/test_namespace.pyuv run --extra tests --extra dev --no-sync pytest python/tests/test_namespace.py::TestPushdownOperations::test_lance_table_to_arrow_uses_query_pushdown python/tests/test_namespace.py::TestAsyncPushdownOperations::test_async_table_to_arrow_uses_query_pushdown python/tests/test_namespace.py::test_local_table_to_arrow_and_to_pandas_are_unchanged -quv run --extra tests --extra dev --no-sync pytest python/tests/test_namespace.py -q