You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1299e4e
Browse filesBrowse the repository at this point in the historyBrowse files
perf: Scan Lance sources lazily in the DuckDB offline store
Register the LanceDataset with DuckDB instead of materializing it with
to_arrow(), so the query's column projection is pushed into the Lance
scanner. Retrievals that select scalar features no longer read the
embedding columns stored next to them.
Signed-off-by: haoxu0 <[email protected]>
Copy file name to clipboardExpand all lines: docs/reference/data-sources/table-formats.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,6 +31,10 @@ lance_format = LanceFormat(
31
31
)
32
32
```
33
33
34
+
The DuckDB offline store reads a `LanceSource` as a lazy scan, so only the
35
+
columns a query selects are read from the dataset. Materializing a few scalar
36
+
features does not load the embedding columns stored beside them.
37
+
34
38
### Apache Iceberg
35
39
36
40
[Apache Iceberg](https://iceberg.apache.org/) is an open table format designed for huge analytic datasets. It provides:
0 commit comments