You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d4e0b24
Browse filesBrowse the repository at this point in the historyBrowse files
Register small in-memory Datasets as native in-memory tables
Every scan of a lazy table goes through datafusion-ffi, which rebuilds
the session and the task context, every registered function included,
on each scan and each partition's execution: about 1.5 ms per scan
(#273). That is noise for one scan of a large Dataset, but DataFusion
inlines a CTE or a view at every reference, so a query over small tables
can scan them thousands of times.
Context.from_dataset now converts a Dataset that is already in memory
and at most 64 MiB as long-form Arrow once, at registration, one
partition per chunk, and registers it as a native DataFusion in-memory
table, which bypasses FFI. Lazy and larger Datasets keep the lazy table,
with partition pruning, projection pushdown and exact statistics.
A scan of a 1,040-row Dataset goes from 1.7 ms to 0.26 ms. On the GPT
of the ML demos, the forward loss query goes from 9.2 s to 2.5 s.
The statistics and pruning tests exercise the lazy table, so they
register Dask-backed Datasets.
Closes#272
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
0 commit comments