Repository navigation
Commit ddf8a47
Share the Dataset's memory in an in-memory table, whatever the chunks
A data variable's Arrow column already shared its NumPy array's memory,
and DataFusion's table keeps those buffers through Arrow's C data
interface. But a partition per chunk flattened each chunk with isel, and
a chunk of a trailing dimension isn't contiguous, so it was copied.
Partitions are now contiguous ranges of the C-order rows, as many as
there are chunks, up to the number of CPUs, so no chunking copies a
C-contiguous array. The table adds only its dimension columns. The
docstring no longer says the Dataset is held twice.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h1 parent f22d938 commit ddf8a47
3 files changed
Lines changed: 51 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
647 | 647 | | |
648 | 648 | | |
649 | 649 | | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
223 | 224 | | |
224 | 225 | | |
225 | 226 | | |
226 | | - | |
227 | | - | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
228 | 237 | | |
229 | 238 | | |
230 | 239 | | |
| |||
239 | 248 | | |
240 | 249 | | |
241 | 250 | | |
242 | | - | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
243 | 268 | | |
244 | | - | |
245 | | - | |
| 269 | + | |
| 270 | + | |
246 | 271 | | |
247 | 272 | | |
248 | 273 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
85 | 87 | | |
86 | 88 | | |
87 | 89 | | |
| |||
0 commit comments