File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -230,10 +230,12 @@ def _in_memory_partitions(
230230 them, as many partitions as ``chunks`` makes chunks, up to the number
231231 of CPUs. So a data variable whose array is C-contiguous isn't copied,
232232 whatever the chunks: its Arrow column shares the array's memory, as
233- does DataFusion's table, through Arrow's C data interface. A NaN adds a validity bitmap beside it, one
234- bit per value. What the table adds is its dimension columns, one value
235- per row for each dimension, which long-form rows need. An array that
236- isn't C-contiguous (Fortran order) is copied, once.
233+ does DataFusion's table, through Arrow's C data interface. A NaN adds a
234+ validity bitmap beside it, one bit per value, made here: a NaN written
235+ into the array later isn't read as NULL. What the table adds is its
236+ dimension columns, one value per row for each dimension, which
237+ long-form rows need. An array that isn't C-contiguous (Fortran order)
238+ is copied, once.
237239
238240 Returns ``None`` for a Dataset that isn't in memory (a Dask array or a
239241 lazily loaded file), is larger than ``limit``, or has no rows; it is
Original file line number Diff line number Diff line change @@ -84,8 +84,12 @@ def from_dataset(
8484 many times, as a query does when it reads a table in several
8585 places. The table shares its data variables' memory with the
8686 Dataset rather than copying it, so, as with a lazy table, changing
87- the Dataset's values in place changes what queries return. It adds
88- a column per dimension, one value per row.
87+ the Dataset's values in place changes what queries return. The one
88+ exception is NaN, which a query reads as NULL: the table records
89+ which values are NaN when it is registered. So an in-place write
90+ must keep the NaNs where they were, or the Dataset must be
91+ registered again. The table adds a column per dimension, one value
92+ per row.
8993
9094 For datasets with non-Gregorian cftime coordinates (e.g. 360_day,
9195 julian), a ``cftime()`` scalar UDF is automatically registered so
You can’t perform that action at this time.
0 commit comments