Skip to content

Commit c3d8224

Browse files
alxmrsclaude
andcommitted
Say that an in-place write must keep a Dataset's NaNs where they were
An in-memory table records which values are NaN (read as NULL) when it is registered, so a NaN that an in-place write moves reads wrongly. See #284. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
1 parent fd5cf9c commit c3d8224

2 files changed

Lines changed: 12 additions & 6 deletions

File tree

‎xarray_sql/reader.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -230,10 +230,12 @@ def _in_memory_partitions(
230230
them, as many partitions as ``chunks`` makes chunks, up to the number
231231
of CPUs. So a data variable whose array is C-contiguous isn't copied,
232232
whatever the chunks: its Arrow column shares the array's memory, as
233-
does DataFusion's table, through Arrow's C data interface. A NaN adds a validity bitmap beside it, one
234-
bit per value. What the table adds is its dimension columns, one value
235-
per row for each dimension, which long-form rows need. An array that
236-
isn't C-contiguous (Fortran order) is copied, once.
233+
does DataFusion's table, through Arrow's C data interface. A NaN adds a
234+
validity bitmap beside it, one bit per value, made here: a NaN written
235+
into the array later isn't read as NULL. What the table adds is its
236+
dimension columns, one value per row for each dimension, which
237+
long-form rows need. An array that isn't C-contiguous (Fortran order)
238+
is copied, once.
237239
238240
Returns ``None`` for a Dataset that isn't in memory (a Dask array or a
239241
lazily loaded file), is larger than ``limit``, or has no rows; it is

‎xarray_sql/sql.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,12 @@ def from_dataset(
8484
many times, as a query does when it reads a table in several
8585
places. The table shares its data variables' memory with the
8686
Dataset rather than copying it, so, as with a lazy table, changing
87-
the Dataset's values in place changes what queries return. It adds
88-
a column per dimension, one value per row.
87+
the Dataset's values in place changes what queries return. The one
88+
exception is NaN, which a query reads as NULL: the table records
89+
which values are NaN when it is registered. So an in-place write
90+
must keep the NaNs where they were, or the Dataset must be
91+
registered again. The table adds a column per dimension, one value
92+
per row.
8993
9094
For datasets with non-Gregorian cftime coordinates (e.g. 360_day,
9195
julian), a ``cftime()`` scalar UDF is automatically registered so

0 commit comments

Comments
 (0)