Skip to content

Commit 87752c5

Browse files
alxmrsclaude
andcommitted
Address review: batched partitions, a shorter docstring, two fallbacks
- Cut partitions with itertools.batched, shimmed for Python 3.10 and 3.11. - Say in from_dataset's docstring, in four lines, when a Dataset is an in-memory table and what an in-place write must keep. - Check the Arrow batches' size against the 64 MiB limit, since the estimate undercounts booleans and long strings. - Read a Dataset with a coordinate on a dimension no data variable has lazily, as main does; the in-memory path raised on it. Every query on such a table fails on main too (#286). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
1 parent c3d8224 commit 87752c5

3 files changed

Lines changed: 53 additions & 18 deletions

File tree

‎tests/test_sql.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -664,3 +664,26 @@ def test_in_memory_table_shares_the_dataset_s_memory(chunks):
664664
ds["w"].values[...] = 7.0
665665

666666
assert ctx.sql("SELECT w FROM t").to_pandas()["w"].tolist() == [7.0] * 24
667+
668+
669+
def test_in_memory_dataset_over_the_limit_in_arrow_is_read_lazily():
670+
"""The 64 MiB limit is on the table as Arrow, which a long string can
671+
exceed when an estimate of its size doesn't."""
672+
ds = xr.Dataset({"s": (["x"], np.array([b"a" * 40 * 2**20] * 2))})
673+
ctx = XarrayContext()
674+
ctx.from_dataset("strings", ds)
675+
676+
assert "XarrayScanExec" in _scans(ctx, "strings")
677+
678+
679+
def test_dimension_no_data_variable_has_registers():
680+
"""A coordinate on a dimension that no data variable has doesn't stop
681+
an in-memory Dataset from registering."""
682+
ds = xr.Dataset(
683+
{"t": (("x",), np.arange(3.0))},
684+
coords={"x": [0, 1, 2], "y": [10, 20]},
685+
)
686+
ctx = XarrayContext()
687+
ctx.from_dataset("t", ds)
688+
689+
assert "t" in ctx.catalog().schema().names()

‎xarray_sql/reader.py‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,10 @@
1010

1111
from __future__ import annotations
1212

13+
import itertools
1314
import os
14-
from collections.abc import Callable, Iterator
15+
import sys
16+
from collections.abc import Callable, Iterable, Iterator
1517
from typing import TYPE_CHECKING
1618

1719
import numpy as np
@@ -35,6 +37,17 @@
3537
if TYPE_CHECKING:
3638
from ._native import LazyArrowStreamTable
3739

40+
if sys.version_info >= (3, 12):
41+
_batched = itertools.batched
42+
else:
43+
44+
def _batched(iterable: Iterable, n: int) -> Iterator[tuple]:
45+
"""``itertools.batched``, which Python 3.12 added: ``iterable`` in
46+
tuples of ``n``, the last one shorter."""
47+
it = iter(iterable)
48+
while batch := tuple(itertools.islice(it, n)):
49+
yield batch
50+
3851

3952
class XarrayRecordBatchReader:
4053
"""A lazy Arrow stream reader for xarray Datasets.
@@ -245,6 +258,11 @@ def _in_memory_partitions(
245258
# loaded: a NumPy array, not a Dask array or a lazily indexed file.
246259
if not all(var._in_memory for var in ds.variables.values()):
247260
return None
261+
# TODO(#286): register a dimension that no data variable has (only a
262+
# coordinate does) as the lazy reader should, then drop this fallback.
263+
data_vars = list(ds.data_vars.values())
264+
if data_vars and set(ds.dims) - set(data_vars[0].dims):
265+
return None
248266
ds = _ensure_default_indexes(ds)
249267
schema = _parse_schema(ds)
250268
num_rows = int(np.prod([ds.sizes[d] for d in ds.dims]))
@@ -267,10 +285,11 @@ def _in_memory_partitions(
267285
per_partition = -(-num_rows // (count * batch_size))
268286
size = -(-num_rows // (count * per_partition))
269287
batches = list(iter_record_batches(ds, schema, size))
270-
return [
271-
batches[i : i + per_partition]
272-
for i in range(0, len(batches), per_partition)
273-
]
288+
# The estimate undercounts booleans and long strings, so check the
289+
# batches themselves.
290+
if sum(batch.nbytes for batch in batches) > limit:
291+
return None
292+
return [list(p) for p in _batched(batches, per_partition)]
274293

275294

276295
def read_xarray_table(

‎xarray_sql/sql.py‎

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -77,19 +77,12 @@ def from_dataset(
7777
)
7878
ctx.sql('SELECT * FROM era5.surface')
7979
80-
A Dataset is read lazily, chunk by chunk, as queries run, unless it
81-
is already in memory (NumPy arrays, not Dask or a lazily opened
82-
file) and at most 64 MiB as a table. Such a Dataset is registered
83-
as a native DataFusion in-memory table, which is faster to scan
84-
many times, as a query does when it reads a table in several
85-
places. The table shares its data variables' memory with the
86-
Dataset rather than copying it, so, as with a lazy table, changing
87-
the Dataset's values in place changes what queries return. The one
88-
exception is NaN, which a query reads as NULL: the table records
89-
which values are NaN when it is registered. So an in-place write
90-
must keep the NaNs where they were, or the Dataset must be
91-
registered again. The table adds a column per dimension, one value
92-
per row.
80+
A Dataset is read lazily, chunk by chunk, as queries run. One
81+
already in memory (NumPy, not Dask or a lazily opened file) and at
82+
most 64 MiB as a table is instead a DataFusion in-memory table,
83+
which is faster to scan many times. It shares the Dataset's memory,
84+
so changing values in place changes what queries return, as long
85+
as no value becomes or stops being NaN (which reads as NULL).
9386
9487
For datasets with non-Gregorian cftime coordinates (e.g. 360_day,
9588
julian), a ``cftime()`` scalar UDF is automatically registered so

0 commit comments

Comments
 (0)