You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Optional mode to preserve floating NaNs instead of constructing Arrow nulls #251
🤖 Follow-up to #152: converting NaNs to Arrow nulls is the correct default for SQL aggregation semantics, but some projection-only consumers already carry an explicit validity mask and need the original float payload rather than SQL null semantics.
For example, Samudra selects complete ocean-state planes for ML batches and applies its own wet mask. iter_record_batches() currently uses pa.array(..., from_pandas=True) for every projected variable, which scans for NaNs and constructs validity information even though the query only filters on dimension/index columns.
Would an opt-in registration/read mode such as nan_as_null=False be reasonable? The default must remain True so AVG, COUNT, IS NULL, etc. preserve the behavior fixed by #152. The opt-in mode should preserve NaN as an ordinary Arrow float and document that SQL null operations will not treat it as missing.
This should be benchmarked before implementation; the goal is to determine whether skipping null discovery/bitmap construction materially improves projection-heavy dense-array scans and reduces memory.
Activity