You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8e18e95
Browse filesBrowse the repository at this point in the historyBrowse files
fix: compare literals with negative-scale decimal columns without panicking (#145)
Arrow allows a negative column scale. Against a scale-38 literal, a
decimal128(38, -1) column is 39 digits coarser, and fit_decimal raised
10 to that gap and panicked. A gap past i128 now places the literal
within one step of zero: 0 or -1 as the floor, exact only for zero.
Below 38 widened digits, a negative-scale column had a second panic.
DataFusion's unwrap_cast simplifier raises 10 to the column's scale as
an unsigned power (apache/datafusion#24896). Typed Decimal literals
started reaching it (`main` sent them as strings, which failed with a
ValueError), and integer literals already panicked on `main`. A
comparison or is_in against a negative-scale column now gives Decimal
and integer literals the column's own type, so DataFusion never casts
the column. Float literals still reach the simplifier and panic as on
`main`; that is #226, and the grid marks those cells strict xfail.
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,7 @@ Literal values in lambdas are typed (#145). They used to be sent to the engine a
18
18
### Fixed
19
19
20
20
-`Decimal` literals: compared with a float column as that float, instead of casting the column to decimal (which failed on NaN and infinity); compared exactly with decimal and integer columns, also when the result would need more than 38 digits; usable in arithmetic.
21
+
- An integer or `Decimal` literal compared with a decimal column of negative scale (`decimal128(5, -2)`) is compared exactly; an integer literal used to panic in DataFusion's simplifier. A float literal still panics there (#226).
21
22
- A timestamp literal finer than the column's unit (a `pandas.Timestamp` with nanoseconds against microseconds, a `datetime` with microseconds against seconds) was truncated; it compares exactly in filters, windows, group filters and `is_in` (#200).
22
23
- A `date` compared with a timezone-aware column is that zone's local midnight, and a naive `datetime` its wall-clock time; a naive time in a DST gap or fold raises `ValueError` naming the zone instead of failing inside the optimizer.
23
24
- A `datetime` before 1677 or after 2262 can be compared with date and timestamp columns.
0 commit comments