Skip to content

Commit c734f78

Browse files
committed
fix: compare literals exactly where DataFusion's coercion would not (#145)
Phase 3 of the redesign approved on #225: comparisons and is_in. DataFusion compares an operand and a literal at a common type. For an exact operand (integer, decimal, date, timestamp) that fails in three ways. A finer timestamp literal is truncated, because the simplifier moves the operand's cast onto the literal by integer division (#200). A negative-scale decimal operand panics the simplifier (apache/datafusion#24896, #226). And when the common type cannot hold the operand or the literal (the 38-digit decimal clamp, the nanosecond range), the comparison fails at collect (#227). literals::comparison asks DataFusion's own BinaryTypeCoercer what it would do with the pair. When it would cast the exact operand to a decimal, date or timestamp type, or the literal does not fit the common type exactly, ltseq places the literal among the operand's own values (exact::place). An exact placement is the same comparison at the operand's type. A literal strictly between two values becomes a floor rewrite (> floor, <= floor). A literal beyond every value becomes a constant verdict that keeps NULL (P8). The operand is never cast, so the simplifier never sees the pattern it mishandles. A float literal is placed as DataFusion reads it, so float semantics stay DataFusion's. Integer common types are left alone, since the simplifier unwraps them exactly. is_in places each literal item the same way: it keeps an exact item at the operand's type and drops an item no value can equal (an empty list is false, NULL kept). The list never needs a common type of its own, so a wide and a fine Decimal no longer overflow together. A naive or date literal whose instant in the operand's zone fits no unit exactly (pandas.Timestamp.max in New York) is now a Reading::Instant from the policy. Only a comparison can use it, by placing it. The placement arithmetic (decimal scales, integer ranges, timestamp units, dates) is unit-tested; the decimal table is the one the earlier #225 branch checked against an exact Fraction oracle. A new property, is_in agreeing with the == disjunction, joins the metamorphic suite. All 94 P3 cases pass and leave the strict xfail list. The grid's #226 strict xfails pass too and are removed. In the differential, exactly the five P3 cases change, each to its classified outcome.
1 parent 9aa0512 commit c734f78

7 files changed

Lines changed: 693 additions & 164 deletions

File tree

‎py-ltseq/tests/literal_redesign_xfail.txt‎

Lines changed: 0 additions & 94 deletions
Original file line numberDiff line numberDiff line change
@@ -4,33 +4,12 @@
44
# makes a test pass removes its line. The list is empty after Phase 7.
55

66
# The operand-rule tests of the earlier #225 branch (characterization).
7-
P3 py-ltseq/tests/test_literal_column_rules.py::test_high_scale_decimal_against_wide_decimal_column
8-
P3 py-ltseq/tests/test_literal_column_rules.py::test_decimal_beyond_the_column_range
9-
P3 py-ltseq/tests/test_literal_column_rules.py::test_decimal_in_list_against_wide_decimal_column
10-
P3 py-ltseq/tests/test_literal_column_rules.py::test_decimal_in_list_mixing_wide_and_fine_items
117
P4 py-ltseq/tests/test_literal_column_rules.py::test_decimal_values_are_fitted_together
128
P4 py-ltseq/tests/test_literal_column_rules.py::test_values_are_typed_by_every_column[decimals]
139
P4 py-ltseq/tests/test_literal_column_rules.py::test_values_no_common_column_type_holds[decimals]
1410
P4 py-ltseq/tests/test_literal_column_rules.py::test_values_no_common_column_type_holds[timestamps]
15-
P3 py-ltseq/tests/test_literal_column_rules.py::test_negative_scale_column_against_a_finer_literal[lit0]
16-
P3 py-ltseq/tests/test_literal_column_rules.py::test_negative_scale_column_against_a_finer_literal[lit1]
17-
P3 py-ltseq/tests/test_literal_column_rules.py::test_negative_scale_column_against_a_finer_literal[lit2]
18-
P3 py-ltseq/tests/test_literal_column_rules.py::test_negative_scale_column_against_a_finer_literal[lit3]
19-
P3 py-ltseq/tests/test_literal_column_rules.py::test_negative_scale_column_against_a_finer_literal[lit4]
2011
P4 py-ltseq/tests/test_literal_column_rules.py::test_negative_scale_column_value_positions
21-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_is_compared_exactly[lt-expected0]
22-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_is_compared_exactly[ge-expected3]
23-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_is_compared_exactly[eq-expected4]
24-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_is_compared_exactly[ne-expected5]
25-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_in_the_window_dialect
26-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_in_is_in
27-
P3 py-ltseq/tests/test_literal_column_rules.py::test_finer_timestamp_literal_in_the_group_dialect
28-
P3 py-ltseq/tests/test_literal_column_rules.py::test_nanosecond_literal_against_microsecond_column
29-
P3 py-ltseq/tests/test_literal_column_rules.py::test_timestamp_literal_beyond_the_nanosecond_range
30-
P3 py-ltseq/tests/test_literal_column_rules.py::test_naive_literal_localized_past_its_own_unit
31-
P3 py-ltseq/tests/test_literal_column_rules.py::test_datetime_against_date_column_outside_the_nanosecond_range
3212
P4 py-ltseq/tests/test_literal_column_rules.py::test_wide_decimal_value_is_rescaled_only_when_exact
33-
P3 py-ltseq/tests/test_literal_column_rules.py::test_datetime_against_date64_column
3413
P4 py-ltseq/tests/test_literal_column_rules.py::test_midnight_datetime_value_with_date_column_is_a_date
3514
P4 py-ltseq/tests/test_literal_column_rules.py::test_datetime_value_with_time_of_day_for_date_column_is_an_error
3615
P4 py-ltseq/tests/test_literal_column_rules.py::test_date_value_with_naive_timestamp_column_keeps_the_column_unit
@@ -43,81 +22,13 @@ P4 py-ltseq/tests/test_literal_column_rules.py::test_shift_default_is_exact_for_
4322
P4 py-ltseq/tests/test_literal_column_rules.py::test_shift_default_is_exact_for_the_column_type[dtype10-1.236-ValueError]
4423
P4 py-ltseq/tests/test_literal_column_rules.py::test_shift_default_is_exact_for_the_column_type[dtype12-1000-ValueError]
4524
P4 py-ltseq/tests/test_literal_column_rules.py::test_shift_default_is_exact_for_the_column_type[dtype13-default13-ValueError]
46-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[i64-gt-decimal-tiny]
47-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[i64-eq-decimal-tiny]
48-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[i64-gt-decimal-tiny-neg]
49-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[i64-eq-decimal-tiny-neg]
50-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[dec-gt-decimal-tiny]
51-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[dec-eq-decimal-tiny]
52-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[dec-gt-decimal-tiny-neg]
53-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[dec-eq-decimal-tiny-neg]
54-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decw-gt-decimal-hiscale]
55-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decw-eq-decimal-hiscale]
56-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decw-gt-decimal-tiny]
57-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decw-eq-decimal-tiny]
58-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decw-gt-decimal-tiny-neg]
59-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decw-eq-decimal-tiny-neg]
60-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-int]
61-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-int]
62-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-decimal]
63-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-decimal]
64-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-decimal-0.1]
65-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-decimal-0.1]
66-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-decimal-hiscale]
67-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-decimal-hiscale]
68-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-decimal-tiny]
69-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-decimal-tiny]
70-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-decimal-tiny-neg]
71-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-decimal-tiny-neg]
72-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-decimal-zero-scale38]
73-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-decimal-zero-scale38]
74-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-gt-np-int]
75-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg-eq-np-int]
76-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-int]
77-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-int]
78-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-decimal]
79-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-decimal]
80-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-decimal-0.1]
81-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-decimal-0.1]
82-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-decimal-hiscale]
83-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-decimal-hiscale]
84-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-decimal-tiny]
85-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-decimal-tiny]
86-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-decimal-tiny-neg]
87-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-decimal-tiny-neg]
88-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-decimal-zero-scale38]
89-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-decimal-zero-scale38]
90-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-gt-np-int]
91-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[decneg_narrow-eq-np-int]
92-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[ts_s-eq-datetime-1.5s]
93-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[ts_s-eq-pd-ns]
94-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[ts_ms-eq-pd-ns]
95-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[ts_us-eq-pd-ns]
96-
P3 py-ltseq/tests/test_literal_crosstype.py::test_comparison_matches_oracle[ts_utc-eq-pd-ns]
9725

9826
# Review regression cases.
99-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_negative_scale_does_not_panic
10027
P4 py-ltseq/tests/test_literal_review_regressions.py::test_lossy_float_default_rejected[dtype0-values0-1.5]
10128
P4 py-ltseq/tests/test_literal_review_regressions.py::test_lossy_float_default_rejected[dtype1-values1-1.236]
102-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_nanosecond_wall_clock_literal_can_localize_into_microseconds
103-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_mixed_decimal_membership_agrees_with_equality_disjunction
10429
P4 py-ltseq/tests/test_literal_review_regressions.py::test_numeric_default_rejects_upper_boundary[dtype0-9.223372036854776e+18]
10530
P4 py-ltseq/tests/test_literal_review_regressions.py::test_numeric_default_rejects_upper_boundary[dtype1-1.8446744073709552e+19]
10631
P4 py-ltseq/tests/test_literal_review_regressions.py::test_numeric_default_rejects_upper_boundary[dtype2-128.0]
107-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal0-dtype0]
108-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal0-dtype1]
109-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal0-dtype2]
110-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal0-dtype3]
111-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal0-dtype4]
112-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal1-dtype0]
113-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal1-dtype1]
114-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal1-dtype2]
115-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal1-dtype3]
116-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal1-dtype4]
117-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal2-dtype1]
118-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal3-dtype1]
119-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal4-dtype0]
120-
P3 py-ltseq/tests/test_literal_review_regressions.py::test_decimal_composition_matches_exact_oracle[literal4-dtype1]
12132
P4 py-ltseq/tests/test_literal_review_regressions.py::test_nary_decimal_values_include_all_column_types
12233
P4 py-ltseq/tests/test_literal_review_regressions.py::test_coalesce_float_literal_decimal_widening[1000000.0-False]
12334
P4 py-ltseq/tests/test_literal_review_regressions.py::test_coalesce_float_literal_decimal_widening[1000000.0-True]
@@ -128,16 +39,11 @@ P4 py-ltseq/tests/test_literal_review_regressions.py::test_coalesce_float_litera
12839
# Differential cases.
12940
P4 py-ltseq/tests/differential/test_differential.py::test_case[composition::decimal_fill]
13041
P4 py-ltseq/tests/differential/test_differential.py::test_case[composition::decimal_fill_sw]
131-
P3 py-ltseq/tests/differential/test_differential.py::test_case[pairs::unit_literal_eq]
132-
P3 py-ltseq/tests/differential/test_differential.py::test_case[pairs::unit_folded_literal_eq]
133-
P3 py-ltseq/tests/differential/test_differential.py::test_case[pairs::far_date_eq_literal]
134-
P3 py-ltseq/tests/differential/test_differential.py::test_case[pairs::big_integer_literal]
13542
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::float_then_fine]
13643
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::fine_then_float]
13744
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::small_float_then_fine]
13845
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::int_then_fine]
13946
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::two_decimals_too_wide]
140-
P3 py-ltseq/tests/differential/test_differential.py::test_case[widening::isin_float_and_fine]
14147
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::float32_column_and_fine]
14248
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::uint64_column_and_fine]
14349
P4 py-ltseq/tests/differential/test_differential.py::test_case[widening::uint64_column_first_and_fine]

‎py-ltseq/tests/test_literal_crosstype.py‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -225,18 +225,8 @@ def table():
225225
return LTSeq.from_arrow(pa.table({"k": pa.array(range(5), pa.int64()), **COLUMNS}))
226226

227227

228-
# DataFusion's unwrap_cast simplifier panics planning a float literal against a
229-
# negative-scale decimal column (#226, apache/datafusion#24896).
230-
KNOWN_PANICS = {("decneg", "float"), ("decneg_narrow", "float")}
231-
232-
233228
def _param(column, lname, op, expected):
234-
marks = (
235-
[pytest.mark.xfail(strict=True, reason="#226: DataFusion's unwrap_cast panics")]
236-
if (column, lname) in KNOWN_PANICS
237-
else []
238-
)
239-
return pytest.param(column, lname, op, expected, marks=marks, id=f"{column}-{op}-{lname}")
229+
return pytest.param(column, lname, op, expected, id=f"{column}-{op}-{lname}")
240230

241231

242232
@pytest.mark.parametrize("column, lname, op, expected", [_param(*cell) for cell in CELLS])

‎py-ltseq/tests/test_literal_properties.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,35 @@ def test_mirror(operand, op, lit):
247247
assert [right[2][i] for i in nulls] == [None] * len(nulls)
248248

249249

250+
# ---------------------------------------------------------------------------
251+
# Membership is equality: is_in agrees with the == disjunction
252+
# ---------------------------------------------------------------------------
253+
254+
MEMBERSHIP = {
255+
"i": [1, 2.0, D("2.5"), 2**53 + 1, D("123456789012345678901234567890")],
256+
"p": [D("1.23"), D("1.230000000000000000000000000000000"), D("123456789012345678901234567890123456"), 2],
257+
"w": [FINE20, 10**18, D("2.5")],
258+
"s": [CUT, NS_FINE, datetime(1970, 1, 1, 0, 0, 2)],
259+
"u": [CUT, NS_FINE, datetime(2300, 1, 1)],
260+
"d": [datetime(1970, 1, 1, 6), date(1970, 1, 2), datetime(2024, 1, 1)],
261+
"z": [datetime(1970, 1, 1, 0, 0, 1), date(1970, 1, 1), _utc(18000)],
262+
}
263+
264+
265+
@pytest.mark.parametrize("name", list(MEMBERSHIP))
266+
def test_is_in_agrees_with_the_equality_disjunction(name):
267+
items = MEMBERSHIP[name]
268+
269+
def disjunction(r):
270+
column = getattr(r, name)
271+
out = column == items[0]
272+
for item in items[1:]:
273+
out = out | (column == item)
274+
return out
275+
276+
assert derived(lambda r: getattr(r, name).is_in(items)) == derived(disjunction)
277+
278+
250279
# ---------------------------------------------------------------------------
251280
# Coalesce permutation (the arguments are never both set on a row)
252281
# ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)