Skip to content

Commit 2c29974

Browse files
committed
test: cover the dispatched map builders under LAST_WIN with a repeated key
The dedup-policy fixtures turn the codegen dispatcher off, so nothing ran the default LAST_WIN path with a key that really repeats. Add map_builders_float_key_last_win.sql, which runs map_from_arrays and map_from_entries through the dispatcher with a repeated string key, NaNs that differ only in the sign bit, and 0.0 / -0.0, comparing the kept value and the stored key with Spark's on every version. Also drop the claim that Spark normalizes map keys before a lookup from the MapKeySupport scaladoc and the lookup fixtures. Normalization is a builder behavior from Spark 4.0, and the lookup compares with nanSafeCompareDoubles on every version.
1 parent ee66473 commit 2c29974

7 files changed

Lines changed: 107 additions & 12 deletions

File tree

‎spark/src/main/scala/org/apache/comet/serde/maps.scala‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,9 @@ private[serde] object MapKeySupport {
4848

4949
/**
5050
* The `SupportLevel` for a map-consuming expression whose stored-key type is `keyType`. Spark
51-
* finds a key with `TypeUtils.getInterpretedOrdering` over the keys `ArrayBasedMapBuilder`
52-
* stored, having first normalized them, while native `map_extract` compares the Arrow values as
53-
* they are, so decline the key types where those disagree:
51+
* finds a key with `TypeUtils.getInterpretedOrdering` over the map's stored keys, while native
52+
* `map_extract` compares the Arrow values as they are, so decline the key types where those
53+
* disagree:
5454
* - `nanSafeCompareDoubles` treats `-0.0` and `+0.0` as equal and all `NaN`s as equal, so
5555
* Spark answers a `-0.0` lookup from a `+0.0` key where native finds nothing, on every
5656
* version. (How `ArrayBasedMapBuilder` treats float keys when it builds a map differs by

‎spark/src/test/resources/sql-tests/expressions/map/element_at_map.sql‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,8 @@ SELECT element_at(map('a', 1, 'b', 2), 'a'), element_at(map('a', 1, 'b', 2), 'mi
5656
-- Spark. These stay on the constructor path here because the SQL harness excludes
5757
-- `ConstantFolding`; `CometMapExpressionSuite` covers the folded-literal form of each.
5858

59-
-- Spark stores `-0.0` map keys as `+0.0` and compares with `nanSafeCompareDoubles`, so a `-0.0`
60-
-- lookup finds the `+0.0` key. Native lookup compares the raw Arrow values.
59+
-- Spark compares with `nanSafeCompareDoubles`, so a `-0.0` lookup finds the `+0.0` key. Native
60+
-- lookup compares the raw Arrow values.
6161
query expect_fallback(Spark looks a key up in its SQL ordering)
6262
SELECT element_at(map(CAST(0 AS DOUBLE), 7), CAST(-0.0 AS DOUBLE))
6363

‎spark/src/test/resources/sql-tests/expressions/map/get_map_value.sql‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,8 @@ SELECT map('a', 1, 'b', 2)['a'], map('a', 1, 'b', 2)['missing'], map('a', 1, 'b'
3636
-- `element_at_map.sql` exercises the same guard on `CometElementAt`). A `map<double,int>` column
3737
-- is used rather than a `map(...)` literal because Spark's `SimplifyExtractValueOps` rewrites
3838
-- `map(...)[key]` over a literal map into a `CASE` before it can reach the native lookup. Spark
39-
-- stores a `-0.0` key as `+0.0` and finds it with `nanSafeCompareDoubles`, so it returns `7` for a
40-
-- `-0.0` lookup; native lookup compares the raw Arrow values.
39+
-- compares with `nanSafeCompareDoubles`, so a `-0.0` lookup finds the `+0.0` key and returns `7`;
40+
-- native lookup compares the raw Arrow values.
4141
statement
4242
CREATE TABLE test_map_double(m map<double, int>) USING parquet
4343

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
-- Under `spark.sql.mapKeyDedupPolicy=LAST_WIN` a repeated map key is not an error: the builder keeps
18+
-- the last value. `CometMapFromArrays` and `CometMapFromEntries` mix in `CodegenDispatchFallback`,
19+
-- so with the dispatcher on (the default) both run Spark's own `ArrayBasedMapBuilder` in the JVM
20+
-- codegen dispatcher. The dedup-policy fixtures turn the dispatcher off to cover the fallback, so
21+
-- these queries cover the dispatched path with a key that really repeats: `from` then goes through
22+
-- `build()`, and both the kept value and the stored key are compared with Spark's.
23+
--
24+
-- A `FLOAT` or `DOUBLE` key is declined natively on every version (#6549), so those dispatch too.
25+
-- NaNs that differ only in the sign bit are one key on every version. `-0.0` and `0.0` are one key
26+
-- from Spark 4.0, unless `spark.sql.legacy.disableMapKeyNormalization` is set, and two keys on
27+
-- earlier releases; either way the dispatcher has to agree with Spark.
28+
29+
-- Config: spark.sql.mapKeyDedupPolicy=LAST_WIN
30+
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
31+
32+
-- Parquet canonicalizes NaN, so negating a stored NaN gives one with the sign bit set.
33+
statement
34+
CREATE TABLE float_map_keys_last_win(id int, d double, f float, k array<string>, v array<int>)
35+
USING parquet
36+
37+
statement
38+
INSERT INTO float_map_keys_last_win VALUES
39+
(1, 0.0, 0.0, array('a', 'a', 'b'), array(1, 2, 3)),
40+
(2, double('NaN'), float('NaN'), array('x', 'x'), array(10, 20)),
41+
(3, 1.5, 1.5, array('a', 'b', 'c'), array(1, 2, 3))
42+
43+
-- A repeated string key: the last value wins.
44+
query expect_dispatch(map_from_arrays)
45+
SELECT id, map_keys(map_from_arrays(k, v)), map_values(map_from_arrays(k, v))
46+
FROM float_map_keys_last_win
47+
48+
query expect_dispatch(map_from_entries)
49+
SELECT id,
50+
map_keys(map_from_entries(arrays_zip(k, v))),
51+
map_values(map_from_entries(arrays_zip(k, v)))
52+
FROM float_map_keys_last_win
53+
54+
-- NaNs with different sign bits are one DOUBLE or FLOAT key: one entry, holding the last value.
55+
query expect_dispatch(map_from_arrays)
56+
SELECT map_keys(map_from_arrays(array(d, -d), array(1, 2))),
57+
map_values(map_from_arrays(array(d, -d), array(1, 2))),
58+
map_keys(map_from_arrays(array(f, -f), array(1, 2))),
59+
map_values(map_from_arrays(array(f, -f), array(1, 2)))
60+
FROM float_map_keys_last_win WHERE id = 2
61+
62+
query expect_dispatch(map_from_entries)
63+
SELECT map_keys(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
64+
map_values(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
65+
map_keys(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2)))),
66+
map_values(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2))))
67+
FROM float_map_keys_last_win WHERE id = 2
68+
69+
-- 0.0 and -0.0: one key holding the last value from Spark 4.0, two keys before that.
70+
query expect_dispatch(map_from_arrays)
71+
SELECT map_keys(map_from_arrays(array(d, -d), array(1, 2))),
72+
map_values(map_from_arrays(array(d, -d), array(1, 2))),
73+
map_keys(map_from_arrays(array(f, -f), array(1, 2))),
74+
map_values(map_from_arrays(array(f, -f), array(1, 2)))
75+
FROM float_map_keys_last_win WHERE id = 1
76+
77+
query expect_dispatch(map_from_entries)
78+
SELECT map_keys(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
79+
map_values(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
80+
map_keys(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2)))),
81+
map_values(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2))))
82+
FROM float_map_keys_last_win WHERE id = 1
83+
84+
-- The stored key, when -0.0 is the first key: 0.0 from Spark 4.0 (normalized), -0.0 before that.
85+
query expect_dispatch(map_from_arrays)
86+
SELECT map_keys(map_from_arrays(array(-d, d), array(1, 2))),
87+
map_values(map_from_arrays(array(-d, d), array(1, 2)))
88+
FROM float_map_keys_last_win WHERE id = 1
89+
90+
query expect_dispatch(map_from_entries)
91+
SELECT map_keys(map_from_entries(array(named_struct('k', -d, 'v', 1), named_struct('k', d, 'v', 2)))),
92+
map_values(map_from_entries(array(named_struct('k', -d, 'v', 1), named_struct('k', d, 'v', 2))))
93+
FROM float_map_keys_last_win WHERE id = 1

‎spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_dedup_policy.sql‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,9 @@
2020
-- Comet's native `map` scalar has no LAST_WIN path. `CometMapFromArrays` mixes in
2121
-- `CodegenDispatchFallback`, so its native `Incompatible` normally routes through the JVM codegen
2222
-- dispatcher; we disable the dispatcher here so the incompat branch surfaces as a genuine Spark
23-
-- fallback rather than in-pipeline codegen. The default `EXCEPTION` mode agrees with Comet and is
24-
-- covered by `map_from_arrays.sql`.
23+
-- fallback rather than in-pipeline codegen. The dispatched path, with a key that really repeats, is
24+
-- covered by `map_builders_float_key_last_win.sql`. The default `EXCEPTION` mode agrees with Comet
25+
-- and is covered by `map_from_arrays.sql`.
2526

2627
-- Config: spark.sql.mapKeyDedupPolicy=LAST_WIN
2728
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=false

‎spark/src/test/resources/sql-tests/expressions/map/map_from_entries_dedup_policy.sql‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,9 @@
1919
-- to `LAST_WIN`. `CometMapFromEntries` mixes in `CodegenDispatchFallback`, so its native
2020
-- `Incompatible` normally routes through the JVM codegen dispatcher; we disable the dispatcher
2121
-- here so the incompat branch surfaces as a genuine Spark fallback rather than in-pipeline
22-
-- codegen. The default `EXCEPTION` mode agrees with Comet and is covered by
23-
-- `map_from_entries.sql`.
22+
-- codegen. The dispatched path, with a key that really repeats, is covered by
23+
-- `map_builders_float_key_last_win.sql`. The default `EXCEPTION` mode agrees with Comet and is
24+
-- covered by `map_from_entries.sql`.
2425

2526
-- Config: spark.sql.mapKeyDedupPolicy=LAST_WIN
2627
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=false

‎spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -489,7 +489,7 @@ class CometMapExpressionSuite extends CometTestBase {
489489
}
490490

491491
// Direct single-level folded map with floating-point keys: `map(CAST(0 AS DOUBLE), 7)` folds to a
492-
// literal and `element_at` with a dynamic `-0.0` lookup must match Spark's `+0.0`-normalized key.
492+
// literal and `element_at` with a dynamic `-0.0` lookup must match Spark's `+0.0` key.
493493
// Native `map_extract` compares raw Arrow values, so `MapKeySupport` declines it at `element_at`.
494494
// Spark returns 7, NULL, NULL. (`element_at_map.sql` covers the folding-off constructor form.)
495495
test("folded map literal with floating-point keys in element_at falls back (multirow)") {

0 commit comments

Comments
 (0)