You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bb9e740
Browse filesBrowse the repository at this point in the historyBrowse files
fix: apply the parent struct's null mask before hashing its fields (apache#5754)
Arrow keeps a `StructArray`'s children validity independent of the parent's, so at a
row where the struct itself is null a child buffer can still hold a value. The
struct branch of `create_hashes_internal!` recursed straight into
`struct_array.columns()` without consulting the parent's null buffer, so a null
struct hashed whatever happened to sit in the child slot instead of leaving the seed
alone as Spark does.
Two rows with the same logical key then hash differently. What the tests here
demonstrate directly is a wrong answer from `hash` and `xxhash64` in a Spark query.
The same hash decides shuffle partition assignment, so equal keys reaching different
partitions follows from it, but that path is not exercised by these tests. The `List`
and `Map` branches already guard on `is_null`; only `Struct` did not.
Uses `StructArray::flatten`, which unions the parent's nulls into each child exactly
the way apache#4432 fixed the same null-mask propagation problem in `GetStructField`, and
does it without revalidating the child data buffers: the union only ever adds nulls,
so the buffers are unchanged. Going through the checked `ArrayData` builder instead
would rescan every child buffer on each call -- for a string child, the whole UTF-8
values buffer -- and this branch runs once per element when hashing a list of
structs.
`flatten` is called only when the parent actually has a null, because it also builds
a fresh `Fields` with every non-nullable field re-marked nullable, which this call
site discards. A struct carrying an all-valid null buffer, which is what slicing
leaves behind, would otherwise pay a `Vec` and an `Arc<[FieldRef]>` for nothing.
`NullBuffer` stores its null count, so the test itself is free.
The branch is shared by murmur3 and xxhash64 through the macro, so both are fixed
and both get a regression test. Each test fails without the change: the null row
hashes 3319311472 rather than the 42 seed under murmur3.
The unit tests hand-build the array, so they do not show that a query can reach this
shape. It can: a nullable struct whose child field is REQUIRED is written as
`optional group c { required int32 a; }`, and on read the child leaf has nowhere to
record a null of its own, so its buffer holds a value at exactly the rows where the
struct is null. Two end-to-end tests in `CometHashExpressionSuite` cover that, one
with a scalar child and one where the child is itself a struct so the union has to
recurse, and both disagree with Spark before the change.
The per-element route through `hash_list_array!` gets an end-to-end test too. Using
the struct as the list element directly does not reproduce, because a null element is
rebuilt on the way into the array and the hidden child values go with it. Wrapping it
does: `array(named_struct('tag', 1, 'b', c))` yields an element that is itself valid,
so it is copied rather than rebuilt, and the null `c` inside keeps the values Parquet
wrote under it. Covered with one element and with two, so the chaining between
elements is exercised, and both disagree with Spark before the change.
Co-authored-by: Claude Code <[email protected]>
0 commit comments