You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1c84bac
Browse filesBrowse the repository at this point in the historyBrowse files
perf: keep dense-null and low-cardinality dictionary inputs off the slow path
Review of #5771 asked for dense-null and dictionary shapes in the no-regression
comparison. Adding them found two real regressions, both now fixed.
Arrow's `unary_opt`, which the replaced `date_part` path uses, visits only valid
indices via `try_for_each_valid_idx`, while `unary` evaluates every slot. An
all-null batch was therefore nearly free before and was not after: dayofweek on a
flat all-null array measured 0.30x, a 3.3x slowdown. `map_dates` and the clock
fast path now switch on null density. The threshold is half the batch rather than
the mere presence of a null, because `unary` was still ahead at 87.5% nulls and
switching on any null gave up most of the win on a lightly-null column
(sparse dropped from 8.97x to 3.45x before the threshold was tuned).
Arrow's `date_part` maps a dictionary's values and rewraps the keys, so the old
chain's calendar work was proportional to cardinality. Decoding to Date32 first
made a cardinality-8 column about 2x slower (0.51x). The weekday kernels now map
the dictionary values and rewrap, then unpack, matching the old cost profile with
a cheaper kernel; cardinality 1024 improves from 1.19x to about 2.1x as a result.
`map_dates` takes a generic `F: Fn(i32) -> i32 + Copy` rather than a `fn` pointer:
a pointer forces an indirect call per element inside `unary` and cost about 2.5x
(no-null dayofweek fell from 9.0x to 3.7x before this was corrected).
Benchmarks: both targets gain dense-null and all-null shapes plus dictionary
inputs at cardinality 8 and 1024, comparing the real UDF against the complete old
chain including the unpacking cast. The old-path arm also had `Cast` and `Add`
in the wrong order; the serde emitted `Add(Cast(datepart(..), Int32), 1)`, and
arrow rejects `Dictionary(Int32, Int32) + Int32`, so the dictionary comparison
could not have been valid before. The clock generator spanned only 1965-1966
despite its mixed-epoch description and now walks 1875 to 2064 across the epoch.
Equivalence assertions compare nullability per row on every shape.
Co-Authored-By: Claude Opus 5 <[email protected]>
0 commit comments