Repository navigation
Commit cc1326a
authored
fix: preserve dictionary-value nulls in scalar regex operators (apache#23966)
## Which issue does this PR close?
- Closes apache#23963.
## Rationale for this change
The scalar regex fast path for Dictionary arrays evaluates each
dictionary value once and gathers the resulting booleans with
`BooleanArray::take_iter`. That gather preserves null keys but drops
null validity from the evaluated dictionary values. A key referencing a
null dictionary value therefore becomes valid `false`, and negated regex
operators can incorrectly turn it into `true`.
This is distinct from apache#23722, which updates the array-pattern
`regex_match_dyn` path. This PR fixes null propagation in
`regex_match_dyn_scalar`.
## What changes are included in this PR?
- Replace the validity-dropping dictionary gather with Arrow `take`,
preserving both key validity and evaluated child validity.
- Add a physical-expression regression test for `RegexMatch`,
`RegexIMatch`, `RegexNotMatch`, and `RegexNotIMatch`. The test covers a
null key and a key referencing a null dictionary value, and compares the
Dictionary result with the equivalent cast-to-`Utf8` evaluation.
## Are these changes tested?
Yes.
- Focused regression: 1 passed.
- `datafusion-physical-expr`: 1,649 passed, including the new
regression.
- Remaining workspace: 7,928 passed, 0 failed, 7 ignored; sqllogictest
completed 499/499 files.
- `datafusion` library: 461 passed.
- `datafusion` core integration: 1,057 passed; four local
memory-validation runners exceeded their host RSS thresholds and were
isolated as environment-sensitive, unrelated to this patch.
- `datafusion-cli`: 106 passed.
- Formatting, workspace clippy with warnings denied, TOML formatting,
license headers, changed-file typo checks, and `git diff --check`
passed.
## Are there any user-facing changes?
Dictionary-encoded string inputs now preserve SQL null semantics for
scalar regex operators. There are no public API changes.
---------
Signed-off-by: discord9 <[email protected]>1 parent 802d8d2 commit cc1326a
2 files changed
Lines changed: 83 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3353 | 3353 | | |
3354 | 3354 | | |
3355 | 3355 | | |
| 3356 | + | |
| 3357 | + | |
| 3358 | + | |
| 3359 | + | |
| 3360 | + | |
| 3361 | + | |
| 3362 | + | |
| 3363 | + | |
| 3364 | + | |
| 3365 | + | |
| 3366 | + | |
| 3367 | + | |
| 3368 | + | |
| 3369 | + | |
| 3370 | + | |
| 3371 | + | |
| 3372 | + | |
| 3373 | + | |
| 3374 | + | |
| 3375 | + | |
| 3376 | + | |
| 3377 | + | |
| 3378 | + | |
| 3379 | + | |
| 3380 | + | |
| 3381 | + | |
| 3382 | + | |
| 3383 | + | |
| 3384 | + | |
| 3385 | + | |
| 3386 | + | |
| 3387 | + | |
| 3388 | + | |
| 3389 | + | |
| 3390 | + | |
| 3391 | + | |
| 3392 | + | |
| 3393 | + | |
| 3394 | + | |
| 3395 | + | |
| 3396 | + | |
| 3397 | + | |
| 3398 | + | |
| 3399 | + | |
| 3400 | + | |
| 3401 | + | |
| 3402 | + | |
| 3403 | + | |
| 3404 | + | |
| 3405 | + | |
| 3406 | + | |
| 3407 | + | |
| 3408 | + | |
| 3409 | + | |
| 3410 | + | |
| 3411 | + | |
| 3412 | + | |
| 3413 | + | |
| 3414 | + | |
| 3415 | + | |
| 3416 | + | |
| 3417 | + | |
| 3418 | + | |
| 3419 | + | |
| 3420 | + | |
| 3421 | + | |
| 3422 | + | |
| 3423 | + | |
| 3424 | + | |
| 3425 | + | |
| 3426 | + | |
| 3427 | + | |
3356 | 3428 | | |
3357 | 3429 | | |
3358 | 3430 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
270 | 270 | | |
271 | 271 | | |
272 | 272 | | |
273 | | - | |
| 273 | + | |
| 274 | + | |
274 | 275 | | |
275 | 276 | | |
276 | 277 | | |
| |||
280 | 281 | | |
281 | 282 | | |
282 | 283 | | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | | - | |
290 | | - | |
291 | | - | |
292 | | - | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
293 | 293 | | |
294 | 294 | | |
295 | 295 | | |
| |||
0 commit comments