Repository navigation
regex_match_dyn (array-array path) does not support Dictionary-encoded needle arrays (SIMILAR TO / ~ operators) #23709
Description
Describe the bug
The regex kernels' array-array path (regex_match_dyn) does not support Dictionary(_, Utf8)-encoded needle arrays, so SIMILAR TO (and the ~ regex operators) fail with an internal error when the value side is dictionary-encoded and the pattern is a non-scalar array:
CREATE TABLE t AS SELECT * FROM (VALUES ('user auth failed')) v(s);
CREATE TABLE p AS SELECT * FROM (VALUES ('(auth|login)')) v(pat);
SELECT arrow_cast(t.s, 'Dictionary(Int32, Utf8)') SIMILAR TO p.pat FROM t CROSS JOIN p;Internal error: Data type Dictionary(Int32, Utf8) not supported for regex_match_dyn on string array.
Expected behavior
The query should evaluate successfully (return true for this data), like the equivalent LIKE query already does:
SELECT arrow_cast(t.s, 'Dictionary(Int32, Utf8)') LIKE p.pat FROM t CROSS JOIN p;
-- returns trueAdditional context
- The scalar fast path (regex_match_dyn_scalar) already handles Dictionary arrays (see datafusion/physical-expr/src/expressions/binary/kernels.rs), and the LIKE kernels handle dictionary needles in the array-array path, so this is a gap specific to the regex kernels' array-array dispatch.
- Suggested fix: add a DataType::Dictionary arm to regex_match_dyn that matches over the dictionary value type (mirroring the scalar path / like kernels).
- Surfaced during review of fix: coerce SIMILAR TO operands to a common string type #23704 (which fixed the missing type coercion for SIMILAR TO and turned the former panic paths into this proper error). Not a regression from that PR — the limitation is pre-existing.
Activity
Jithendra2608 commentedon Jul 20, 2026
take
u70b3 commentedon Jul 20, 2026
Hi @Jithendra2608 — that was quick, thanks for picking this up! I had originally planned to implement this one myself as a follow-up to #23704 (where the idea came up during review), but I'm glad to see it in good hands — it's all yours. If any context from that discussion would be useful, feel free to ping me. Good luck, looking forward to the PR!
Jithendra2608 commentedon Jul 20, 2026
Hi @u70b3 , thank you for the warm welcome and the excellent context in the issue description!
I actually just finished putting together the implementation and opened the PR here: #23722. I mirrored the dictionary unpacking logic from the scalar path exactly as you suggested.
Since you have the context from #23704, I would absolutely love your feedback or review on the PR whenever you have a free moment. Thanks again!
adriangb commentedon Jul 21, 2026
@u70b3 review on other PRs is very welcome - it's our biggest bottleneck and a great way to get involved in the community
u70b3 commentedon Jul 21, 2026
Status update from reviewing #23722: this repro is already fixed at the SQL level on current main — as a side effect of #23704, which added
regex_coercionforExpr::SimilarToin the analyzer. The dictionary value side now gets a decode cast before planning reaches the regex kernel:I verified the original repro above returns
trueon main (c111f0a) without any further patch. The same holds for~/~*/!~*and the regexp UDFs, which were already coerced via casts.What remains is kernel-level robustness: physical plans that bypass the analyzer (hand-built
BinaryExpr, FFI/Substrait consumers) can still deliver aDictionaryarray toregex_match_dynand hit theinternal_err!reported here. That is the scope #23722 now covers — details in my review there. So this issue can be closed either way: as fixed by #23704, or once #23722 lands with the arm covered by a direct unit test.fix: support Dictionary arrays in regex_match_dyn (apache#23709)
Jithendra2608 commentedon Jul 21, 2026
Hi @u70b3, thank you for the incredible review and for providing the unit test snippet!
I have updated the PR with your feedback:
1.Restored the removed doc-comments (which also fixed the cargo fmt issue).
2.Cleaned up the take import.
3.Added the direct unit test for the Dictionary arm.
4.Rebased the branch onto the latest main.
fix: support Dictionary arrays in regex_match_dyn (apache#23709)
fix: support Dictionary arrays in regex_match_dyn (apache#23709)