Repository navigation
substr on Dictionary(_, Utf8) inserts implicit CAST to Utf8View #26170
Description
substr on a dictionary-encoded string column plans an implicit cast to Utf8View, materializing the dictionary before the scalar function runs.
Repro (datafusion-cli 54.0.0):
CREATE TABLE t AS
SELECT arrow_cast('hello' || i::VARCHAR, 'Dictionary(Int32, Utf8)') AS k
FROM generate_series(1, 10) AS s(i);
EXPLAIN SELECT substr(k, 1, 4) FROM t;Plan:
ProjectionExec: expr=[substr(CAST(k@0 AS Utf8View), 1, 4) as substr(t.k,Int64(1),Int64(4))]
DataSourceExec: ...
The arrow-rs substring kernel already handles this by delegating to dictionary.values() and reusing the keys (arrow-string/src/substring.rs:79-82). SubstrFunc's signature in datafusion/functions/src/unicode/substr.rs doesn't advertise Dictionary support, so type coercion inserts the cast. This should be avoided for performance!
Metadata
Metadata
Assignees
Labels
No labels
Activity
Rich-T-kid commentedon Oct 9, 2026
take
perf: preserve dictionary encoding in substr
perf: preserve dictionary encoding in substr
perf: preserve dictionary encoding in substr
perf: preserve dictionary encoding in substr
perf: opt substr into make_scalar_function dict fast path (apache#25874)