Repository navigation
Widen dictionary codes in Dict::fill_null - #10806
Conversation
When the fill value is in the dictionary values but its index does not fit the codes type, the Dict fill_null kernel declined and the generic path decoded the whole array. Since the kernel only rewrites the null codes, cast the codes to the smallest unsigned type that holds the fill index and keep the result dictionary-encoded instead. Signed-off-by: Vincent <[email protected]>
Merging this PR will regress 1 benchmark4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Summary
Dict::fill_null#10804Follow-up to #10745, as suggested in review there. When the fill value already exists in the dictionary values but its index does not fit the codes type (e.g.
u8codes and the fill value at index 300), theDictfill_nullkernel returnedNone, so the generic path decoded the whole array before filling. The kernel only rewrites the null codes, so it can widen them instead and keep the result dictionary-encoded.Changes
PType::min_unsigned_ptype_for_value(fill_index)(kept nullable) and then fill the nulls with the index. Codes are never negative, so the cast keeps every code. Codes that already fit and non-nullable codes take the same path as before.fill_unreferenced_value_widens_codescalls the kernel directly and checks that it returns aDictwith widened, non-nullable codes and the right values, foru8 -> u16(256, 300),i8 -> u8(128),i8 -> u16(256) andu16 -> u32(65536). All five cases fail ondevelop, where the kernel returnsNone.Tested with
cargo test -p vortex-array --lib(4437 passed),cargo clippy -p vortex-array --all-targets --all-features -- -D warnings, andcargo +nightly-2026-09-10 fmt -p vortex-array.AI assistance: I used an AI coding assistant to help write this change and the description; I reviewed the code and ran the tests above myself.