Repository navigation
Commit 92ad99e
fix: keep the dictionary hash fast path off nested and reseeded buffers (#5757)
The dictionary fast path hashes each distinct dictionary value once and reuses that
result for every key. It was selected by the column's position, `i == 0`, and it
restarted from a hardcoded seed of 42.
Both parts are wrong. `create_hashes_internal!` also runs on recursion, so a
dictionary nested in a list, struct or map arrives as the only column of its call and
looks like a first column even though the buffer already holds the hash accumulated
for earlier elements of that row. That hash was discarded, and a dictionary-encoded
list element hashed differently from the identical decoded value. Separately, the
hardcoded 42 is wrong whenever the caller supplies its own seed, as `hash(col, seed)`
and `xxhash64(col, seed)` allow, so even a genuine first column disagreed with its
decoded form for a non-default seed.
The reuse is valid exactly when every row carries the same incoming hash, so that is
what is now checked, and the per-value hashes start from the seed the buffer actually
holds rather than an assumed 42. A top-level dictionary keeps the optimisation.
The uniformity check is a scan of the hash buffer, which is measurable: running it for
every column cost 17% on an int column and 11% on a string column in a local
criterion benchmark. It is therefore done inside the dictionary arm, so only
dictionary columns pay it and other types are untouched.
Both hash implementations share this structure and both are fixed, with regression
tests that fail without the change: a dictionary as a list element hashes 3853467749
rather than the 1401423033 of the decoded data.
The single-row cases above pin the hardcoded seed but not the uniformity check, since
one row is uniform by definition. Each algorithm therefore also gets a multi-row case
whose incoming seeds all differ, which forces the unpacking fallback, and which
includes a null key and a key pointing at a null dictionary value. Dropping the
uniformity check leaves the other twenty hash tests green and fails exactly those two.
Co-authored-by: Claude Code <[email protected]>1 parent 7f1e001 commit 92ad99e
3 files changed
Lines changed: 183 additions & 13 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
142 | | - | |
| 142 | + | |
143 | 143 | | |
144 | 144 | | |
145 | | - | |
| 145 | + | |
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
150 | | - | |
151 | | - | |
152 | | - | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
153 | 153 | | |
154 | | - | |
155 | | - | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
156 | 159 | | |
157 | 160 | | |
158 | 161 | | |
| |||
205 | 208 | | |
206 | 209 | | |
207 | 210 | | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
208 | 305 | | |
209 | 306 | | |
210 | 307 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
576 | 576 | | |
577 | 577 | | |
578 | 578 | | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
579 | 588 | | |
580 | 589 | | |
581 | 590 | | |
| |||
729 | 738 | | |
730 | 739 | | |
731 | 740 | | |
732 | | - | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
733 | 748 | | |
734 | 749 | | |
735 | 750 | | |
| |||
788 | 803 | | |
789 | 804 | | |
790 | 805 | | |
791 | | - | |
| 806 | + | |
| 807 | + | |
792 | 808 | | |
793 | 809 | | |
794 | 810 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
99 | | - | |
100 | | - | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
101 | 104 | | |
102 | 105 | | |
103 | 106 | | |
| |||
151 | 154 | | |
152 | 155 | | |
153 | 156 | | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
154 | 211 | | |
155 | 212 | | |
156 | 213 | | |
| |||
0 commit comments