Repository navigation
Commit 1a9ed68
authored
perf: enable POPCNT in the BMI2, AVX2 and AVX-512 kernels (#10456)
## Summary
`#[target_feature(enable = "bmi2")]`, `"avx2"` and `"avx512f,..."` do
not imply POPCNT, so `count_ones()` inside these kernels compiles to a
SWAR sequence (with `imul` for narrow masks) in any build without
`target-cpu`. The bench runners use `target-cpu=native`, so they won't
show a difference.
## Changes
Add `popcnt` to the target features and to the runtime detection of the
`intersect_by_rank` BMI2 kernels and the x86 `simd_compress` kernels. No
other `#[target_feature]` function in the tree counts bits without it
(the `fearless_simd` levels already enable POPCNT).
Default build, Zen 5, divan medians:
| bench | before | after | |
|---|---|---|---|
| `intersect_by_rank` 100k random | 3.17 µs | 1.93 µs | 1.64x |
| `random_rotating` 0.5/0.5 | 9.71 µs | 5.67 µs | 1.71x |
| `very_sparse_mask_cached` 10%/1% | 1.91 µs | 0.81 µs | 2.36x |
| `filter_fixed_width` `random_i16` 0.8 | 1.60 µs | 1.06 µs | 1.51x |
| `filter_fixed_width` `random_i32` 0.5 | 1.49 µs | 1.03 µs | 1.45x |
| `filter_fixed_width` `random_i64` 0.5 | 1.13 µs | 0.99 µs | 1.14x |
| `filter_fixed_width` `random_i128` 0.5 (no kernel) | 1.45 µs | 1.42 µs
| 1.02x |
The filter numbers are the AVX-512 kernels; the AVX2 ones should show up
in the CodSpeed simulation shard, where AFAIK valgrind has no AVX-512.
Signed-off-by: mightsleep <[email protected]>1 parent fdba755 commit 1a9ed68
4 files changed
Lines changed: 28 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| |||
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
112 | | - | |
| 112 | + | |
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
| |||
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
129 | | - | |
| 129 | + | |
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
137 | | - | |
| 137 | + | |
138 | 138 | | |
139 | 139 | | |
140 | 140 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
| 68 | + | |
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
| 75 | + | |
75 | 76 | | |
76 | 77 | | |
77 | 78 | | |
78 | | - | |
| 79 | + | |
79 | 80 | | |
80 | 81 | | |
81 | 82 | | |
| |||
187 | 188 | | |
188 | 189 | | |
189 | 190 | | |
190 | | - | |
| 191 | + | |
191 | 192 | | |
192 | 193 | | |
193 | 194 | | |
| |||
200 | 201 | | |
201 | 202 | | |
202 | 203 | | |
203 | | - | |
| 204 | + | |
204 | 205 | | |
205 | 206 | | |
206 | 207 | | |
| |||
213 | 214 | | |
214 | 215 | | |
215 | 216 | | |
216 | | - | |
| 217 | + | |
217 | 218 | | |
218 | 219 | | |
219 | 220 | | |
| |||
226 | 227 | | |
227 | 228 | | |
228 | 229 | | |
229 | | - | |
| 230 | + | |
230 | 231 | | |
231 | 232 | | |
232 | 233 | | |
| |||
329 | 330 | | |
330 | 331 | | |
331 | 332 | | |
332 | | - | |
| 333 | + | |
333 | 334 | | |
334 | 335 | | |
335 | 336 | | |
| |||
409 | 410 | | |
410 | 411 | | |
411 | 412 | | |
412 | | - | |
| 413 | + | |
413 | 414 | | |
414 | 415 | | |
415 | 416 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
49 | | - | |
| 48 | + | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
62 | | - | |
| 61 | + | |
| 62 | + | |
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
74 | | - | |
| 73 | + | |
| 74 | + | |
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
443 | 443 | | |
444 | 444 | | |
445 | 445 | | |
446 | | - | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
447 | 449 | | |
448 | 450 | | |
449 | 451 | | |
| |||
460 | 462 | | |
461 | 463 | | |
462 | 464 | | |
463 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
464 | 468 | | |
465 | 469 | | |
466 | 470 | | |
| |||
523 | 527 | | |
524 | 528 | | |
525 | 529 | | |
526 | | - | |
527 | | - | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
528 | 533 | | |
529 | 534 | | |
530 | 535 | | |
| |||
0 commit comments