Skip to content

Commit 9beb5db

Browse files
committed
Inline the multiversioned bit packing loop in static AVX-512 builds
A static AVX-512 build inlines collect_bool_words_avx512 only when rustc emits that copy in the caller's codegen unit. This branch moved it out of the RowFn deferred Boolean kernel's unit, adding 15% more instructions per row. Such a build has nothing wider to detect, so use collect_bool_words_inline, which already packs with AVX-512. Signed-off-by: Connor Tsui <[email protected]>
1 parent 0f446f7 commit 9beb5db

1 file changed

Lines changed: 13 additions & 1 deletion

File tree

‎vortex-buffer/src/bit/pack.rs‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,9 @@ where
111111
/// conversions, or unchecked slice comparisons like the `between` kernels — where the fused
112112
/// AVX-512 loop is worth another ~2x over the baseline kernel.
113113
///
114+
/// A build with AVX-512F and AVX-512BW enabled at compile time uses `collect_bool_words_inline`
115+
/// instead, which already packs with AVX-512.
116+
///
114117
/// `words` must hold at least `len.div_ceil(64)` entries and `f` is invoked with `0..len`,
115118
/// exactly once per index in ascending order.
116119
///
@@ -133,7 +136,16 @@ where
133136
return collect_bool_words_inline(words, len, f);
134137
}
135138

136-
#[cfg(all(target_arch = "x86_64", not(miri)))]
139+
// NB: A static AVX-512 build must skip the `#[target_feature]` copies, which add nothing there.
140+
// Each copy is a generic function without `#[inline]`, so rustc emits it in one codegen unit,
141+
// and LLVM can inline it only into callers in that unit. So an unrelated change can put a hot
142+
// caller's packing loop behind a call, as happened to the deferred Boolean RowFn loop in the
143+
// bench profile (16 codegen units, 15% more instructions per row).
144+
#[cfg(all(
145+
target_arch = "x86_64",
146+
not(all(target_feature = "avx512f", target_feature = "avx512bw")),
147+
not(miri)
148+
))]
137149
{
138150
if is_x86_feature_detected!("avx512f") && is_x86_feature_detected!("avx512bw") {
139151
// SAFETY: runtime detection guarantees the required target features.

0 commit comments

Comments
 (0)