[CPU][Spec Decode] Warn about throughput loss when libiomp5 is not preloaded by jmamou · Pull Request #44419 · vllm-project/vllm

jmamou · 2026-06-03T12:10:47Z

Summary

Add an explicit warning when speculative decoding is enabled on CPU without libiomp5 in LD_PRELOAD. Without Intel's OpenMP runtime, SD throughput drops by ~2× due to the overhead of GNU libgomp in the frequent short parallel regions that speculative decoding uses.

Motivation

While investigating issue #44191, we discovered that the performance gap between the official Docker image and source builds is entirely explained by libiomp5 being preloaded in the Docker entrypoint but not in typical conda/venv setups.

Benchmark (Xeon 8580, Llama-3.1-8B + Llama-3.2-1B, k=7, BS=1):

Config	SD TPOT	Speedup
Without `libiomp5` in LD_PRELOAD	115.6 ms	1.37×
With `libiomp5` in LD_PRELOAD	58.1 ms	2.74×

The existing generic warning is easy to overlook. This PR adds a targeted warning specifically for the SD + missing iomp case, quantifying the impact and suggesting the fix.

Changes

check_preloaded_libs now returns a bool indicating whether the library was found
When speculative decoding is configured and libiomp is missing, emit an additional warning with the expected impact (~2× throughput loss) and the suggested LD_PRELOAD command
Attempt to locate libiomp5.so via ctypes.util.find_library and include the path in the hint

Test Plan

Verified on Xeon 8580 that the warning fires when launching SD without LD_PRELOAD
Verified that the warning does not fire when libiomp5.so is in LD_PRELOAD
Verified that the warning does not fire for non-SD workloads (even without iomp)
No functional change to inference behavior

mergify · 2026-06-03T12:14:49Z

Documentation preview: https://vllm--44419.org.readthedocs.build/en/44419/

mergify · 2026-06-03T12:15:06Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jmamou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…eloaded Add a targeted warning when speculative decoding is enabled on CPU without libiomp5 in LD_PRELOAD. Without Intel's OpenMP runtime, SD throughput drops by ~2x due to GNU libgomp overhead in the frequent short parallel regions that speculative decoding uses. Discovered while investigating issue vllm-project#44191 where the reported regression was actually caused by missing LD_PRELOAD in source builds vs the Docker image which sets it automatically. Signed-off-by: jmamou <[email protected]>

bigPYJ1151

Thanks for your investigation @jmamou :) I have some suggestions, please check them.
It is surprising the OpenMP version will affect SD performance on CPU, usually the impact is very limited.

jmamou · 2026-06-07T11:41:29Z

This pull request has merge conflicts that must be resolved before it can be merged. Please rebase the PR, @jmamou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@bigPYJ1151

Thanks for the feedback!

I agree it's surprising — for standard inference (large GEMMs dominating runtime), the difference between libiomp5 and libgomp is negligible. But speculative decoding is a different workload pattern: SD performs many very short parallel regions per token — the draft model runs k single-token forward passes, then the target model runs a short verify pass. Each of these is a small computation relative to batch inference. With libgomp, threads spin down between regions (its default blocktime is effectively 0), so every parallel region pays thread wake-up / barrier overhead. With libiomp5 + KMP_BLOCKTIME > 0, threads stay spinning between regions, which eliminates this cost.

In standard (non-SD) inference, a single forward pass is large enough that the fork-join overhead is amortized. In SD, you're doing ~2k+1 short passes per generation step, so the overhead accumulates.

The non-SD baseline is ~159 ms TPOT in both cases (confirming the impact is SD-specific).

Co-authored-by: Li, Jiang <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]>

Signed-off-by: jmamou <[email protected]>

…eloaded (vllm-project#44419) Signed-off-by: jmamou <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Signed-off-by: Ekagra Ranjan <[email protected]>

…eloaded (vllm-project#44419) Signed-off-by: jmamou <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Signed-off-by: Waqar Ahmed <[email protected]>

…eloaded (vllm-project#44419) Signed-off-by: jmamou <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Li, Jiang <[email protected]>

…eloaded (vllm-project#44419) Signed-off-by: jmamou <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Signed-off-by: divineearthly <[email protected]>

…eloaded (vllm-project#44419) Signed-off-by: jmamou <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Li, Jiang <[email protected]>

mergify Bot added documentation Improvements or additions to documentation ci/build deepseek Related to DeepSeek models frontend nvidia rocm Related to AMD ROCm labels Jun 3, 2026

github-project-automation Bot added this to NVIDIA and AMD Jun 3, 2026

mergify Bot added the cpu Related to CPU backends label Jun 3, 2026

github-project-automation Bot moved this to Todo in AMD Jun 3, 2026

mergify Bot added the v1 label Jun 3, 2026

mergify Bot added the needs-rebase label Jun 3, 2026

jmamou force-pushed the cpu-sd-iomp-warning branch from 14d7aaa to c8f8f5f Compare June 3, 2026 12:16

mergify Bot removed the needs-rebase label Jun 3, 2026

jmamou marked this pull request as ready for review June 3, 2026 12:21

jmamou requested review from bigPYJ1151 and xuechendi as code owners June 3, 2026 12:21

bigPYJ1151 reviewed Jun 5, 2026

View reviewed changes

Comment thread vllm/v1/worker/cpu_worker.py Outdated

Comment thread vllm/v1/worker/cpu_worker.py Outdated

jmamou force-pushed the cpu-sd-iomp-warning branch from a3b605e to 5b8f34a Compare June 7, 2026 08:58

jmamou requested review from AndreasKaratzas, WoosukKwon, benchislett, luccafong, mgoin, tlrmchlsmth, yewentao256 and zyongye as code owners June 7, 2026 08:58

jmamou requested review from LucasWilkinson and MatthewBonanni as code owners June 7, 2026 08:58

mergify Bot added the speculative-decoding label Jun 7, 2026

jmamou force-pushed the cpu-sd-iomp-warning branch from 5b8f34a to 2b75fe3 Compare June 7, 2026 11:32

jmamou and others added 3 commits June 7, 2026 14:41

Update vllm/v1/worker/cpu_worker.py

d71a431

Co-authored-by: Li, Jiang <[email protected]> Signed-off-by: Jonathan Mamou <[email protected]>

Address review: use docs link instead of ctypes.util hint

034c704

Signed-off-by: jmamou <[email protected]>

Merge branch 'main' into cpu-sd-iomp-warning

eb3b8b0

bigPYJ1151 approved these changes Jun 8, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Jun 8, 2026

bigPYJ1151 added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 8, 2026

bigPYJ1151 and others added 2 commits June 8, 2026 14:37

Merge branch 'main' into cpu-sd-iomp-warning

368888f

Merge branch 'main' into cpu-sd-iomp-warning

034adf3

bigPYJ1151 enabled auto-merge (squash) June 8, 2026 09:01

bigPYJ1151 merged commit d5fe994 into vllm-project:main Jun 8, 2026
57 checks passed

github-project-automation Bot moved this from Todo to Done in AMD Jun 8, 2026

github-project-automation Bot moved this from Ready to Done in NVIDIA Jun 8, 2026

jmamou deleted the cpu-sd-iomp-warning branch June 8, 2026 10:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[CPU][Spec Decode] Warn about throughput loss when libiomp5 is not preloaded#44419

[CPU][Spec Decode] Warn about throughput loss when libiomp5 is not preloaded#44419
bigPYJ1151 merged 6 commits into
vllm-project:mainfrom
jmamou:cpu-sd-iomp-warning

jmamou commented Jun 3, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Jun 3, 2026

Uh oh!

mergify Bot commented Jun 3, 2026

Uh oh!

bigPYJ1151 left a comment

Uh oh!

Uh oh!

Uh oh!

jmamou commented Jun 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

jmamou commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Test Plan

Uh oh!

mergify Bot commented Jun 3, 2026

Uh oh!

mergify Bot commented Jun 3, 2026

Uh oh!

bigPYJ1151 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jmamou commented Jun 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmamou commented Jun 3, 2026 •

edited

Loading