perf: Optimize task queue communication from engine to worker #4531
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
|
|
sunlei1024
force-pushed
the
perf/engine_work_queue
branch
from
8ee1bab to
c30b772
Compare
kevincheng2
pushed a commit
to kevincheng2/FastDeploy
that referenced
this pull request
Oct 27, 2025
…Paddle#4531) * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <[email protected]>
Jiang-Jia-Jun
added a commit
that referenced
this pull request
Oct 27, 2025
* support mm prefix caching * update code * fix mm_hashes * support encoder cache * add encoder cache * update code * update encoder cache * fix features bug * fix worker bug * support processor cache, need to optimize yet * refactor multimodal data cache * update code * update code * update v1 scheduler * update code * update code * update codestyle * support turn off processor cache and encoder cache * update pre-commit * fix code * solve review * update code * update code * update test case * set processor cache in GiB * update test case * support mm prefix caching for qwen model * fix code style check * update pre-commit * fix unit test * fix unit test * add ci test case * fix rescheduled bug * change text_after_process to prompt_tokens * fix unit test * fix chat template * change model path * [EP] fix adapter bugs (#4572) * Update expert_service.py * Update common_engine.py * Update expert_service.py * fix v1 hang bug (#4573) * fix import image_ops error on some platforms (#4559) * [CLI]Update parameters in bench latecy cli tool and fix collect-env cli tool (#4558) * add collect-env * del files * [Graph Optimization] Add dy_runnable and introduce cudagraph_switch_threshold for cudagraph mode switching (#4578) * add new branch for sot * reorder * fix batch bug * [XPU]Moe uses a new operator (#4585) * [XPU]Moe uses a new operator * [XPU]Moe uses a new operator * update response * [Feature] Support Paddle-OCR (#4396) * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <[email protected]> Co-authored-by: zhangyue66 <[email protected]> * [DataProcessor] add reasoning_tokens into usage info (#4520) * add reasoning_tokens into usage info initial commit * add unit tests * modify unit test * modify and add unit tests * fix unit test * move steam usage to processor * modify processor * modify test_logprobs * modify test_logprobs.py * modify stream reasoning tokens accumulation * fix unit test * perf: Optimize task queue communication from engine to worker (#4531) * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <[email protected]> * Clean up ports after processing results (#4587) * [CI] Add /re-run command in PR comments to restart failed CI workflows (#4593) * [Others] api server exits when worker process is dead (#3271) * [fix] fix terminal hangs when worker process is dead * [chore] change sleep time of monitor * [chore] remove redundant comments * update docs --------- Co-authored-by: ApplEOFDiscord <[email protected]> Co-authored-by: ApplEOFDiscord <[email protected]> Co-authored-by: ltd0924 <[email protected]> Co-authored-by: yinwei <[email protected]> Co-authored-by: JYChen <[email protected]> Co-authored-by: qwes5s5 <[email protected]> Co-authored-by: Ryan <[email protected]> Co-authored-by: yyssys <[email protected]> Co-authored-by: ming1753 <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: zhangyue66 <[email protected]> Co-authored-by: kxz2002 <[email protected]> Co-authored-by: SunLei <[email protected]> Co-authored-by: Jiang-Jia-Jun <[email protected]> Co-authored-by: Zhang Yulong <[email protected]> Co-authored-by: YuBaoku <[email protected]> Co-authored-by: 李泳桦 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
优化 Engine 与 Worker 之间的任务队列通信，通过在任务调度中（put_tasks & get_tasks）将多模态输入（图像）在 NumPy 数组和 PaddlePaddle Tensor 之间转换，降低通信成本，提高多模态数据处理性能。
Modifications
to_tensor静态方法：当FD_ENABLE_MAX_PREFILL开启时，将任务中的 NumPy 图像数组转换为 PaddlePaddle Tensor。to_numpy静态方法：当FD_ENABLE_MAX_PREFILL开启时，将任务中的 PaddlePaddle Tensor 转回 NumPy 数组，以便在结果处理前使用。Usage or Command
FD_ENABLE_MAX_PREFILL开启时，这些方法会自动应用。Accuracy Tests（准确性测试）