Pull Request Summary

vLLM 0.2.5 is not pinned on torch or xformers and torch 2.2 caused things to break https://github.com/vllm-project/vllm/blob/v0.2.5/requirements.txt#L8

Test Plan and Usage Guide

already running in prod

naive question: why not just bump vllm version instead of adding pins for xformers, etc? Is this also an issue in newer vllm images like 0.3.2? And if so, do we need to consider this for other packages as well?

Pull Request Summary

vLLM 0.2.5 is not pinned on torch or xformers and torch 2.2 caused things to break https://github.com/vllm-project/vllm/blob/v0.2.5/requirements.txt#L8

Test Plan and Usage Guide

already running in prod

naive question: why not just bump vllm version instead of adding pins for xformers, etc? Is this also an issue in newer vllm images like 0.3.2? And if so, do we need to consider this for other packages as well?

yes i'd like to bump vLLM to 0.3.2, but since batch inference code references model engine code there's pydantic conflict. i'll resolve that later. want to get current improvements in first