🐛 Describe the bug

The device in function addmm_patterns_init() is assigned with cuda or cpu, which missed xpu, and lead the actual device = xpu to device = cpu.
And after the PR #139048 introduced config.cpp.enable_concat_linear, the addmm_patterns_init() not works for xpu.

Versions

PyTorch version: 2.6.0a0+git8aaf194
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.4
Libc version: glibc-2.35

Python version: 3.9.20 | packaged by conda-forge | (main, Sep 30 2024, 17:49:10) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.15.47+prerelease23.5.18-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

cc @gujinghui @EikanWang @fengyuan14 @guangyey