When running XGBoost in a containerized environment, where cgroups v1 is used, the library is correctly aware of the CPU limits set by the cgroup and spawns the appropriate amount of threads to do its calculations.
But when running in an environment where cgroups v2 is used, XGBoost no longer recognized the limits. Instead, it's using the host machine's core count as a reference for how many threads to spawn. This almost definitely results in needless context switching. In one of our cases, we are seeing about ~300 times slower processing.
I believe this is the code that does not work with the new cgroups API:
|
auto const cfs_quota(read_int("/sys/fs/cgroup/cpu/cpu.cfs_quota_us")); |
|
auto const cfs_period(read_int("/sys/fs/cgroup/cpu/cpu.cfs_period_us")); |
As a comparison, in a simple python:3.11-based container running in the cgroups-v2-environment, the "/sys/fs/cgroup" structure looks like this (I've removed the non-cpu "files" for readability):
/sys/fs/cgroup/cpu.idle
/sys/fs/cgroup/cpu.max
/sys/fs/cgroup/cpu.max.burst
/sys/fs/cgroup/cpu.pressure
/sys/fs/cgroup/cpu.stat
/sys/fs/cgroup/cpu.uclamp.max
/sys/fs/cgroup/cpu.uclamp.min
/sys/fs/cgroup/cpu.weight
/sys/fs/cgroup/cpu.weight.nice
/sys/fs/cgroup/cpuset.cpus
/sys/fs/cgroup/cpuset.cpus.effective
/sys/fs/cgroup/cpuset.cpus.partition
/sys/fs/cgroup/cpuset.mems
/sys/fs/cgroup/cpuset.mems.effective
For now, we have solved our issue by manually setting the OMP_NUM_THREADS and OMP_THREAD_LIMIT environment variables to the number of threads appropriate for our use case. But it would be nice if XGBoost can once again handle this on its own.
When running XGBoost in a containerized environment, where cgroups v1 is used, the library is correctly aware of the CPU limits set by the cgroup and spawns the appropriate amount of threads to do its calculations.
But when running in an environment where cgroups v2 is used, XGBoost no longer recognized the limits. Instead, it's using the host machine's core count as a reference for how many threads to spawn. This almost definitely results in needless context switching. In one of our cases, we are seeing about ~300 times slower processing.
I believe this is the code that does not work with the new cgroups API:
xgboost/src/common/threading_utils.cc
Lines 34 to 35 in 4d7a187
As a comparison, in a simple python:3.11-based container running in the cgroups-v2-environment, the "/sys/fs/cgroup" structure looks like this (I've removed the non-cpu "files" for readability):
For now, we have solved our issue by manually setting the OMP_NUM_THREADS and OMP_THREAD_LIMIT environment variables to the number of threads appropriate for our use case. But it would be nice if XGBoost can once again handle this on its own.