Support CgroupsV2 in info.cpp's calculations#2519
Conversation
|
Still working on building the code locally, I'd also like to get some info about how to better test it :) |
|
I am able to build the Python wheel via |
|
Please, agree to the CLA. |
|
|
@andrey-khropov Hi! Thanks a lot for the feedback! I have a couple of questions:
|
|
I am still trying to figure out how to run tests (reading https://catboost.ai/en/docs/concepts/development-and-contributions#run-tests), any suggestion? I tried: But I can't find anything that is related to the info.cpp ut test file.. |
Somehow CMake build for unittests in |
|
Thanks! I'll concentrate on finding a good unit test for the new code :) |
|
I don't have a better idea to test this than to run a special unit test that prints the number of cpus with different container specs. It will be a one-off, but it is inline with the current tests for util/info. If there is a way to mock some sysfs file contents in the unit tests we may be able to have something better, but I don't have any expertise in c++ testing so I'd need some guidance. |
|
Built the static library of catboost and used the following to test cgroups v2: On Debian Bookworm (default is using cgroups v2) I tested:
|
|
I agree with the CLA outlined in https://yandex.ru/legal/cla/?lang=en |
andrey-khropov
left a comment
There was a problem hiding this comment.
Please, change the names of the local variables to camelCase style according to the C++ style guide
97602e0 to
356eb3b
Compare
|
@andrey-khropov thanks! Sent a new patch with variables in camel case and the ) on the previous line. Re-tested as explained above as well. |
|
@andrey-khropov thanks a lot for the approval, what is the next step? Should I help/work on #2525 ? |
There is currently support for CPUs count approximation for Cgroups V1, but not for Cgroups V2. This change adds a similar heuristic as dmlc/xgboost#9622, so that the count doesn't fall back to /proc/cpuinfo on more recent systems. issue: catboost#2518
|
Gentle ping, not sure what to do folks :) |
I've discovered some strange buffer underflow issue when I tried to merge this PR but it is not related to your changes directly, I'm investigating. You don't have to do anything for now. |
Thanks a lot! I asked since I thought the code may have needed more reviewers, and where I work (Wikimedia foundation) we are trying to solve a k8s performance bug when running catboost with cgroups v2. Nothing urgent but ideally we'll love to work with you to have a new minor release in Pypi with the patch to fix the issue :) |
Yes, CatBoost 1.2.3 that is expected to contain this change is planned to be released this week. |
|
@andrey-khropov Hi! Is there anything still pending or shall we merge this? Thanks in advance :) |
…oid exceptions under normal operation
…from catboost/catboost#2519 Pull Request resolved: catboost/catboost#2519
…from catboost/catboost#2519 Pull Request resolved: catboost/catboost#2519
…from catboost/catboost#2519 Pull Request resolved: catboost/catboost#2519
There is currently support for CPUs count approximation for Cgroups V1, but not for Cgroups V2.
This change adds a similar heuristic as
dmlc/xgboost#9622, so
that the count doesn't fall back to /proc/cpuinfo
on more recent systems.
issue: #2518
Before submitting a pull request, please do the following steps:
ya makein catboost folder to make sure the code builds.ya make -t -Acommand.