Configure udevd children-max for root-test#13562
Conversation
GHA runners occasionally experience I/O constraints during root-test test execution. While concurrent tests rapidly allocate loopback devices, background udev probing stalls. This quickly exhausts systemd-udevd's default worker pool ceiling (20 children max), stalling netlink uevent processing so device-mapper device nodes are never created for subsequent dm-verity test execution. Logging cgroups v2 pids.peak telemetry confirmed peak in-flight udev workers accumulate to 325 during test execution. Raising the children-max limit to 500 provides comfortable buffer room so udevd freely spawns worker processes without entering event lockup or causing test timeouts. Assisted-by: Antigravity Signed-off-by: Chris Henzie <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR adjusts the Linux GitHub Actions integration workflow to prevent systemd-udevd worker pool exhaustion during root-test, which can stall uevent processing and cause dm-verity-related tests to time out on I/O-constrained runners.
Changes:
- Increase the runtime
systemd-udevdworker ceiling in CI by settingudevadm control --children-max=500before running tests.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
github-project-automation
Bot
moved this from Needs Triage
to Review In Progress
in Pull Request Review
github-project-automation
Bot
moved this from Review In Progress
to Done
in Pull Request Review
|
/cherrypick release/2.3 |
|
@chrishenzie: new pull request created: #13564 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@chrishenzie: new pull request created: #13565 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@chrishenzie: new pull request created: #13566 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@chrishenzie: new pull request created: #13567 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@chrishenzie: new pull request created: #13568 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
samuelkarp
added
cherry-pick/1.7.x
samuelkarp
added
cherry-pick/2.2.x
GHA runners occasionally experience I/O constraints during root-test test execution. While concurrent tests rapidly allocate loopback devices, background udev probing stalls. This quickly exhausts systemd-udevd's default worker pool ceiling (20 children max), stalling netlink uevent processing so device-mapper device nodes are never created for subsequent dm-verity test execution.
Logging cgroups v2 pids.peak telemetry confirmed peak in-flight udev workers accumulate to 325 during test execution. Raising the children-max limit to 500 provides comfortable buffer room so udevd freely spawns worker processes without entering event lockup or causing test timeouts.
Assisted-by: Antigravity