Skip to content

runtime: add sysmon#4407

Closed
hidva wants to merge 1 commit intotokio-rs:masterfrom
hidva:master
Closed

runtime: add sysmon#4407
hidva wants to merge 1 commit intotokio-rs:masterfrom
hidva:master

Conversation

@hidva
Copy link
Copy Markdown
Contributor

@hidva hidva commented Jan 17, 2022

Motivation

At present, there may be a few CPU-bound tasks in our system occasionally. These tasks may execute for about 100ms. If all worker threads are executing these tasks, they will block other small tasks.

Solution

In this PR, we implemented sysmon introduced by golang. This will greatly reduce the latency of small tasks without increasing CPU usage.

min latency(ns) 90% 95% 99% 99.9% 99.99%
tokio-sysmon-without-takset
small task
141370 5895199 7007711 14243199 21968767 35083263
tokio-sysmon-taskset
small task
140634 8958271 13051967 17270271 26251775 55513087
tokio small task 141328 57064191 58398207 60379391 103762431 306423807
tokio-sysmon-without-takset
big task
55069952 59975423 60470527 61988607 81391103 329990143
tokio-sysmon-taskset
big task
55195136 125155327 135566335 160543743 184667135 363567103
tokio big task 55031552 61137151 62021631 106275839 114610175 329254911

We use workload.rs and workload-server.rs to simulate small tasks, big tasks.

tokio-sysmon-taskset limits the CPU usage by using taskset, tokio-sysmon-taskset and tokio are using the same CPU usage, ~400%. tokio-sysmon-without-takset doesn't use taskset, its CPU usage is about 553%.

@Darksonn Darksonn added A-tokio Area: The main tokio crate M-runtime Module: tokio/runtime labels Jan 17, 2022
@Darksonn
Copy link
Copy Markdown
Member

This particular trick has previously been discussed in this blog post, where we said that we would not implement this.

@hidva hidva closed this Jan 17, 2022
@hidva
Copy link
Copy Markdown
Contributor Author

hidva commented Jan 17, 2022

This particular trick has previously been discussed in this blog post, where we said that we would not implement this.

Thanks for your reply. I have read this article before, and I still want to try the effect of sysmon. According to the data obtained from this attempt, I have a better understanding of the reason mentioned in the article. It seems that we really don't need to introduce sysmon. Sorry to bother you, I should discuss it in Discussions first.

@carllerche
Copy link
Copy Markdown
Member

I’m not against a sysmon style strategy. What is key is that it isn’t applied against everything. What we could investigate is an annotation (“might_block”), which opts into the sysmon. We also need to think about how to deal with blocking the rest of the task (e.g. select).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-tokio Area: The main tokio crate M-runtime Module: tokio/runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants