Even though based on Discord discussions there is no intention of fixing this, I'm opening an issue for people who may run into it.
We've been observing that when starting an application that immediately receives important load, the zio 2 scheduler is creating a unbounded number of threads, leading to contention at the classloader level as well as loading lazy vals.
A couple pictures first (note the number of ZScheduler threads: 378 😱):
They're all waiting on the same locks (here, a simple lazy val):
What seems to happen is that the first access to a given class or lazy val is slow (as the JVM just started and classes are not loaded yet), so we expect a little latency. But the ZScheduler detects that the thread is "blocking" and decides to create another one. Since we have multiple fibers doing that at the same time, this process repeats multiple times, leading to a thread explosion. What makes it worse is that all those threads are competing for the same locks (see the second picture), so it completely freezes the JVM until those resources are finally loaded. For us it resulted in 10~20 seconds freeze in production when starting to receive load.
As a workaround, we switched to a custom executor based on zio 1 truly bounded executor (using Runtime.setExecutor(Executor.fromThreadPoolExecutor(...)). Because it is bounded, it does not create new threads and with very little contention all the classloading happens rather quickly (<1s).
It might be a rare use case: large application with lots of classes (which makes it difficult to "pre-load"), instant intense usage. But this behavior seems to me like it could cascade easily even in other use cases.
I think it could be nice to have the option to keep a bounded number of threads (basically disabling auto-blocking behavior). Auto-blocking might be nice for users who don't know or don't want to care about what's blocking but I would love to have the possibility to decide where my code runs and keep a bounded threadpool.
Even though based on Discord discussions there is no intention of fixing this, I'm opening an issue for people who may run into it.
We've been observing that when starting an application that immediately receives important load, the zio 2 scheduler is creating a unbounded number of threads, leading to contention at the classloader level as well as loading lazy vals.
A couple pictures first (note the number of
ZSchedulerthreads: 378 😱):They're all waiting on the same locks (here, a simple lazy val):
What seems to happen is that the first access to a given class or lazy val is slow (as the JVM just started and classes are not loaded yet), so we expect a little latency. But the
ZSchedulerdetects that the thread is "blocking" and decides to create another one. Since we have multiple fibers doing that at the same time, this process repeats multiple times, leading to a thread explosion. What makes it worse is that all those threads are competing for the same locks (see the second picture), so it completely freezes the JVM until those resources are finally loaded. For us it resulted in 10~20 seconds freeze in production when starting to receive load.As a workaround, we switched to a custom executor based on zio 1 truly bounded executor (using
Runtime.setExecutor(Executor.fromThreadPoolExecutor(...)). Because it is bounded, it does not create new threads and with very little contention all the classloading happens rather quickly (<1s).It might be a rare use case: large application with lots of classes (which makes it difficult to "pre-load"), instant intense usage. But this behavior seems to me like it could cascade easily even in other use cases.
I think it could be nice to have the option to keep a bounded number of threads (basically disabling auto-blocking behavior). Auto-blocking might be nice for users who don't know or don't want to care about what's blocking but I would love to have the possibility to decide where my code runs and keep a bounded threadpool.