Describe the feature
Wall clock profiling mode (-e wall) is currently implemented by a dedicated thread periodically sampling all application threads, regardless of their status. This means, if an application has 1000 threads and wall clock profiler runs with 100ms interval, profiler will record 10K samples per second, sending 10K signals and writing about 150kB data every second. This may cause notable performance overhead, both in CPU usage (signals are expensive) and in JFR file size.
In wall clock mode, most of the sampled threads are sleeping, e.g., idle thread pools waiting for incoming tasks. Signals will awake them, causing even more overhead by letting kernel manage threads that would be otherwise intact.
We can significantly reduce impact of wall clock profiling if we skip sampling threads that are known to be idle.
Use Case
- Lower wall clock profiling overhead for applications with thousands of threads.
- Make smaller wall clock intervals feasible, thus improving profile accuracy.
- Reduce size of JFR recordings when wall clock mode is enabled.
Proposed Solution
After sampling a thread in IDLE state, record its total CPU usage.
Next time, if CPU usage of this thread has not changed, we will not sample it again, considering the thread stays IDLE at the very same location we've seen it before. We will start sampling this thread again if its CPU time changes or after 1000 skipped iterations, whatever occurs first.
Introduce new JFR event profiler.WallClockSample having all the fields of jdk.ExecutionSample event plus a new int samples field containing a counter for skipped samples. Instead of recording 500 similar ExecutionSample events for a sleeping thread, we will issue only one ExecutionSample event and one WallClockSample event with samples=499.
Amend JfrReader to seamlessly handle WallClockSample events as if there was a single event per sample as before. Nothing should change from user's perspective; all implementation details should be encapsulated.
If for some reason previous behavior is desired, wall clock optimization can be disabled by adding nobatch profiler argument.
Acknowledgements
Describe the feature
Wall clock profiling mode (
-e wall) is currently implemented by a dedicated thread periodically sampling all application threads, regardless of their status. This means, if an application has 1000 threads and wall clock profiler runs with 100ms interval, profiler will record 10K samples per second, sending 10K signals and writing about 150kB data every second. This may cause notable performance overhead, both in CPU usage (signals are expensive) and in JFR file size.In wall clock mode, most of the sampled threads are sleeping, e.g., idle thread pools waiting for incoming tasks. Signals will awake them, causing even more overhead by letting kernel manage threads that would be otherwise intact.
We can significantly reduce impact of wall clock profiling if we skip sampling threads that are known to be idle.
Use Case
Proposed Solution
After sampling a thread in IDLE state, record its total CPU usage.
Next time, if CPU usage of this thread has not changed, we will not sample it again, considering the thread stays IDLE at the very same location we've seen it before. We will start sampling this thread again if its CPU time changes or after 1000 skipped iterations, whatever occurs first.
Introduce new JFR event
profiler.WallClockSamplehaving all the fields ofjdk.ExecutionSampleevent plus a newint samplesfield containing a counter for skipped samples. Instead of recording 500 similarExecutionSampleevents for a sleeping thread, we will issue only oneExecutionSampleevent and oneWallClockSampleevent with samples=499.Amend
JfrReaderto seamlessly handleWallClockSampleevents as if there was a single event per sample as before. Nothing should change from user's perspective; all implementation details should be encapsulated.If for some reason previous behavior is desired, wall clock optimization can be disabled by adding
nobatchprofiler argument.Acknowledgements