src: add new metrics APIs #3749

trevnorris · 2022-09-14T23:37:46Z

src: add new metrics APIs

The following metrics are now always recorded and available via the new
uv_metrics_info() API.

* loop_count: Number of event loop iterations.
* events: Total number of events processed by the event handler.
* events_waiting: Total number of events waiting in the event queue when
  the event provider request was made.

Benchmarking has shown no noticeable impact recording these metrics.

This is a patch we are currently using internally at NodeSource. We have been testing more reliable ways to measure the health of the event loop and have found that tracking the number of events processed, and more importantly, the number of events waiting in the event queue to be processed before the event provider is called is a better measure of event loop health.

Allowing a callback to be called upon completion of the event loop iteration, we can measure how long the individual iteration took. Using this we can calculate the statistical average amount of time each event waited. Not only events that were received by the event handler, but also how long each event waited in the event queue.

As an example, here is a graph that plots number of requests (req/sec), total number of events (poll_events), number of event loop iterations that had events waiting to be handled (loop_w_e), and the total number of events waiting (events_waiting):

This curvature is similar to almost every HTTP server workload. Using this data we can determine the optimal time to scale the server without knowing the actual processing time the event loop has taken.

In addition to this data we use an exponential moving average using the Poisson window function combined with a time constant adjustment so even though the calculations are done once every loop iteration, the smoothing curve acts as if it was time-series data. The equation for this is:

                       /-ΔT \
                       |----|
                       \ τ  /
s   =  s      + (1 - e       ) * (x  - s     )
 t      (t-1)                      t    (t-1)

ΔT - the difference in time since the last calculation in seconds
τ  - the time constant for the calculation in seconds
x  - new value to add to the exponential rolling avg

Using the moving average data can be reliably stored in a time series database and used for analysis of the health of an application over more than one process.

Hopefully, these advantages outweigh the burden of adding additional metrics APIs.

test/test-metrics.c

docs/src/metrics.rst

src/uv-common.c

src/uv-common.h

trevnorris · 2022-09-19T20:38:20Z

@cjihrig Thanks for the review. Fixed all comments.

trevnorris · 2022-09-19T20:51:50Z

I'm converting this to a draft and rethinking the implementation to make things simpler. Will ping when the update is complete.

The following metrics are now always recorded and available via the new uv_metrics_info() API. * loop_count: Number of event loop iterations. * events: Total number of events processed by the event handler. * events_waiting: Total number of events waiting in the event queue when the event provider request was made. Benchmarking has shown no noticeable impact recording these metrics. PR-URL: libuv#3749

trevnorris · 2022-09-20T21:01:18Z

@cjihrig @santigimeno I've redone the implementation and simplified the API along with adding more tests. PTAL.

src/win/core.c

test/test-metrics.c

trevnorris · 2022-09-21T16:02:43Z

@santigimeno Everything's taken care of.

include/uv.h

The following metrics are now always recorded and available via the new uv_metrics_info() API. * loop_count: Number of event loop iterations. * events: Total number of events processed by the event handler. * events_waiting: Total number of events waiting in the event queue when the event provider request was made. Benchmarking has shown no noticeable impact recording these metrics. PR-URL: libuv#3749

cjihrig

LGTM with one minor comment.

The GitHub actions CI is green. I also started a Jenkins CI run: https://ci.nodejs.org/view/libuv/job/libuv-test-commit/2258/

EDIT: None of the failures look related to this PR.

include/uv.h

trevnorris · 2022-10-03T22:48:35Z

Looks like several errors are from the build failing to find stdatomic.h.

The following metrics are now always recorded and available via the new uv_metrics_info() API. * loop_count: Number of event loop iterations. * events: Total number of events processed by the event handler. * events_waiting: Total number of events waiting in the event queue when the event provider request was made. Benchmarking has shown no noticeable impact recording these metrics. PR-URL: libuv#3749

santigimeno · 2022-11-11T11:12:37Z

CI: https://ci.nodejs.org/view/libuv/job/libuv-test-commit/2260/

santigimeno · 2022-11-11T15:21:47Z

CI failures unrelated: #3714

- linux: introduce io_uring support libuv/libuv#3952 - src: add new metrics APIs libuv/libuv#3749 - unix,win: give thread pool threads an 8 MB stack libuv/libuv#3787 - win,unix: change execution order of timers libuv/libuv#3927 - Fixes: nodejs#43931 - Fixes: nodejs#42496

- linux: introduce io_uring support libuv/libuv#3952 - src: add new metrics APIs libuv/libuv#3749 - unix,win: give thread pool threads an 8 MB stack libuv/libuv#3787 - win,unix: change execution order of timers libuv/libuv#3927 Fixes: nodejs#43931 Fixes: nodejs#42496 Fixes: nodejs#47715 Fixes: nodejs#47259 Fixes: nodejs#47241

- linux: introduce io_uring support libuv/libuv#3952 - src: add new metrics APIs libuv/libuv#3749 - unix,win: give thread pool threads an 8 MB stack libuv/libuv#3787 - win,unix: change execution order of timers libuv/libuv#3927 Fixes: #43931 Fixes: #42496 Fixes: #47715 Fixes: #47259 Fixes: #47241 PR-URL: #48078 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Jiawen Geng <[email protected]>

- linux: introduce io_uring support libuv/libuv#3952 - src: add new metrics APIs libuv/libuv#3749 - unix,win: give thread pool threads an 8 MB stack libuv/libuv#3787 - win,unix: change execution order of timers libuv/libuv#3927 Fixes: nodejs#43931 Fixes: nodejs#42496 Fixes: nodejs#47715 Fixes: nodejs#47259 Fixes: nodejs#47241 PR-URL: nodejs#48078 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Jiawen Geng <[email protected]>

- linux: introduce io_uring support libuv/libuv#3952 - src: add new metrics APIs libuv/libuv#3749 - unix,win: give thread pool threads an 8 MB stack libuv/libuv#3787 - win,unix: change execution order of timers libuv/libuv#3927 Fixes: #43931 Fixes: #42496 Fixes: #47715 Fixes: #47259 Fixes: #47241 PR-URL: #48078 Backport-PR-URL: #49591 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Jiawen Geng <[email protected]>

vcunat · 2024-06-19T13:41:50Z

uint64_t* reserved[13];

So the structure has three uint64_t and then 13 pointers? That feels really weird on 32-bit platforms. Did you mean to use this?

uint64_t reserved[13];

Or is it too late and the 32-bit ABI set in stone now? (though the struct length will be a PITA to keep when "unreserving" in future)

EDIT: honestly I'm afraid that there may be no good solution to this anymore. Especially as you memcpy with sizeof(uv_metrics_t) in uv_metrics_info().

bnoordhuis · 2024-06-20T07:17:33Z

It's released so it's set in stone now, yes.

vcunat · 2024-06-20T07:22:21Z

Then I suggest at least a comment in the struct to warn about changing/unreserving, as that won't be easy to do while keeping both 32-bit and 64-bit struct lengths.

trevnorris · 2024-06-25T15:05:17Z

This fuck up will haunt me to the end of time. I can't believe I made it a pointer. Sorry.

vcunat · 2024-06-25T15:30:27Z

You were also discussing it during review but noone seems to have noticed 🤷🏽

though the struct length will be a PITA to keep when "unreserving" in future

Actually, using union is possible to simplify that at least. I even now saw this somewhere in libuv. (though it won't warn you about getting over)

trevnorris · 2024-06-25T15:49:54Z

This is ugly, but here's an idea on how to handle this to make it easier to use the reserved space in the future:

struct uv_metrics_s {
  uint64_t loop_count;
  uint64_t events;
  uint64_t events_waiting;
  /* private */
  uint64_t reserved[6];

#if UINTPTR_MAX == 0xffffffff
  uint64_t* dead[1];
#elif UINTPTR_MAX == 0xffffffffffffffff
  uint64_t* dead[7];
#else
#error "Platform not supported"
#endif
};

What ideas did you have in mind for using a union?

vcunat · 2024-06-25T15:56:18Z

Testing UINTPTR_MAX feels ugly, but off the top of my head I don't know a better preprocessor symbol.

I meant e.g. b6bf6d2

The following metrics are now always recorded and available via the new uv_metrics_info() API. * loop_count: Number of event loop iterations. * events: Total number of events processed by the event handler. * events_waiting: Total number of events waiting in the event queue when the event provider request was made. Benchmarking has shown no noticeable impact recording these metrics. PR-URL: libuv/libuv#3749

santigimeno approved these changes Sep 15, 2022

View reviewed changes

test/test-metrics.c Outdated Show resolved Hide resolved

test/test-metrics.c Outdated Show resolved Hide resolved

test/test-metrics.c Outdated Show resolved Hide resolved

docs/src/metrics.rst Outdated Show resolved Hide resolved

docs/src/metrics.rst Show resolved Hide resolved

trevnorris force-pushed the more-metrics branch from d69dbd5 to 461e4e4 Compare September 15, 2022 22:08

trevnorris mentioned this pull request Sep 15, 2022

loop: add elapsed time callback #3690

Closed

cjihrig reviewed Sep 19, 2022

View reviewed changes

docs/src/metrics.rst Outdated Show resolved Hide resolved

docs/src/metrics.rst Outdated Show resolved Hide resolved

src/uv-common.c Outdated Show resolved Hide resolved

src/uv-common.h Outdated Show resolved Hide resolved

src/uv-common.h Outdated Show resolved Hide resolved

trevnorris force-pushed the more-metrics branch from 461e4e4 to e603f44 Compare September 19, 2022 19:06

trevnorris marked this pull request as draft September 19, 2022 20:50

trevnorris force-pushed the more-metrics branch 3 times, most recently from 830d4ca to 75bc9f7 Compare September 19, 2022 21:34

trevnorris force-pushed the more-metrics branch from 75bc9f7 to 8edc60e Compare September 20, 2022 20:03

trevnorris force-pushed the more-metrics branch from 8edc60e to 70b0426 Compare September 20, 2022 20:14

trevnorris marked this pull request as ready for review September 20, 2022 21:00

santigimeno requested changes Sep 21, 2022

View reviewed changes

src/win/core.c Outdated Show resolved Hide resolved

test/test-metrics.c Outdated Show resolved Hide resolved

test/test-metrics.c Outdated Show resolved Hide resolved

santigimeno approved these changes Sep 21, 2022

View reviewed changes

santigimeno reviewed Sep 21, 2022

View reviewed changes

include/uv.h Outdated Show resolved Hide resolved

trevnorris force-pushed the more-metrics branch from 350866c to 268b010 Compare September 21, 2022 22:49

trevnorris force-pushed the more-metrics branch from 268b010 to efaead4 Compare September 26, 2022 14:00

trevnorris force-pushed the more-metrics branch from efaead4 to 27b0c85 Compare September 27, 2022 17:34

cjihrig approved these changes Sep 30, 2022

View reviewed changes

include/uv.h Show resolved Hide resolved

trevnorris force-pushed the more-metrics branch from a0e16d9 to f789e18 Compare October 27, 2022 19:30

trevnorris force-pushed the more-metrics branch from f789e18 to a60cf15 Compare November 10, 2022 21:55

santigimeno merged commit e141586 into libuv:v1.x Nov 11, 2022

trevnorris deleted the more-metrics branch February 14, 2023 21:51

santigimeno mentioned this pull request May 19, 2023

deps: upgrade to libuv 1.45.0 nodejs/node#48078

Closed

aantron mentioned this pull request May 23, 2023

Upgrade libuv to 1.45.0 aantron/luv#151

Closed

anthonyalayo mentioned this pull request Mar 6, 2024

Additional Libuv Metrics #4343

Closed

src: add new metrics APIs #3749

src: add new metrics APIs #3749

Uh oh!

Conversation

trevnorris commented Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trevnorris commented Sep 19, 2022

Uh oh!

trevnorris commented Sep 19, 2022

Uh oh!

trevnorris commented Sep 20, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trevnorris commented Sep 21, 2022

Uh oh!

Uh oh!

cjihrig left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

trevnorris commented Oct 3, 2022

Uh oh!

santigimeno commented Nov 11, 2022

Uh oh!

santigimeno commented Nov 11, 2022

Uh oh!

vcunat commented Jun 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bnoordhuis commented Jun 20, 2024

Uh oh!

vcunat commented Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trevnorris commented Jun 25, 2024

Uh oh!

vcunat commented Jun 25, 2024

Uh oh!

trevnorris commented Jun 25, 2024

Uh oh!

vcunat commented Jun 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

trevnorris commented Sep 14, 2022 •

edited

Loading

cjihrig left a comment •

edited

Loading

vcunat commented Jun 19, 2024 •

edited

Loading

vcunat commented Jun 20, 2024 •

edited

Loading