Describe the bug
This is a follow-up to #135 as it was found rolling out junixsocket to all services in prod caused gc spikes leading to p99/max latency. Cannot share heap dumps but the symptoms were as follows:
- Services would see spikes in p99 latency growing over time. In a span of 24 hours post deploy, spikes in latency would be seen growing 30s, 1min, 3min, 5min, etc. Max we saw was 6 min.
- These spikes would correlate with gc pause times (with g1gc). Memory seems to getting reclaimed but gc ends up running longer each time.
- These services are running jetty 10.
We had to rollback usage of junixsocket across our services.
Describe the bug
This is a follow-up to #135 as it was found rolling out junixsocket to all services in prod caused gc spikes leading to p99/max latency. Cannot share heap dumps but the symptoms were as follows:
We had to rollback usage of junixsocket across our services.