Skip to content

Windows gRPC C core memory leak #13088

@JakaBac

Description

@JakaBac

Should this be an issue in the gRPC issue tracker?

Yes, I believe so

What version of gRPC and what language are you using?

C++ master branch and also v1.6.x (current https://grpc.io/release)

What operating system (Linux, Windows, …) and version?

Windows 10 version 1709 OS Build 16299.19

What runtime / compiler are you using (e.g. python version or version of gcc)

MSVC 2017 (15.4.1)

What did you do?

Run standard helloworld sample server and client on same machine.
Modified helloworld client so it opens 100 connections and does 100 requests (one per connection), run more instances of modified helloworld client (in order to open and close "large" amount of tcp connections)

What did you expect to see?

helloworld server & client doing their thing.
helloworld server's process ram should not increase constantly

What did you see instead?

helloworld server process ram was growing slowly with each open tcp connection. When all connections were closed ram was not reclaimed.
Memory was also not freed after calling server->Shutdown();

Anything else we should know about your project / environment?

When trying find out why this is happening I found out that the memory which is not freed is getting allocated in grpc_resource_quota_create called by grpc_tcp_create

By tracing the execution of the function and also tracing the related tcp_free i noticed that in grpc_tcp_create resource_quota->refs.count is set to 1 when resource_quota is created. This refcount is then increased when tcp->resource_user = grpc_resource_user_create(resource_quota, peer_string); is called near the end of the function. Since this looked a bit suspicious I checked how grpc_tcp_create is implemented in tcp_posix.cc and I saw that grpc_resource_quota_unref_internal(exec_ctx, resource_quota); is called before returning. This call is not present in tcp_windows.cc

Adding that line fixed the leak problem. I tested it with master and v1.6.x branch
I did not do run any gRPC tests

Then looking at the function as a whole I could see that there is some refcount increasing and decreasing going on if resource_quota is received from grpc_channel_args *channel_args, but tcp_posix.cc does the same thing.

So the patch is (no Pull request since I just started playing with gRPC today):

diff --git a/src/core/lib/iomgr/tcp_windows.cc b/src/core/lib/iomgr/tcp_windows.cc
index dc84e564a9..ad939d3b50 100644
--- a/src/core/lib/iomgr/tcp_windows.cc
+++ b/src/core/lib/iomgr/tcp_windows.cc
@@ -442,6 +442,7 @@ grpc_endpoint *grpc_tcp_create(grpc_exec_ctx *exec_ctx, grpc_winsocket *socket,
   tcp->resource_user = grpc_resource_user_create(resource_quota, peer_string);
   /* Tell network status tracking code about the new endpoint */
   grpc_network_status_register_endpoint(&tcp->base);
+  grpc_resource_quota_unref_internal(exec_ctx, resource_quota);

   return &tcp->base;
 }

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions