ServerBuilder defaults affected by Netty memory leak

[craigday]

GRPC 1.0.0 and 1.0.1 are both based on Netty 4.1.3.Final. This version of Netty is affected by a memory leak bug referenced and fixed in the following commit:

netty/netty@94d7557

The vanilla GRPC ServerBuilder uses a Executors.newCachedThreadPool() executor that expands and contracts based on the load. Any thread that is cleaned up leaks a whole lot of Netty cached data. The workaround is to specify your own Executor that reuses the threads and doesn't recycle them, something like:

ServerBuilder
    .forPort(port)
    .executor(Executors.newFixedThreadPool(8))
    .addService(myService)
    .build()
    .start();

[ejona86]

Just today we bumped our Netty version to 4.1.6. The other versions we had problems with. It's unclear whether we should backport that to v1.0.x.

In general, if you know your workload, specifying a fixed thread pool is a very good idea. Even without the Netty issues, the cached thread pool degrades very poorly under high load because it creates more and more threads. I am quite disappointed with Java's thread pool implementation because of that; there's not one that expands only when necessary and has a maximum size.

[buchgr]

@craigday are you able to quantify the amount of leaked data in your application somehow? Did it lead to OOM errors or measurable slowdown?

[craigday]

Our application(s) leaked about 300 meg of data over 2-3 days of low/moderate use, then OOM. This guy (https://groups.google.com/forum/#!topic/grpc-io/4wHa_fxaT-Q) blew 13gig in 7 days. I could build a test case to demonstrate the issue if you really need it, but the issue is fixed in Netty 4.1.6.Final.

[dmvk]

Hi, can this pls be backported to 1.0.x? It's major issue and it'd be real pain having to wait for 1.1.x. Thanks

D.

[ejona86]

#2349 is already marked for backport.

[ejona86]

This was backported as 7ef8ced.