{"id":4239,"date":"2019-09-11T17:15:10","date_gmt":"2019-09-11T14:15:10","guid":{"rendered":"https:\/\/www.systemcodegeeks.com\/?p=4239"},"modified":"2019-09-11T11:32:14","modified_gmt":"2019-09-11T08:32:14","slug":"tcp-out-of-memory-consider-tuning-tcp_mem","status":"publish","type":"post","link":"https:\/\/www.systemcodegeeks.com\/linux\/tcp-out-of-memory-consider-tuning-tcp_mem\/","title":{"rendered":"TCP: out of memory \u2014 consider tuning tcp_mem"},"content":{"rendered":"\n

Recently we experienced an interesting production problem. This application was running on multiple AWS EC2 instances behind Elastic Load Balancer. The application was running on GNU\/Linux OS, Java 8, Tomcat 8 application server. All of sudden one of the application instances became unresponsive. All other application instances were handling the traffic properly. Whenever the HTTP request was sent to this application instance from the browser, we were getting following response to be printed on the browser.<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n<\/td>\n
\n
\n
Proxy Error <\/code><\/div>\n
 <\/div>\n
The proxy server received an invalid response from an upstream server. <\/code><\/div>\n
The proxy server could not handle the request GET \/.<\/code><\/div>\n
 <\/div>\n
Reason: Error reading from remote server<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

We used our APM (Application Performance Monitoring) tool to examine the problem. From the APM tool, we could observe CPU, memory utilization to be perfect. On the other hand, from the APM tool, we could observe that traffic wasn\u2019t coming into this particular application instance. It was really puzzling. Why traffic wasn\u2019t coming in?<\/p>\n\n\n\n

We logged in to this problematic AWS EC2 instance. We executed vmstat, iostat, netstat, top, df<\/strong> commands to see whether we can uncover any anomaly. To our surprise, all these great tools didn\u2019t report any issue.<\/p>\n\n\n\n

As the next step, we restarted the Tomcat application server in which this application was running. It didn\u2019t make any difference either. Still, this application instance wasn\u2019t responding at all.<\/p>\n\n\n\n

DMESG command<\/h2>\n\n\n\n

Then we issued \u2018dmesg\u2019<\/strong> command on this EC2 instance.  This command prints the message buffer of the kernel. The output of this command typically contains the messages produced by the device drivers. In the output generated by this command, we noticed the following interesting messages to be printed repeatedly:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n
10<\/div>\n
11<\/div>\n
12<\/div>\n
13<\/div>\n
14<\/div>\n
15<\/div>\n
16<\/div>\n
17<\/div>\n
18<\/div>\n
19<\/div>\n
20<\/div>\n
21<\/div>\n
22<\/div>\n<\/td>\n
\n
\n
[4486500.513856] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487211.020449] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487369.441522] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487535.908607] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487639.802123] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487717.564383] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487784.382403] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487816.378638] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487855.352405] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487862.816227] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4487928.859785] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4488215.969409] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4488642.426484] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4489347.800558] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4490054.414047] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4490763.997344] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4491474.743039] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4491859.749745] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4492182.082423] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4496318.377316] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4505666.858267] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n
[4521592.915616] TCP: out of memory -- consider tuning tcp_mem<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

We were intrigued to see this error message: \u201cTCP: out of memory \u2014 consider tuning tcp_mem\u201d. It means out of memory error is happening at the TCP level. We had always taught out of memory error happens only at the application level and never at the TCP level.<\/p>\n\n\n\n

Problem was intriguing because we breathe this OutOfMemoryError problem day in and out. We have built troubleshooting tools like GCeasy<\/a>, HeapHero<\/a> to facilitate engineers to debug OutOfMemoryError that happens at the application level (Java, Android, Scala, Jython\u2026 applications). We have written several blogs on this OutOfMemoryError<\/a> topic. But we were stumped to see OutOfMemory happening at the device driver level. We never thought there would be a problem at the device driver level, that too in, stable Linux operating system. Being stumped by this problem, we weren\u2019t sure how to proceed further.<\/p>\n\n\n\n

Thus, we resorted to Google god\u2019s help \ud83d\ude0a. Googling for the search term: \u201cTCP: out of memory \u2014 consider tuning tcp_mem\u201d, showed only 12 search results. But for one article, none of them had much content  \u2639. Even that one article was written in a foreign language that we couldn\u2019t understand. So, we aren\u2019t sure how to troubleshoot this problem.<\/p>\n\n\n\n

Now left with no other solutions, we went ahead and implemented universal solution i.e. \u201crestart\u201d. We restarted the EC2 instance to put-off immediate burning fire. Hurray!! Restarting the server cleared the problem immediately. Apparently, this server wasn\u2019t restarted for several days (like more than 70+ days), maybe due to that application might have saturated TCP memory limits.<\/p>\n\n\n\n

We reached out to one of our intelligent friends who works for a world-class technology company for help. This friend asked us the values that we are setting for the below kernel properties:<\/p>\n\n\n\n