{"id":1019,"date":"2016-04-08T17:11:29","date_gmt":"2016-04-08T14:11:29","guid":{"rendered":"http:\/\/www.systemcodegeeks.com\/?p=1019"},"modified":"2016-04-03T15:11:46","modified_gmt":"2016-04-03T12:11:46","slug":"crouching-supervisor-hidden-file-descriptor-setting","status":"publish","type":"post","link":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/","title":{"rendered":"Crouching Supervisor, Hidden File Descriptor Setting"},"content":{"rendered":"

Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances that are used to load balance to nodes in our RabbitMQ cluster.<\/p>\n

We\u2019ve done this a lot of times before and set all the usual user settings required under limits.d<\/code> to ensure proper file descriptors are allocated for the haproxy process. While creating this new role we also decided to use supervisor to supervise the haproxy process as it was previously observed in an older release that it didn\u2019t automatically restart when it crashed (which in itself is a rarity).<\/p>\n

Everything looked solid and we began throwing some traffic at the new balancer. Eventually we discovered something had gone horribly wrong! Tons of connection refused errors began showing up and the behavior exhibited was what one would expect if file descriptors weren\u2019t being allocated correctly. Sure enough a quick look at \/proc\/<pid>\/limits<\/code> revealed that maximum open file descriptors were set to the very low value of 1024. We directed traffic back to the old balancer and began the investigation. How could this be? All of the settings were correct so why is it being set to 1024?<\/p>\n

Supervisor was one new variable in the mix so I decided to begin pursuing the supervisor documentation<\/a> and scanning for the number 1024 to see what might be tied to that. Sure enough, I came to discover the minfds<\/code> setting. Let\u2019s take a look at what the supervisor documentation has to say about this setting.<\/p>\n

The minimum number of file descriptors that must be available before supervisord will start successfully. A call to setrlimit<\/code> will be made to attempt to raise the soft and hard limits of the supervisord process to satisfy minfds<\/code>.<\/p>\n

The hard limit may only be raised if supervisord is run as root. supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be obtained from the OS, so it\u2019s useful to be able to specify a minimum value to ensure it doesn\u2019t run out of them during execution. This option is particularly useful on Solaris, which has a low per-process fd limit by default.<\/p>\n

Default<\/em>: 1024<\/p><\/blockquote>\n

Well that doesn\u2019t make much sense\u2026 if I\u2019m reading this correctly it\u2019s simply saying that the number specified is the minimum that should be available, right? The devil as they say is in the details. If we look at the documentation on setrlimit<\/a> we\u2019ll clearly see that this will actually set the limits without any reservations on what it currently is.<\/p>\n

The call basically is going to set max open files to whatever the value minfds is defined to in supervisor. Sure enough, as an experiment I set minfds<\/code> in supervisor\u2019s configuration to a higher number and after restarting supervisor the number of open file descriptors allocated to the haproxy process were greatly increased and reflected what minfds<\/code> was set to.<\/p>\n

In the end this pain also turned out to be unnecessary. While we had used supervisor because it was \u201cwhat we know well\u201d it turned out that the newer distribution we were releasing on already managed services via systemd which by default was also configured to respawn on failure.<\/p>\n

\"iWKad22\"<\/a><\/p>\n

Hopefully this story will prevent a similar trail of sorrow for others who may encounter the same situation!<\/p>\n

TLDR; If you\u2019re having supervisor supervise an application that is sensitive to max open file descriptors you\u2019ll want to ensure minfds<\/code> is set to match!<\/p>\n

\n\n\n\n
Reference: <\/span><\/td>\nCrouching Supervisor, Hidden File Descriptor Setting<\/a> from our SCG partner<\/a>\u00a0James Carr<\/span> at the Rants and Musings of an Agile Developer<\/a> blog.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances that are used to load balance to nodes in our RabbitMQ cluster. We\u2019ve done this a lot of times before and set all the usual user settings required under limits.d to ensure …<\/p>\n","protected":false},"author":22,"featured_media":192,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[67],"class_list":["post-1019","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux","tag-supervisor"],"yoast_head":"\nCrouching Supervisor, Hidden File Descriptor Setting - System Code Geeks - 2026<\/title>\n<meta name=\"description\" content=\"Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crouching Supervisor, Hidden File Descriptor Setting - System Code Geeks - 2026\" \/>\n<meta property=\"og:description\" content=\"Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\" \/>\n<meta property=\"og:site_name\" content=\"System Code Geeks\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/systemcodegeeks\" \/>\n<meta property=\"article:published_time\" content=\"2016-04-08T14:11:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"150\" \/>\n\t<meta property=\"og:image:height\" content=\"150\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"James Carr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@systemcodegeeks\" \/>\n<meta name=\"twitter:site\" content=\"@systemcodegeeks\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"James Carr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\"},\"author\":{\"name\":\"James Carr\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/#\/schema\/person\/9504aaff478a805b152e506abe45ece4\"},\"headline\":\"Crouching Supervisor, Hidden File Descriptor Setting\",\"datePublished\":\"2016-04-08T14:11:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\"},\"wordCount\":608,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg\",\"keywords\":[\"Supervisor\"],\"articleSection\":[\"Linux\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\",\"url\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\",\"name\":\"Crouching Supervisor, Hidden File Descriptor Setting - System Code Geeks - 2026\",\"isPartOf\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg\",\"datePublished\":\"2016-04-08T14:11:29+00:00\",\"description\":\"Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances\",\"breadcrumb\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage\",\"url\":\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg\",\"contentUrl\":\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg\",\"width\":150,\"height\":150},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.systemcodegeeks.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Linux\",\"item\":\"https:\/\/www.systemcodegeeks.com\/category\/linux\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Crouching Supervisor, Hidden File Descriptor Setting\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/#website\",\"url\":\"https:\/\/www.systemcodegeeks.com\/\",\"name\":\"System Code Geeks\",\"description\":\"Operating System Developers Resource Center\",\"publisher\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.systemcodegeeks.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/#organization\",\"name\":\"Exelixis Media P.C.\",\"url\":\"https:\/\/www.systemcodegeeks.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png\",\"contentUrl\":\"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png\",\"width\":864,\"height\":246,\"caption\":\"Exelixis Media P.C.\"},\"image\":{\"@id\":\"https:\/\/www.systemcodegeeks.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/systemcodegeeks\",\"https:\/\/x.com\/systemcodegeeks\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/#\/schema\/person\/9504aaff478a805b152e506abe45ece4\",\"name\":\"James Carr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.systemcodegeeks.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/751c55f22a64fa5ac630c3bddaf1a4f4e11c442aaa9fc308004564cdca64b900?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/751c55f22a64fa5ac630c3bddaf1a4f4e11c442aaa9fc308004564cdca64b900?s=96&d=mm&r=g\",\"caption\":\"James Carr\"},\"sameAs\":[\"http:\/\/blog.james-carr.org\/\"],\"url\":\"https:\/\/www.systemcodegeeks.com\/author\/james-carr\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crouching Supervisor, Hidden File Descriptor Setting - System Code Geeks - 2026","description":"Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/","og_locale":"en_US","og_type":"article","og_title":"Crouching Supervisor, Hidden File Descriptor Setting - System Code Geeks - 2026","og_description":"Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances","og_url":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/","og_site_name":"System Code Geeks","article_publisher":"https:\/\/www.facebook.com\/systemcodegeeks","article_published_time":"2016-04-08T14:11:29+00:00","og_image":[{"width":150,"height":150,"url":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg","type":"image\/jpeg"}],"author":"James Carr","twitter_card":"summary_large_image","twitter_creator":"@systemcodegeeks","twitter_site":"@systemcodegeeks","twitter_misc":{"Written by":"James Carr","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#article","isPartOf":{"@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/"},"author":{"name":"James Carr","@id":"https:\/\/www.systemcodegeeks.com\/#\/schema\/person\/9504aaff478a805b152e506abe45ece4"},"headline":"Crouching Supervisor, Hidden File Descriptor Setting","datePublished":"2016-04-08T14:11:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/"},"wordCount":608,"commentCount":0,"publisher":{"@id":"https:\/\/www.systemcodegeeks.com\/#organization"},"image":{"@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage"},"thumbnailUrl":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg","keywords":["Supervisor"],"articleSection":["Linux"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/","url":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/","name":"Crouching Supervisor, Hidden File Descriptor Setting - System Code Geeks - 2026","isPartOf":{"@id":"https:\/\/www.systemcodegeeks.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage"},"image":{"@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage"},"thumbnailUrl":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg","datePublished":"2016-04-08T14:11:29+00:00","description":"Here\u2019s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances","breadcrumb":{"@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#primaryimage","url":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg","contentUrl":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2016\/01\/linux-logo.jpg","width":150,"height":150},{"@type":"BreadcrumbList","@id":"https:\/\/www.systemcodegeeks.com\/linux\/crouching-supervisor-hidden-file-descriptor-setting\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.systemcodegeeks.com\/"},{"@type":"ListItem","position":2,"name":"Linux","item":"https:\/\/www.systemcodegeeks.com\/category\/linux\/"},{"@type":"ListItem","position":3,"name":"Crouching Supervisor, Hidden File Descriptor Setting"}]},{"@type":"WebSite","@id":"https:\/\/www.systemcodegeeks.com\/#website","url":"https:\/\/www.systemcodegeeks.com\/","name":"System Code Geeks","description":"Operating System Developers Resource Center","publisher":{"@id":"https:\/\/www.systemcodegeeks.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.systemcodegeeks.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.systemcodegeeks.com\/#organization","name":"Exelixis Media P.C.","url":"https:\/\/www.systemcodegeeks.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.systemcodegeeks.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","contentUrl":"https:\/\/www.systemcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","width":864,"height":246,"caption":"Exelixis Media P.C."},"image":{"@id":"https:\/\/www.systemcodegeeks.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/systemcodegeeks","https:\/\/x.com\/systemcodegeeks"]},{"@type":"Person","@id":"https:\/\/www.systemcodegeeks.com\/#\/schema\/person\/9504aaff478a805b152e506abe45ece4","name":"James Carr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.systemcodegeeks.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/751c55f22a64fa5ac630c3bddaf1a4f4e11c442aaa9fc308004564cdca64b900?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/751c55f22a64fa5ac630c3bddaf1a4f4e11c442aaa9fc308004564cdca64b900?s=96&d=mm&r=g","caption":"James Carr"},"sameAs":["http:\/\/blog.james-carr.org\/"],"url":"https:\/\/www.systemcodegeeks.com\/author\/james-carr\/"}]}},"_links":{"self":[{"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/posts\/1019","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/comments?post=1019"}],"version-history":[{"count":0,"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/posts\/1019\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/media\/192"}],"wp:attachment":[{"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/media?parent=1019"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/categories?post=1019"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.systemcodegeeks.com\/wp-json\/wp\/v2\/tags?post=1019"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}