Skip to content

Thread leak in API with unclosed sockets #5148

New issue
New issue
Closed
Closed
Thread leak in API with unclosed sockets#5148
Labels
area/apiREST APIbugSomething isn't workingcore/crashShouldn't happen, requires attention
@julianbrost

Description

@julianbrost
julianbrost
opened on Apr 6, 2017

When doing lots of requests to the API using the command check_http -H localhost -p 5665 -S -e '401 Unauthorized' -N, the icinga2 process leaks threads over time until it reaches the process limit for its user and then either segfaults or aborts (I have seen both so far).

Expected Behavior

Icinga should handle the requests an not leak threads.

Current Behavior

Icinga leaks threads over time. The leaked threads look like this in gdb (for Icinga 2.6.3):

Thread 68 (Thread 0x7f52e4fa8700 (LWP 18968)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f52ec29812b in boost::condition_variable::wait (this=this@entry=0x7f52dc065f68, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x00007f52ec227f5b in icinga::WorkQueue::Join (this=this@entry=0x7f52dc065e78, stop=stop@entry=true) at /build/icinga2-2.6.3/lib/base/workqueue.cpp:112
#3  0x00007f52ec25c2d2 in icinga::WorkQueue::~WorkQueue (this=0x7f52dc065e78, __in_chrg=<optimized out>) at /build/icinga2-2.6.3/lib/base/workqueue.cpp:48
#4  0x00007f52eb980806 in ~HttpServerConnection (this=0x7f52dc065dd0, __in_chrg=<optimized out>) at /build/icinga2-2.6.3/lib/remote/httpserverconnection.hpp:37
#5  icinga::HttpServerConnection::~HttpServerConnection (this=0x7f52dc065dd0, __in_chrg=<optimized out>) at /build/icinga2-2.6.3/lib/remote/httpserverconnection.hpp:37
#6  0x00007f52eb980a17 in ~intrusive_ptr (this=0x7f52dc060af0, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:97
#7  ~value (this=0x7f52dc060af0, __in_chrg=<optimized out>) at /usr/include/boost/bind/bind.hpp:112
#8  ~storage1 (this=0x7f52dc060af0, __in_chrg=<optimized out>) at /usr/include/boost/bind/storage.hpp:40
#9  boost::_bi::storage2<boost::_bi::value<boost::intrusive_ptr<icinga::HttpServerConnection> >, boost::_bi::value<icinga::HttpRequest> >::~storage2 (this=this@entry=0x7f52dc060af0, __in_chrg=<optimized out>) at /usr/include/boost/bind/storage.hpp:76
#10 0x00007f52eb980ac1 in ~list2 (this=0x7f52dc060af0, __in_chrg=<optimized out>) at /usr/include/boost/bind/bind.hpp:275
#11 ~bind_t (this=0x7f52dc060ae0, __in_chrg=<optimized out>) at /usr/include/boost/bind/bind.hpp:864
#12 manager (op=<optimized out>, out_buffer=..., in_buffer=...) at /usr/include/boost/function/function_base.hpp:389
#13 manager (op=<optimized out>, out_buffer=..., in_buffer=...) at /usr/include/boost/function/function_base.hpp:412
#14 boost::detail::function::functor_manager<boost::_bi::bind_t<void, boost::_mfi::mf1<void, icinga::HttpServerConnection, icinga::HttpRequest&>, boost::_bi::list2<boost::_bi::value<boost::intrusive_ptr<icinga::HttpServerConnection> >, boost::_bi::value<icinga::HttpRequest> > > >::manage (in_buffer=..., out_buffer=..., op=<optimized out>) at /usr/include/boost/function/function_base.hpp:440
#15 0x00007f52ec28baea in clear (this=<optimized out>, functor=...) at /usr/include/boost/function/function_template.hpp:504
#16 boost::function0<void>::clear (this=0x7f52dc065f94) at /usr/include/boost/function/function_template.hpp:877
#17 0x00007f52ec2389e6 in ~function0 (this=0x7f52e4fa7c70, __in_chrg=<optimized out>) at /usr/include/boost/function/function_template.hpp:759
#18 ~function (this=0x7f52e4fa7c70, __in_chrg=<optimized out>) at /usr/include/boost/function/function_template.hpp:1048
#19 operator= (f=<unknown type in /usr/lib/debug/.build-id/38/1781c10d7937a5e6d15b7b791c33e01f8e9cdb.debug, CU 0xaaed, DIE 0x2c41bd>, this=0x7f52e4fa7c90) at /usr/include/boost/function/function_template.hpp:1108
#20 operator= (this=0x7f52e4fa7c90) at /build/icinga2-2.6.3/lib/base/workqueue.hpp:43
#21 icinga::WorkQueue::WorkerThreadProc (this=0x7f52dc065e78) at /build/icinga2-2.6.3/lib/base/workqueue.cpp:248
#22 0x00007f52ecf1eaea in boost::(anonymous namespace)::thread_proxy (param=<optimized out>) at libs/thread/src/pthread/thread.cpp:164
#23 0x00007f52ec56c064 in start_thread (arg=0x7f52e4fa8700) at pthread_create.c:309
#24 0x00007f52e9b3362d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

In most cases (4 out of 5 so far), this lead to a segfault at some point: icinga2[16628]: segfault at ffffffffffffffff ip ffffffffffffffff sp 00002acd7791a2a8 error 15. In one case, Icinga aborted with the following log messages:

icinga2.log in case of SIGABRT (Icinga 2.6.2) 
[2017-04-06 02:40:50 +0200] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable

        (0) libbase.so.2.6.2: <unknown function> (+0xdb790) [0x7fb77da9d790]
        (1) libbase.so.2.6.2: boost::thread::start_thread() (+0x64) [0x7fb77da9da94]
        (2) libbase.so.2.6.2: icinga::WorkQueue::Enqueue(boost::function<void ()>&&, icinga::WorkQueuePriority, bool) (+0x625) [0x7fb77da3df05]
        (3) libremote.so.2.6.2: icinga::HttpServerConnection::ProcessMessage() (+0x314) [0x7fb77d1675b4]
        (4) libremote.so.2.6.2: icinga::HttpServerConnection::DataAvailableHandler() (+0x98) [0x7fb77d167d78]
        (5) libremote.so.2.6.2: icinga::HttpServerConnection::Start() (+0x19c) [0x7fb77d16808c]
        (6) libremote.so.2.6.2: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x4d9) [0x7fb77d178b09]
        (7) libremote.so.2.6.2: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x23) [0x7fb77d179f23]
        (8) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (9) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (10) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]


        (0) Handling new API client connection

Context:
        (0) Handling new API client connection

[2017-04-06 02:40:50 +0200] information/ApiListener: New client connection from [::1]:59250 (no client certificate)
[2017-04-06 02:40:50 +0200] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable

        (0) libbase.so.2.6.2: <unknown function> (+0xdb790) [0x7fb77da9d790]
        (1) libbase.so.2.6.2: boost::thread::start_thread() (+0x64) [0x7fb77da9da94]
        (2) libbase.so.2.6.2: icinga::WorkQueue::Enqueue(boost::function<void ()>&&, icinga::WorkQueuePriority, bool) (+0x625) [0x7fb77da3df05]
        (3) libremote.so.2.6.2: icinga::HttpServerConnection::ProcessMessage() (+0x314) [0x7fb77d1675b4]
        (4) libremote.so.2.6.2: icinga::HttpServerConnection::DataAvailableHandler() (+0x98) [0x7fb77d167d78]
        (5) libremote.so.2.6.2: icinga::HttpServerConnection::Start() (+0x19c) [0x7fb77d16808c]
        (6) libremote.so.2.6.2: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x4d9) [0x7fb77d178b09]
        (7) libremote.so.2.6.2: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x23) [0x7fb77d179f23]
        (8) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (9) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (10) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]


        (0) Handling new API client connection

Context:
        (0) Handling new API client connection

[2017-04-06 02:40:50 +0200] information/ApiListener: New client connection from [::1]:59252 (no client certificate)
[2017-04-06 02:40:50 +0200] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable

        (0) libbase.so.2.6.2: <unknown function> (+0xdb790) [0x7fb77da9d790]
        (1) libbase.so.2.6.2: boost::thread::start_thread() (+0x64) [0x7fb77da9da94]
        (2) libbase.so.2.6.2: icinga::WorkQueue::Enqueue(boost::function<void ()>&&, icinga::WorkQueuePriority, bool) (+0x625) [0x7fb77da3df05]
        (3) libremote.so.2.6.2: icinga::HttpServerConnection::ProcessMessage() (+0x314) [0x7fb77d1675b4]
        (4) libremote.so.2.6.2: icinga::HttpServerConnection::DataAvailableHandler() (+0x98) [0x7fb77d167d78]
        (5) libremote.so.2.6.2: icinga::HttpServerConnection::Start() (+0x19c) [0x7fb77d16808c]
        (6) libremote.so.2.6.2: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x4d9) [0x7fb77d178b09]
        (7) libremote.so.2.6.2: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x23) [0x7fb77d179f23]
        (8) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (9) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (10) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]


        (0) Handling new API client connection

Context:
        (0) Handling new API client connection

[2017-04-06 02:40:51 +0200] information/ApiListener: New client connection from [::1]:59256 (no client certificate)
[2017-04-06 02:40:51 +0200] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable

        (0) libbase.so.2.6.2: <unknown function> (+0xdb790) [0x7fb77da9d790]
        (1) libbase.so.2.6.2: boost::thread::start_thread() (+0x64) [0x7fb77da9da94]
        (2) libbase.so.2.6.2: icinga::WorkQueue::Enqueue(boost::function<void ()>&&, icinga::WorkQueuePriority, bool) (+0x625) [0x7fb77da3df05]
        (3) libremote.so.2.6.2: icinga::HttpServerConnection::ProcessMessage() (+0x314) [0x7fb77d1675b4]
        (4) libremote.so.2.6.2: icinga::HttpServerConnection::DataAvailableHandler() (+0x98) [0x7fb77d167d78]
        (5) libremote.so.2.6.2: icinga::HttpServerConnection::Start() (+0x19c) [0x7fb77d16808c]
        (6) libremote.so.2.6.2: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x4d9) [0x7fb77d178b09]
        (7) libremote.so.2.6.2: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x23) [0x7fb77d179f23]
        (8) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (9) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (10) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]


        (0) Handling new API client connection

Context:
        (0) Handling new API client connection

[2017-04-06 02:40:51 +0200] information/ApiListener: New client connection from [::1]:59258 (no client certificate)
[2017-04-06 02:40:51 +0200] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable

        (0) libbase.so.2.6.2: <unknown function> (+0xdb790) [0x7fb77da9d790]
        (1) libbase.so.2.6.2: boost::thread::start_thread() (+0x64) [0x7fb77da9da94]
        (2) libbase.so.2.6.2: icinga::WorkQueue::Enqueue(boost::function<void ()>&&, icinga::WorkQueuePriority, bool) (+0x625) [0x7fb77da3df05]
        (3) libremote.so.2.6.2: icinga::HttpServerConnection::ProcessMessage() (+0x314) [0x7fb77d1675b4]
        (4) libremote.so.2.6.2: icinga::HttpServerConnection::DataAvailableHandler() (+0x98) [0x7fb77d167d78]
        (5) libremote.so.2.6.2: icinga::HttpServerConnection::Start() (+0x19c) [0x7fb77d16808c]
        (6) libremote.so.2.6.2: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x4d9) [0x7fb77d178b09]
        (7) libremote.so.2.6.2: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr<icinga::Socket> const&, icinga::String const&, icinga::ConnectionRole) (+0x23) [0x7fb77d179f23]
        (8) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (9) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (10) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]


        (0) Handling new API client connection

Context:
        (0) Handling new API client connection

[2017-04-06 02:40:51 +0200] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1491439251.043945'

[2017-04-06 02:40:51 +0200] critical/checker: Exception occured while checking 'somehost': Error: parse error: premature EOF

                     (right here) ------^


        (0) libbase.so.2.6.2: <unknown function> (+0xde377) [0x7fb77daa0377]
        (1) libbase.so.2.6.2: <unknown function> (+0xde437) [0x7fb77daa0437]
        (2) libbase.so.2.6.2: icinga::JsonDecode(icinga::String const&) (+0x3c6) [0x7fb77da287b6]
        (3) libbase.so.2.6.2: icinga::Process::Run(boost::function<void (icinga::ProcessResult const&)> const&) (+0x43d) [0x7fb77da5eaed]
        (4) libicinga.so.2.6.2: icinga::PluginUtility::ExecuteCommand(boost::intrusive_ptr<icinga::Command> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, boost::function<void (icinga::Value const&, icinga::ProcessResult const&)> const&) (+0x7f3) [0x7fb77842cd03]
        (5) libmethods.so.2.6.2: icinga::PluginCheckTask::ScriptFunc(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) (+0x5e3) [0x7fb7780b4f83]
        (6) libmethods.so.2.6.2: <unknown function> (+0x1cd50) [0x7fb7780bcd50]
        (7) libmethods.so.2.6.2: <unknown function> (+0x1849d) [0x7fb7780b849d]
        (8) libbase.so.2.6.2: icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) (+0x4a) [0x7fb77da3039a]
        (9) libicinga.so.2.6.2: icinga::CheckCommand::Execute(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) (+0x153) [0x7fb7783dda23]
        (10) libicinga.so.2.6.2: icinga::Checkable::ExecuteCheck() (+0x58f) [0x7fb77845eedf]
        (11) libchecker.so.2.6.2: <unknown function> (+0x1638b) [0x7fb7639a638b]
        (12) libbase.so.2.6.2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x369) [0x7fb77da50869]
        (13) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (14) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (15) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]


        (0) Executing check for object 'somehost'

[2017-04-06 02:40:51 +0200] warning/Process: PID 9800 ('/usr/lib/nagios/plugins/check_ping' '-4' '-H' 'someaddr' '-c' '200,15%' '-w' '100,5%') died mysteriously: waitpid failed
[2017-04-06 02:40:51 +0200] warning/PluginCheckTask: Check command for object 'somehost!ping4' (PID: 9800, arguments: '/usr/lib/nagios/plugins/check_ping' '-4' '-H' 'someaddr' '-c' '200,15%' '-w' '100,5%') terminated with exit code 128, output: PING OK - Packet loss = 0%, RTA = 0.86 ms|rta=0.857000ms;100.000000;200.000000;0.000000 pl=0%;5;15;0

[2017-04-06 02:40:51 +0200] warning/Process: PID 9838 ('/usr/lib/nagios/plugins/check_ping' '-6' '-H' 'someaddr' '-c' '200,15%' '-w' '100,5%') died mysteriously: waitpid failed
[2017-04-06 02:40:51 +0200] warning/PluginCheckTask: Check command for object 'somehost!ping6' (PID: 9838, arguments: '/usr/lib/nagios/plugins/check_ping' '-6' '-H' 'someaddr' '-c' '200,15%' '-w' '100,5%') terminated with exit code 128, output: PING OK - Packet loss = 0%, RTA = 1.21 ms|rta=1.205000ms;100.000000;200.000000;0.000000 pl=0%;5;15;0

[2017-04-06 02:40:51 +0200] warning/Process: PID 9940 ('/usr/lib/nagios/plugins/check_ping' '-H' 'someaddr' '-c' '5000,100%' '-w' '3000,80%') died mysteriously: waitpid failed
[2017-04-06 02:40:51 +0200] warning/PluginCheckTask: Check command for object 'somehost' (PID: 9940, arguments: '/usr/lib/nagios/plugins/check_ping' '-H' 'someaddr' '-c' '5000,100%' '-w' '3000,80%') terminated with exit code 128, output: PING OK - Packet loss = 0%, RTA = 1.36 ms|rta=1.362000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0
report.1491439251.043945 
Application information:
  Application version: r2.6.2-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 4.9.13
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: smithers
Stacktrace:

        (0) libpthread.so.0: <unknown function> (+0xf890) [0x7fb77dd79890]
        (1) libc.so.6: gsignal (+0x37) [0x7fb77b287067]
        (2) libc.so.6: abort (+0x148) [0x7fb77b288448]
        (3) libc.so.6: <unknown function> (+0x2e266) [0x7fb77b280266]
        (4) libc.so.6: <unknown function> (+0x2e312) [0x7fb77b280312]
        (5) libbase.so.2.6.2: <unknown function> (+0x4be30) [0x7fb77da0de30]
        (6) libbase.so.2.6.2: icinga::ObjectLock::LockMutex(icinga::Object const*) (+0x9c) [0x7fb77da9ddac]
        (7) libbase.so.2.6.2: icinga::Dictionary::Get(icinga::String const&) const (+0x38) [0x7fb77da1f078]
        (8) libremote.so.2.6.2: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x433) [0x7fb77d1669a3]
        (9) libremote.so.2.6.2: icinga::HttpServerConnection::ProcessMessage() (+0x3c) [0x7fb77d1672dc]
        (10) libremote.so.2.6.2: icinga::HttpServerConnection::DataAvailableHandler() (+0x98) [0x7fb77d167d78]
        (11) libbase.so.2.6.2: <unknown function> (+0x11a3bc) [0x7fb77dadc3bc]
        (12) libbase.so.2.6.2: icinga::Stream::SignalDataAvailable() (+0x40) [0x7fb77da7f370]
        (13) libbase.so.2.6.2: icinga::TlsStream::OnEvent(int) (+0x298) [0x7fb77da7faf8]
        (14) libbase.so.2.6.2: icinga::SocketEventEngineEpoll::ThreadProc(int) (+0x42b) [0x7fb77da4f33b]
        (15) libboost_thread.so.1.55.0: <unknown function> (+0xdaea) [0x7fb77e724aea]
        (16) libpthread.so.0: <unknown function> (+0x8064) [0x7fb77dd72064]
        (17) libc.so.6: clone (+0x6d) [0x7fb77b33a62d]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

I suspect both behaviors are only a symptom of using up the ulimit for allowed processes.

Steps to Reproduce (for bugs)

  1. apt install icinga2
  2. icinga2 api setup
  3. systemctl restart icinga2
  4. Check the number of icinga2 threads: for pid in $(pidof icinga2); do ps -T -p $pid | grep -F icinga2; done | wc -l (16 in my case)
  5. Send lots of requests to the API: while :; do /usr/lib/nagios/plugins/check_http -H localhost -p 5665 -S -e '401 Unauthorized' -N > /dev/null; done (let this run for some time, see below)
  6. Again, check the number of icinga2 threads: for pid in $(pidof icinga2); do ps -T -p $pid | grep -F icinga2; done | wc -l (now for example 70 in my case)

Some remarks: I'm not entirely sure what's the best way to reproduce the issue. For the numbers above, I used a fresh Icinga2 instance on a new VM with the Debian standard config (i.e. monitoring localhost with a few checks) and running the loop sending the requests 6 times in parallel for about an hour. There it does not leak threads very fast but it does so over time.

On the other hand, with an instance that handles quite some checks, including many passive ones, and regularly receiving real API requests, running one loop doing requests is enough to get a thread leaked every few seconds.

Context

I tried to to debug another issue and forgot that loop running which made Icinga segfault. That's not good ;)

Your Environment

  • Version used (icinga2 --version): r2.6.2-1, steps to reproduce were also tested on r2.6.3-1
  • Operating System and version: Debian jessie 8.7
  • Enabled features (icinga2 feature list): api checker mainlog notification for the 2.6.3 instance I used for reproducing and api checker command ido-pgsql livestatus mainlog notification statusdata syslog on the 2.6.2 with which I noticed the issue
  • Config validation (icinga2 daemon -C): no errors

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiREST APIbugSomething isn't workingcore/crashShouldn't happen, requires attention

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions