Skip to content

Crash when destroy LoggerProvider and TracerProvider #1244

@owent

Description

@owent

Describe your environment

OS: Linux/CentOS 7
Compiler: GCC 4.8.5

Steps to reproduce

It may happen when destroy the last LoggerProvider and TracerProvider. But it only rarely happen.

Additional context

(gdb) info thread
  Id   Target Id         Frame 
  8    Thread 0x7f5b85538700 (LWP 17277) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
  7    Thread 0x7f5b84d37700 (LWP 17280) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
  6    Thread 0x7f5b85d39700 (LWP 17276) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
  5    Thread 0x7f5b89b9e8c0 (LWP 17267) 0x00007f5b8ab9f017 in pthread_join () from /lib64/libpthread.so.0
  4    Thread 0x7f5b612ce700 (LWP 31542) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
  3    Thread 0x7f5b60acd700 (LWP 17452) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
  2    Thread 0x7f5b83d35700 (LWP 17286) 0x00007f5b8aba1de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1    Thread 0x7f5b84536700 (LWP 17283) 0x00007f5b8a32ef8b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) ()
   from /lib64/libstdc++.so.6

Thread 1:

(gdb) bt
#0  0x00007f5b8a32ef8b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) () from /lib64/libstdc++.so.6
#1  0x00000000023cc018 in CreateInternal<std::basic_string<char>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (arena=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arena.h:696
#2  Create<std::basic_string<char>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (arena=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arena.h:316
#3  google::protobuf::internal::ArenaStringPtr::Set (this=0x7f5b83eb57b0, default_value=<optimized out>, 
    value=<error reading variable: Cannot access memory at address 0xffffffffffffffe8>, arena=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arenastring.cc:76
#4  0x00000000023cc0f2 in google::protobuf::internal::ArenaStringPtr::Set (this=<optimized out>, value=..., arena=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arenastring.cc:102
#5  0x00000000020c581b in set_name<std::basic_string<char> const&> (arg0=..., this=0x7f5b83eb57a0)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/build_jobs_linux/_deps/opentelemetry_cpp-v1.2.0/linux-x86_64-gnu/generated/third_party/opentelemetry-proto/opentelemetry/proto/common/v1/common.pb.h:2000
#6  opentelemetry::v1::exporter::otlp::OtlpRecordable::GetProtoInstrumentationLibrary (this=this@entry=0x8c90680)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/exporters/otlp/src/otlp_recordable.cc:70
#7  0x00000000020c76cd in opentelemetry::v1::exporter::otlp::OtlpRecordableUtils::PopulateRequest (spans=..., request=request@entry=0x7f5b83eb5890)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/exporters/otlp/src/otlp_recordable_utils.cc:270
#8  0x00000000020add5c in opentelemetry::v1::exporter::otlp::OtlpGrpcExporter::Export (this=0x8b16660, spans=...)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/exporters/otlp/src/otlp_grpc_exporter.cc:114
#9  0x00000000020f9eb9 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::Export (this=this@entry=0x8af7e20, was_force_flush_called=was_force_flush_called@entry=false)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:160
#10 0x00000000020fa03a in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DrainQueue (this=this@entry=0x8af7e20)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:177
#11 0x00000000020fa1b0 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DoBackgroundWork (this=0x8af7e20)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:98
#12 0x00007f5b8a325340 in ?? () from /lib64/libstdc++.so.6
#13 0x00007f5b8ab9dea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f5b89c9e9fd in clone () from /lib64/libc.so.6

Thread 5:

#0  0x00007f5b8ab9f017 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007f5b8a325107 in std::thread::join() () from /lib64/libstdc++.so.6
#2  0x00000000020f9653 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::Shutdown (this=0x8af7e20, timeout=...)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:186
#3  0x00000000020fb673 in Shutdown (timeout=..., this=0x8b4fdd0)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/multi_span_processor.h:131
#4  ~MultiSpanProcessor (this=0x8b4fdd0, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/multi_span_processor.h:139
#5  opentelemetry::v1::sdk::trace::MultiSpanProcessor::~MultiSpanProcessor (this=0x8b4fdd0, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/multi_span_processor.h:141
#6  0x00000000020f6e58 in operator() (this=<optimized out>, __ptr=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:67
#7  ~unique_ptr (this=0x89d7958, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#8  ~TracerContext (this=0x89d7908, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer_context.h:32
#9  destroy<opentelemetry::v1::sdk::trace::TracerContext> (this=<optimized out>, __p=0x89d7908) at /usr/include/c++/4.8.2/ext/new_allocator.h:124
#10 _S_destroy<opentelemetry::v1::sdk::trace::TracerContext> (__a=..., __p=0x89d7908) at /usr/include/c++/4.8.2/bits/alloc_traits.h:281
#11 destroy<opentelemetry::v1::sdk::trace::TracerContext> (__a=..., __p=0x89d7908) at /usr/include/c++/4.8.2/bits/alloc_traits.h:405
#12 std::_Sp_counted_ptr_inplace<opentelemetry::v1::sdk::trace::TracerContext, std::allocator<opentelemetry::v1::sdk::trace::TracerContext>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:407
#13 0x00000000020f69e2 in _M_release (this=0x89d78f0) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#14 ~__shared_count (this=0xc97c510, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
#15 ~__shared_ptr (this=0xc97c508, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781
#16 ~shared_ptr (this=0xc97c508, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr.h:93
#17 ~Tracer (this=0xc97c4e0, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer.h:26
#18 ~Tracer (this=0xc97c4e0, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer.h:26
#19 std::_Sp_counted_ptr<opentelemetry::v1::sdk::trace::Tracer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:290
#20 0x00000000020f6d2f in _M_release (this=0xb739080) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#21 ~__shared_count (this=0xbc76e48, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
#22 ~__shared_ptr (this=0xbc76e40, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781
#23 ~shared_ptr (this=0xbc76e40, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr.h:93
#24 _Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer> > (__pointer=0xbc76e40) at /usr/include/c++/4.8.2/bits/stl_construct.h:93
#25 __destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*> (__last=<optimized out>, __first=0xbc76e40) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
#26 _Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_construct.h:126
#27 _Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*, std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer> > (__last=0xbc76e50, __first=0xbc76e30)
    at /usr/include/c++/4.8.2/bits/stl_construct.h:151
#28 ~vector (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_vector.h:415
#29 ~TracerProvider (this=<optimized out>, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer_provider.h:25
#30 opentelemetry::v1::sdk::trace::TracerProvider::~TracerProvider (this=<optimized out>, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer_provi
#31 0x00000000013eed69 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x89cfe90) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#32 0x00000000014b0d0a in ~shared_ptr (this=0x7ffdbd793870, __in_chrg=<optimized out>)
    at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/install/linux-x86_64-gnu/include/opentelemetry/nostd/shared_ptr.h:98
#33 rpc::telemetry::_opentelemetry_cleanup_global_provider (app=...)

This may related to destructing order of members in LoggerProvider and Logger.
And also, I need Shutdown myself sometimes but the destructor of MultiSpanProcessor and MultiLogProcessor will shutdown again and them will cause crash when worker_thread_.join() is called twice in BatchLogProcessor and BatchSpanProcessor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions