-
Notifications
You must be signed in to change notification settings - Fork 512
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe your environment
OS: Linux/CentOS 7
Compiler: GCC 4.8.5
Steps to reproduce
It may happen when destroy the last LoggerProvider and TracerProvider. But it only rarely happen.
Additional context
(gdb) info thread
Id Target Id Frame
8 Thread 0x7f5b85538700 (LWP 17277) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
7 Thread 0x7f5b84d37700 (LWP 17280) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
6 Thread 0x7f5b85d39700 (LWP 17276) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
5 Thread 0x7f5b89b9e8c0 (LWP 17267) 0x00007f5b8ab9f017 in pthread_join () from /lib64/libpthread.so.0
4 Thread 0x7f5b612ce700 (LWP 31542) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
3 Thread 0x7f5b60acd700 (LWP 17452) 0x00007f5b89c98d19 in syscall () from /lib64/libc.so.6
2 Thread 0x7f5b83d35700 (LWP 17286) 0x00007f5b8aba1de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7f5b84536700 (LWP 17283) 0x00007f5b8a32ef8b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) ()
from /lib64/libstdc++.so.6Thread 1:
(gdb) bt
#0 0x00007f5b8a32ef8b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) () from /lib64/libstdc++.so.6
#1 0x00000000023cc018 in CreateInternal<std::basic_string<char>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (arena=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arena.h:696
#2 Create<std::basic_string<char>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (arena=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arena.h:316
#3 google::protobuf::internal::ArenaStringPtr::Set (this=0x7f5b83eb57b0, default_value=<optimized out>,
value=<error reading variable: Cannot access memory at address 0xffffffffffffffe8>, arena=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arenastring.cc:76
#4 0x00000000023cc0f2 in google::protobuf::internal::ArenaStringPtr::Set (this=<optimized out>, value=..., arena=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/protobuf-v3.19.4/src/google/protobuf/arenastring.cc:102
#5 0x00000000020c581b in set_name<std::basic_string<char> const&> (arg0=..., this=0x7f5b83eb57a0)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/build_jobs_linux/_deps/opentelemetry_cpp-v1.2.0/linux-x86_64-gnu/generated/third_party/opentelemetry-proto/opentelemetry/proto/common/v1/common.pb.h:2000
#6 opentelemetry::v1::exporter::otlp::OtlpRecordable::GetProtoInstrumentationLibrary (this=this@entry=0x8c90680)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/exporters/otlp/src/otlp_recordable.cc:70
#7 0x00000000020c76cd in opentelemetry::v1::exporter::otlp::OtlpRecordableUtils::PopulateRequest (spans=..., request=request@entry=0x7f5b83eb5890)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/exporters/otlp/src/otlp_recordable_utils.cc:270
#8 0x00000000020add5c in opentelemetry::v1::exporter::otlp::OtlpGrpcExporter::Export (this=0x8b16660, spans=...)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/exporters/otlp/src/otlp_grpc_exporter.cc:114
#9 0x00000000020f9eb9 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::Export (this=this@entry=0x8af7e20, was_force_flush_called=was_force_flush_called@entry=false)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:160
#10 0x00000000020fa03a in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DrainQueue (this=this@entry=0x8af7e20)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:177
#11 0x00000000020fa1b0 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DoBackgroundWork (this=0x8af7e20)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:98
#12 0x00007f5b8a325340 in ?? () from /lib64/libstdc++.so.6
#13 0x00007f5b8ab9dea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f5b89c9e9fd in clone () from /lib64/libc.so.6Thread 5:
#0 0x00007f5b8ab9f017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f5b8a325107 in std::thread::join() () from /lib64/libstdc++.so.6
#2 0x00000000020f9653 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::Shutdown (this=0x8af7e20, timeout=...)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/src/trace/batch_span_processor.cc:186
#3 0x00000000020fb673 in Shutdown (timeout=..., this=0x8b4fdd0)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/multi_span_processor.h:131
#4 ~MultiSpanProcessor (this=0x8b4fdd0, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/multi_span_processor.h:139
#5 opentelemetry::v1::sdk::trace::MultiSpanProcessor::~MultiSpanProcessor (this=0x8b4fdd0, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/multi_span_processor.h:141
#6 0x00000000020f6e58 in operator() (this=<optimized out>, __ptr=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:67
#7 ~unique_ptr (this=0x89d7958, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#8 ~TracerContext (this=0x89d7908, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer_context.h:32
#9 destroy<opentelemetry::v1::sdk::trace::TracerContext> (this=<optimized out>, __p=0x89d7908) at /usr/include/c++/4.8.2/ext/new_allocator.h:124
#10 _S_destroy<opentelemetry::v1::sdk::trace::TracerContext> (__a=..., __p=0x89d7908) at /usr/include/c++/4.8.2/bits/alloc_traits.h:281
#11 destroy<opentelemetry::v1::sdk::trace::TracerContext> (__a=..., __p=0x89d7908) at /usr/include/c++/4.8.2/bits/alloc_traits.h:405
#12 std::_Sp_counted_ptr_inplace<opentelemetry::v1::sdk::trace::TracerContext, std::allocator<opentelemetry::v1::sdk::trace::TracerContext>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:407
#13 0x00000000020f69e2 in _M_release (this=0x89d78f0) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#14 ~__shared_count (this=0xc97c510, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
#15 ~__shared_ptr (this=0xc97c508, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781
#16 ~shared_ptr (this=0xc97c508, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr.h:93
#17 ~Tracer (this=0xc97c4e0, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer.h:26
#18 ~Tracer (this=0xc97c4e0, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer.h:26
#19 std::_Sp_counted_ptr<opentelemetry::v1::sdk::trace::Tracer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>)
at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:290
#20 0x00000000020f6d2f in _M_release (this=0xb739080) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#21 ~__shared_count (this=0xbc76e48, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
#22 ~__shared_ptr (this=0xbc76e40, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781
#23 ~shared_ptr (this=0xbc76e40, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr.h:93
#24 _Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer> > (__pointer=0xbc76e40) at /usr/include/c++/4.8.2/bits/stl_construct.h:93
#25 __destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*> (__last=<optimized out>, __first=0xbc76e40) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
#26 _Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_construct.h:126
#27 _Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*, std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer> > (__last=0xbc76e50, __first=0xbc76e30)
at /usr/include/c++/4.8.2/bits/stl_construct.h:151
#28 ~vector (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_vector.h:415
#29 ~TracerProvider (this=<optimized out>, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer_provider.h:25
#30 opentelemetry::v1::sdk::trace::TracerProvider::~TracerProvider (this=<optimized out>, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/packages/opentelemetry-cpp-v1.2.0/sdk/include/opentelemetry/sdk/trace/tracer_provi
#31 0x00000000013eed69 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x89cfe90) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#32 0x00000000014b0d0a in ~shared_ptr (this=0x7ffdbd793870, __in_chrg=<optimized out>)
at /data2/devops/workspace/p-50619563e9c746fca28a7a6ab5f648d6/src/repo/Server/dev/third_party/install/linux-x86_64-gnu/include/opentelemetry/nostd/shared_ptr.h:98
#33 rpc::telemetry::_opentelemetry_cleanup_global_provider (app=...)This may related to destructing order of members in LoggerProvider and Logger.
And also, I need Shutdown myself sometimes but the destructor of MultiSpanProcessor and MultiLogProcessor will shutdown again and them will cause crash when worker_thread_.join() is called twice in BatchLogProcessor and BatchSpanProcessor.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working