GlobalLogHandler::GetLogHandler() singleton is not unique. #1409
Description
Greetings.
The title is correct.
In my application, the code is calling:
(1) GlobalLogHandler::SetLogHandler(), to set a custom handler
(2) OTEL_INTERNAL_LOG_XXX(), to print internal logs.
These calls produce logs in the custom handler, as expected so far.
But then, later:
(3) OTEL_INTERNAL_LOG_XXX() calls from opentelemetry libraries
end up calling the default handler DefaultLogHandler instead of the custom
one, which defeats the purpose of a singleton SetLogHandler().
Analysis:
GlobalLogHandler::GetHandlerAndLevel() is implemented inline in a header
file, not a .cc file.
When building static libraries, this "singleton" ends up in multiple
libraries:
[email protected]:lib64> pwd
/usr/local/lib64
[email protected]:lib64> for L in `ls libopentelemetry_*.a`; do echo $L; nm $L | c++filt | grep GlobalLogHandler::GetHandlerAndLevel; done
libopentelemetry_common.a
libopentelemetry_exporter_ostream_metrics.a
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_ostream_span.a
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_otlp_grpc.a
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_otlp_grpc_log.a
libopentelemetry_exporter_otlp_http.a
libopentelemetry_exporter_otlp_http_client.a
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_http_client_curl.a
libopentelemetry_metrics.a
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_otlp_recordable.a
libopentelemetry_proto.a
libopentelemetry_resources.a
libopentelemetry_trace.a
0000000000000000 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000000000 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000000000 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_version.a
Note, interestingly, how the GlobalLogHandler::GetHandlerAndLevel() code is
not part of libopentelemetry_common.a, but defined (not referenced) in every use instead.
(4)
man nm gives, for my platform and for "u":
"u" The symbol is a unique global symbol. This is a GNU extension to the standard set of ELF symbol bindings. For such a symbol the dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use.
I am willing to accept that the linker, when linking several libopentelemetry_*.a static libraries into a final binary, will resolve duplicates, making the singleton unique.
However, there is more to it.
Now with dynamic libraries:
[email protected]:lib64> pwd
/usr/local/lib64
[email protected]:lib64> for L in `ls libopentelemetry_*.so`; do echo $L; nm $L | c++filt | grep GlobalLogHandler::GetHandlerAndLevel; done
libopentelemetry_common.so
libopentelemetry_exporter_ostream_metrics.so
000000000023f788 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
000000000002da83 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
000000000023f760 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_ostream_span.so
0000000000308048 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
00000000000c8765 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000308020 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_otlp_grpc_log.so
libopentelemetry_exporter_otlp_grpc.so
00000000017a82e8 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
0000000000c1d273 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
00000000017a82c0 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_otlp_http_client.so
0000000000727688 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
00000000002ccd8f W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
0000000000727660 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_exporter_otlp_http.so
libopentelemetry_http_client_curl.so
libopentelemetry_metrics.so
00000000004646c8 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
00000000001b0a4f W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
00000000004646a0 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_otlp_recordable.so
libopentelemetry_resources.so
libopentelemetry_trace.so
00000000002e4a08 u guard variable for opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
00000000000b06d1 W opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()
00000000002e49e0 u opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()::handler_and_level
libopentelemetry_version.so
My reading of (4) is that the linker, when producing a -- single -- opentelemetry_xyz.so library, will ensure the symbol there is unique ... for this .so alone.
The linker is never given all the code at once, and can not enforce uniqueness for all the *.so libraries combined.
Looking further, at runtime: how is dlopen() supposed to reconcile uniqueness of a symbol, when defined in multiple *.so shared libraries ?
Note how the symbol is "W" (weak)
"W" "w" The symbol is a weak symbol that has not been specifically tagged as a weak object symbol. When a weak defined symbol is linked with a normal defined symbol, the normal defined symbol is used with no error. When a weak undefined symbol is linked and the symbol is not defined, the value of the symbol is determined in a system-specific manner without error. On some systems, uppercase indicates that a default value has been specified.
This possibly explains why multiple instances of the same singleton are not reported as duplicate symbols.
I was not able to investigate if/how dlopen() resolves this, and whether there are indeed multiple instances of the singleton in the final loaded binary or not, but this is highly suspicious, and the behavior seen is consistent with duplicates.
Suggestions for a fix:
Abandon the idea of defining singleton in header files, use a *.cc file instead.
For GlobalLogHandler::GetHandlerAndLevel(),
implement this in global_log_handler.cc, and adjust makefiles accordingly.
Now, this raises the interesting questions of every other singleton defined in
header files, in particular:
- per signal provider singletons
- global propagator singletons
- internal spin locks singletons
- etc
For example:
trace::Provider::GetTracerProvider()trace::Provider::GetLock()
are at risk of not being unique.
Please consider moving every singleton implementation in a *.cc file,
so it gets defined in only one library (.a or .so), not many.
A good home for these singletons can be libopentelemetry_common.so, possibly libopentelemetry_version.so
Regards.