new redis contrib integration by LeoCavaille · Pull Request #9 · DataDog/dd-trace-py

LeoCavaille · 2016-06-30T13:50:24Z

No description provided.

LotharSee · 2016-06-30T14:10:52Z

The purpose of our __init__.py structure is that you can safely import redis here.

LeoCavaille · 2016-06-30T15:13:25Z

all comments should be fixed

LotharSee · 2016-06-30T15:20:27Z

LGTM

## Description `test_wrapper()` segfaults with following stack backtrace ``` #3 handle_posix_sigaction () at src/collector/crash_handler.rs:108 #4 <signal handler called> #5 0x000075fb182a92b9 in Datadog::Sample::push_release (count=1, lock_time=3896666036706324, this=<optimized out>) at /home/bits/project/ddtrace/internal/datadog/profiling/dd_wrapper/src/sample.cpp:235 #6 Datadog::Sample::push_release (this=<optimized out>, lock_time=3896666036706324, count=1) at /home/bits/project/ddtrace/internal/datadog/profiling/dd_wrapper/src/sample.cpp:231 #7 0x000075fb182bec5c in __pyx_pf_7ddtrace_8internal_7datadog_9profiling_4ddup_5_ddup_12SampleHandle_10push_release ( __pyx_v_count=<optimized out>, __pyx_v_value=0x75fb187c7250, __pyx_v_self=0x75fb187c7630) at /tmp/tmpxso4scef.build-lib/ddtrace.internal.datadog.profiling.ddup._ddup/_ddup.cpp:7956 #8 __pyx_pw_7ddtrace_8internal_7datadog_9profiling_4ddup_5_ddup_12SampleHandle_11push_release ( __pyx_v_self=0x75fb187c7630, __pyx_args=<optimized out>, __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>) at /tmp/tmpxso4scef.build-lib/ddtrace.internal.datadog.profiling.ddup._ddup/_ddup.cpp:7907 #9 0x000075fb1c35b495 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x59d0af703300, ``` As it indexes into empty `values` [vector](https://github.com/DataDog/dd-trace-py/blob/d033f37c48994b061a00748a00724b54a110de39/ddtrace/internal/datadog/profiling/dd_wrapper/include/sample.hpp#L81). It is typically initialized by a call to `ddup.config()`. Shuffle the test code into `TestThreadingLockCollector` class to do so.  ## Testing Run `CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) DD_FAST_BUILD=1 DD_TRACE_AGENT_URL=http://localhost:8126 riot -v run 9bb19fe --pass-env -- -s -vv -k test_wrapper`  ## Risks  ## Additional Notes  [PROF-12678](https://datadoghq.atlassian.net/browse/PROF-12678) [PROF-12678]: https://datadoghq.atlassian.net/browse/PROF-12678?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

## Description https://datadoghq.atlassian.net/browse/PROF-13114 This PR makes sure the Python Profiler's signal handler (for `SIGSEGV` and `SIGBUS`) is properly installed when the Sampler thread starts. Note that this (reinstalling our signal handler) does NOT break any other signal handler (Python's or another extension's) as our signal handler only swallows faults / jumps to the recovery path if it's been "armed" (otherwise it re-raises). What matters is that we should be the "first responder" when a fault happens. This is an attempt to fix a crash we saw in the testing environment where some workloads receive segmentation faults clearly coming from `safe_memcpy` in FastAPI / Django apps. The "real" root cause isn't yet known of me – Django and FastAPI don't seem to use `PYTHONFAULTHANDLER` or `faulthandler` based on the Github code – but after deploying those changes, we stopped seeing those crashes (0 in the past 4 days). <img width="2089" height="480" alt="image" src="https://github.com/user-attachments/assets/c554ec0c-cfae-4311-bded-8082d8f79ed9" /> ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x0000755d4295c18f safe_memcpy #1 0x0000755d42954879 copy_memory #2 0x0000755d42956826 GenInfo::create #3 0x0000755d42958c25 TaskInfo::create #4 0x0000755d42958cc7 TaskInfo::create #5 0x0000755d4295a618 ThreadInfo::unwind_tasks #6 0x0000755d4295b98b std::_Function_handler<void (_ts*, ThreadInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}::operator()(InterpreterInfo&) const::{lambda(_ts*, ThreadInfo&)#1}>::_M_invoke #7 0x0000755d42959906 for_each_thread #8 0x0000755d429599e5 std::_Function_handler<void (InterpreterInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}>::_M_invoke #9 0x0000755d4295c5b2 Datadog::Sampler::sampling_thread #10 0x0000755d4295c685 call_sampling_thread #11 0x0000755d45aeeea7 start_thread #12 0x0000755d45c05def clone ```

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it.

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. (cherry picked from commit 4c69fdd)

…15858) Backport 4c69fdd from #15798 to 4.1. ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. Co-authored-by: Thomas Kowalski <[email protected]>

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. (cherry picked from commit 4c69fdd)

…15861) Backport 4c69fdd from #15798 to 4.0. ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. Co-authored-by: Thomas Kowalski <[email protected]>

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree DataDog#1 0x00007f9a7accc6b5 pthread_create DataDog#2 0x00007f9a7a63aaa5 std::thread::_M_start_thread DataDog#3 0x00007f9a7a639d18 PeriodicThread_start DataDog#4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) DataDog#5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) DataDog#6 0x00007f9a7ad17073 __fork DataDog#7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) DataDog#8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) DataDog#9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) DataDog#10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) DataDog#11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) DataDog#12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) DataDog#13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) DataDog#14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) DataDog#15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) DataDog#16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) DataDog#17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) DataDog#18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) DataDog#19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) DataDog#20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) DataDog#21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) DataDog#22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) DataDog#23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) DataDog#24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) DataDog#25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) DataDog#26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) DataDog#27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) DataDog#28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) DataDog#29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) DataDog#30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) DataDog#31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) DataDog#32 0x00007f9a7b065c25 PyObject_VectorcallMethod DataDog#33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) DataDog#34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) DataDog#35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it.

## Description This fixes a rare (but real, found in Crash Logs) race condition where we would see segmentation faults in `Frame::read`. After analysis, it seems this comes from a race condition where the number of bytes copied in the `StackChunk` did not match (was less than) the number of bytes in the `StackChunk` according to the header, if the `StackChunk` was updated by Python as we were reading it. When this happens, we would try to read from an invalid pointer (we would "believe" our copy of the `StackChunk` had more bytes than it actually did). --- **Example crash stack** ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007fcde855cfc6 Frame::read #1 0x00007fcde855d114 unwind_frame #2 0x00007fcde855ec68 ThreadInfo::unwind #3 0x00007fcde855eda2 ThreadInfo::sample #4 0x00007fcde855f02e std::_Function_handler<void (_ts*, ThreadInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}::operator()(InterpreterInfo&) const::{lambda(_ts*, ThreadInfo&)#1}>::_M_invoke #5 0x00007fcde855f2c0 for_each_thread #6 0x00007fcde855f382 std::_Function_handler<void (InterpreterInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}>::_M_invoke #7 0x00007fcde855c2f9 for_each_interp #8 0x00007fcde855f6e7 Datadog::Sampler::sampling_thread #9 0x00007fcde855f853 call_sampling_thread ``` Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes a crash coming from IAST due to an inconsistent reference count contract between `new_pyobject_id` and its callers, where the callers would expect a new owned reference like it [already does today](https://github.com/DataDog/dd-trace-py/blob/c02775f9db03c05f90356181323d000b86aba7da/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.cpp#L169-L171) but some code paths were missing the `Py_INCREF`, causing segmentation faults (see [example usage](https://github.com/DataDog/dd-trace-py/blob/c02775f9db03c05f90356181323d000b86aba7da/ddtrace/appsec/_iast/_taint_tracking/aspects/aspect_operator_add.cpp#L30-L31)). This error has been around at least since 3.11.0 and is currently causing approximately [50k errors per week](https://app.datadoghq.com/error-tracking/issue/01522162-6bf3-11f0-b96b-da7ad0900002?query=%28%40tags.severity%3Acrash%20OR%20severity%3Acrash%20OR%20signum%3A%2A%20OR%20%40error.is_crash%3Atrue%29%20%40lib_language%3Apython&index=&tb=%40org_id&from_ts=1775841064700&to_ts=1776445864700&live=true). ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x000061335a8b72d4 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2126:1) #1 0x000061335a89e11c PyObject_TypeCheck (/usr/src/python/./Include/object.h:381:36) #2 0x000061335a89e11c object_isinstance (/usr/src/python/Objects/abstract.c:2571:18) #3 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2606:16) #4 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2628:17) #5 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2602:1) #6 0x000061335a89cbeb PyObject_IsInstance (/usr/src/python/Objects/abstract.c:2670:12) #7 0x000061335a8c89ed _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3036:26) #8 0x000061335a98fd11 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #9 0x000061335a884945 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:267:11) #10 0x000061335a8a0bf4 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #11 0x000061335a8a0bf4 object_vacall (/usr/src/python/Objects/call.c:850:14) #12 0x000061335a8fdf8e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:957:14) #13 0x000074f48affeb28 WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024:18) #14 0x000061335a8a12e2 PyObject_Call #15 0x000061335a8cb1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #16 0x000061335a8a3ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #17 0x000061335a8a3ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #18 0x000074f48d48bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22) #19 0x000074f48d48c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11) #20 0x000061335a8af877 cfunction_vectorcall_O (/usr/src/python/Objects/methodobject.c:509:24) #21 0x000074f48a434f69 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15) #22 0x000074f48a434f69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25) #23 0x000074f48a43a96b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25) #24 0x000074f48a434e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24) #25 0x000074f48a436c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19) #26 0x000074f48a452311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1) #27 0x000074f48a44f647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5) #28 0x000074f48a370db5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23) #29 0x000074f48a3d8e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18) #30 0x000074f48a3e9cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18) #31 0x000074f48a3e9cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13) #32 0x000061335a8a159c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #33 0x000061335a8a159c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887:24) #34 0x000074f48a3edd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33768:23) #35 0x000074f48a3ef591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33318:13) #36 0x000061335a8a0a18 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #37 0x000061335a8a0a18 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12) #38 0x000061335a8c7807 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #39 0x000061335a8a3ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #40 0x000061335a8a3ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #41 0x000074f48d48bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22) #42 0x000074f48d48c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11) #43 0x000061335a8a06fe _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18) #44 0x000061335a82380c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:90:16) #45 0x000061335a82380c context_run (/usr/src/python/Python/context.c:668:29) #46 0x000061335a912d7b cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:438:24) #47 0x000061335a8cb1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #48 0x000061335a94a4b9 PyEval_EvalCode (/usr/src/python/Python/ceval.c:578:21) #49 0x000061335a96852c run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1722:9) #50 0x000061335a9684a4 run_mod (/usr/src/python/Python/pythonrun.c:1743:19) #51 0x000061335a968061 pyrun_file (/usr/src/python/Python/pythonrun.c:1643:15) #52 0x000061335a967ea7 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433:13) #53 0x000061335a967cc7 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78:15) #54 0x000061335a972230 pymain_run_file_obj (/usr/src/python/Modules/main.c:360:15) #55 0x000061335a972230 pymain_run_file (/usr/src/python/Modules/main.c:379:15) #56 0x000061335a972230 pymain_run_python (/usr/src/python/Modules/main.c:633:21) #57 0x000061335a972230 Py_RunMain (/usr/src/python/Modules/main.c:713:5) #58 0x000061335a971dbd Py_BytesMain (/usr/src/python/Modules/main.c:767:12) #59 0x000074f48e000e40 __libc_start_main #60 0x000061335a8ea2d5 _start ``` Co-authored-by: thomas.kowalski <[email protected]>

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes the following crash. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00005a87adf31055 dictkeys_get_index (/usr/src/python/Objects/dictobject.c:344:13) #1 0x00005a87adf31055 unicodekeys_lookup_generic (/usr/src/python/Objects/dictobject.c:876:14) #2 0x00005a87adf31055 _Py_dict_lookup (/usr/src/python/Objects/dictobject.c:1056:18) #3 0x00005a87adf31055 PyDict_GetItemWithError (/usr/src/python/Objects/dictobject.c:1789:10) #4 0x00007b2e6fc1a8e0 __pyx_pw_7ddtrace_8internal_9telemetry_18metrics_namespaces_15MetricNamespace_5add_metric #5 0x00005a87ae00c955 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #6 0x00005a87ae00c955 PyObject_Vectorcall (/usr/src/python/Objects/call.c:299:12) #7 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #8 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #9 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #10 0x00005a87ae00d587 PyObject_Call #11 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #12 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #13 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #14 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #15 0x00005a87ae00e0ba _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #16 0x00005a87ae00e0ba method_vectorcall (/usr/src/python/Objects/classobject.c:59:18) #17 0x00005a87ae00c955 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #18 0x00005a87ae00c955 PyObject_Vectorcall (/usr/src/python/Objects/call.c:299:12) #19 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #20 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #21 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #22 0x00005a87ae00d02f _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:393:16) #23 0x00005a87ae00d02f _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:141:15) #24 0x00005a87ae00d02f _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:482:24) #25 0x00005a87ae0bafd9 slot_tp_call (/usr/src/python/Objects/typeobject.c:7623:15) #26 0x00005a87ae00c4d9 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:214:18) #27 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #28 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #29 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #30 0x00005a87ae00d02f _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:393:16) #31 0x00005a87ae00d02f _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:141:15) #32 0x00005a87ae00d02f _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:482:24) #33 0x00005a87ae041f7b slot_tp_init (/usr/src/python/Objects/typeobject.c:7854:15) #34 0x00005a87ae040731 type_call (/usr/src/python/Objects/typeobject.c:1103:19) #35 0x00005a87ae00c4d9 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:214:18) #36 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #37 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #38 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #39 0x00005a87ae00e134 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #40 0x00005a87ae00e134 method_vectorcall (/usr/src/python/Objects/classobject.c:89:18) #41 0x00005a87ae00d587 PyObject_Call #42 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #43 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #44 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #45 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #46 0x00005a87ae00d587 PyObject_Call #47 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #48 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #49 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #50 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #51 0x00005a87ae00e04b _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #52 0x00005a87ae00e04b method_vectorcall (/usr/src/python/Objects/classobject.c:67:20) #53 0x00005a87ae00d587 PyObject_Call #54 0x00005a87ae10e9b3 thread_run (/usr/src/python/./Modules/_threadmodule.c:1124:21) #55 0x00005a87ae0f1098 pythread_wrapper (/usr/src/python/Python/thread_pthread.h:241:5) #56 0x00007b2e71c49609 start_thread #57 0x00007b2e71a14353 clone ``` Co-authored-by: brettlangdon <[email protected]>

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes the following crash. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00005a87adf31055 dictkeys_get_index (/usr/src/python/Objects/dictobject.c:344:13) #1 0x00005a87adf31055 unicodekeys_lookup_generic (/usr/src/python/Objects/dictobject.c:876:14) #2 0x00005a87adf31055 _Py_dict_lookup (/usr/src/python/Objects/dictobject.c:1056:18) #3 0x00005a87adf31055 PyDict_GetItemWithError (/usr/src/python/Objects/dictobject.c:1789:10) #4 0x00007b2e6fc1a8e0 __pyx_pw_7ddtrace_8internal_9telemetry_18metrics_namespaces_15MetricNamespace_5add_metric #5 0x00005a87ae00c955 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #6 0x00005a87ae00c955 PyObject_Vectorcall (/usr/src/python/Objects/call.c:299:12) #7 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #8 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #9 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #10 0x00005a87ae00d587 PyObject_Call #11 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #12 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #13 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #14 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #15 0x00005a87ae00e0ba _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #16 0x00005a87ae00e0ba method_vectorcall (/usr/src/python/Objects/classobject.c:59:18) #17 0x00005a87ae00c955 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #18 0x00005a87ae00c955 PyObject_Vectorcall (/usr/src/python/Objects/call.c:299:12) #19 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #20 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #21 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #22 0x00005a87ae00d02f _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:393:16) #23 0x00005a87ae00d02f _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:141:15) #24 0x00005a87ae00d02f _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:482:24) #25 0x00005a87ae0bafd9 slot_tp_call (/usr/src/python/Objects/typeobject.c:7623:15) #26 0x00005a87ae00c4d9 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:214:18) #27 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #28 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #29 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #30 0x00005a87ae00d02f _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:393:16) #31 0x00005a87ae00d02f _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:141:15) #32 0x00005a87ae00d02f _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:482:24) #33 0x00005a87ae041f7b slot_tp_init (/usr/src/python/Objects/typeobject.c:7854:15) #34 0x00005a87ae040731 type_call (/usr/src/python/Objects/typeobject.c:1103:19) #35 0x00005a87ae00c4d9 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:214:18) #36 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #37 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #38 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #39 0x00005a87ae00e134 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #40 0x00005a87ae00e134 method_vectorcall (/usr/src/python/Objects/classobject.c:89:18) #41 0x00005a87ae00d587 PyObject_Call #42 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #43 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #44 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #45 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #46 0x00005a87ae00d587 PyObject_Call #47 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #48 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #49 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #50 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #51 0x00005a87ae00e04b _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #52 0x00005a87ae00e04b method_vectorcall (/usr/src/python/Objects/classobject.c:67:20) #53 0x00005a87ae00d587 PyObject_Call #54 0x00005a87ae10e9b3 thread_run (/usr/src/python/./Modules/_threadmodule.c:1124:21) #55 0x00005a87ae0f1098 pythread_wrapper (/usr/src/python/Python/thread_pthread.h:241:5) #56 0x00007b2e71c49609 start_thread #57 0x00007b2e71a14353 clone ``` Co-authored-by: brettlangdon <[email protected]>

## Description This PR fixes a crash in the Profiler that occurred when `fork` was called while the Sampling Thread was actively modifying the `LRUCache` for Frames (about 100 crashes per week). ``` Error UnixSignal: Process terminated with SI_TKILL (SIGABRT) #0 0x0000ffffb293cb3c raise #1 0x0000ffffb2927e00 abort #2 0x0000ffffb2996f98 free #3 0x0000ffffa3f47cc4 std::_List_base<std::pair<unsigned long, std::unique_ptr<Frame, std::default_delete<Frame> > >, std::allocator<std::pair<unsigned long, std::unique_ptr<Frame, std::default_delete<Frame> > > > >::_M_clear #4 0x0000ffffa3f50ea4 Datadog::Sampler::postfork_child #5 0x0000ffffa3f51cc0 _stack_atfork_child #6 0x0000ffffb29c1cd4 __libc_fork #7 0x0000aaaabbd4b998 os_fork_impl #8 0x0000aaaabbdaff58 cfunction_vectorcall_NOARGS.llvm.1085662745629047260 #9 0x0000aaaabbe197d4 _PyEval_EvalFrameDefault #10 0x0000aaaabbd87548 _PyObject_VectorcallTstate.llvm.15733451206353458062 #11 0x0000aaaabbd89df8 object_vacall.llvm.15733451206353458062 #12 0x0000aaaabbe7f378 PyObject_CallFunctionObjArgs #13 0x0000ffffb1a4453c WraptFunctionWrapperBase_call (/project/src/wrapt/_wrappers.c:2455:14) #14 0x0000aaaabbe15fe0 _PyEval_EvalFrameDefault #15 0x0000aaaabbdcaa78 slot_tp_init #16 0x0000aaaabbdc32b0 type_call #17 0x0000aaaabbe15fe0 _PyEval_EvalFrameDefault #18 0x0000aaaabbd8a464 method_vectorcall.llvm.15917642511948773313 #19 0x0000aaaabbe197d4 _PyEval_EvalFrameDefault #20 0x0000aaaabbd8e354 gen_iternext #21 0x0000aaaabbde9a58 builtin_next #22 0x0000aaaabbe187d0 _PyEval_EvalFrameDefault #23 0x0000aaaabbd8a4c4 method_vectorcall.llvm.15917642511948773313 #24 0x0000aaaabbe1a02c _PyEval_EvalFrameDefault #25 0x0000aaaabbd8a4c4 method_vectorcall.llvm.15917642511948773313 #26 0x0000aaaabbe1a02c _PyEval_EvalFrameDefault #27 0x0000aaaabbd89480 _PyObject_Call_Prepend #28 0x0000aaaabbebba50 slot_tp_call #29 0x0000aaaabbe15fe0 _PyEval_EvalFrameDefault #30 0x0000aaaabbf15ea4 PyEval_EvalCode #31 0x0000aaaabbf48b8c run_mod.llvm.17615948296688877498 #32 0x0000aaaabbcb34b4 pyrun_file #33 0x0000aaaabbcb288c _PyRun_SimpleFileObject #34 0x0000aaaabbcb2488 _PyRun_AnyFileObject #35 0x0000aaaabbcbfc1c pymain_run_file_obj #36 0x0000aaaabbcbf994 pymain_run_file #37 0x0000aaaabbf60d2c Py_RunMain #38 0x0000aaaabbf611f0 pymain_main.llvm.144270552765144348 #39 0x0000aaaabbe589d0 main #40 0x0000ffffb2928598 __libc_start_main ``` The Sampling Thread is not stopped before forking. This means when `fork` is called, the Sampling Thread could be mutating `frame_cache_` (backed by `std::list` + `std::unordered_map`) via `lookup` (splice) and `store` (`emplace_front`/`pop_back`). As a result, those data structures could end up in a corrupt state at the moment the child process is created. When the child process resumes work, calling `frame_cache_.clear()` post-fork, the crash can happen. The most natural thing to do to avoid this would be to ask the Sampling Thread to stop at fork time, then making sure it is restarted both in the parent and the child post-fork (if it was running). However, this approach has a significant caveat: depending on the current state of adaptive sampling, doing this may result in blocking the fork for up to 100ms (worst case scenario) which is not an acceptable side effect for the user. By not stopping the Thread but cleaning up in the child process, we keep state consistency without having to wait, at the cost of a small one-time leak (the data in the containers that we placement-new). Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes the [following crash](https://app.datadoghq.com/error-tracking/issue/a7910426-126f-11f1-8982-da7ad0900002) that is still happening in recent versions of `ddtrace` and that we attribute to small max stack sizes in certain circumstances. The updated version has the same behaviour without needing recursion, at the cost of having to reverse the container before returning. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007fc46c0e8b5a GenInfo::create_impl #1 0x00007fc46c0e8c28 GenInfo::create_impl #2 0x00007fc46c0e8c28 GenInfo::create_impl #3 0x00007fc46c0e8c28 GenInfo::create_impl #4 0x00007fc46c0e8c28 GenInfo::create_impl #5 0x00007fc46c0e8d2e GenInfo::create #6 0x00007fc46c0e8da3 TaskInfo::create_impl #7 0x00007fc46c0e8f90 TaskInfo::create #8 0x00007fc46c0e911c ThreadInfo::get_all_tasks #9 0x00007fc46c0e97e2 ThreadInfo::unwind_tasks #10 0x00007fc46c0edb21 ThreadInfo::sample #11 0x00007fc46c0edc50 std::_Function_handler<void (_ts*, ThreadInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}::operator()(InterpreterInfo&) const::{lambda(_ts*, ThreadInfo&)#1}>::_M_invoke #12 0x00007fc46c0ea8d0 for_each_thread #13 0x00007fc46c0ea968 std::_Function_handler<void (InterpreterInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}>::_M_invoke #14 0x00007fc46c0e76ad for_each_interp #15 0x00007fc46c0ead36 Datadog::Sampler::sampling_thread #16 0x00007fc46c0eaf9d call_sampling_thread #17 0x00007fc46eef0ea7 start_thread #18 0x00007fc46f006adf clone ``` Co-authored-by: thomas.kowalski <[email protected]>

## What does this PR do? This PR changes the Profiler codebase to use the new two-time upload API (1. create the `tokio` runtime; 2. actually start the upload) to avoid a race condition often reported by TSan the `profiling_native` job -- see DataDog/libdatadog#1733. pr ``` [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from ForkDeathTest [ RUN ] ForkDeathTest.SampleInThreadsAndForkMany /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137: Failure Death test: sample_in_threads_and_fork(16, 10e3) Result: died but not with expected exit code: Exited with exit status 66 Actual msg: [ DEATH ] ================== [ DEATH ] WARNING: ThreadSanitizer: data race (pid=10692) [ DEATH ] Read of size 8 at 0x72b0000100e0 by main thread: [ DEATH ] #0 write <null> (libclang_rt.tsan-x86_64.so+0x82976) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 std::sys::fd::unix::FileDesc::write::h03a0f593da82637a /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/sys/fd/unix.rs:346:13 (_native.cpython-314-x86_64-linux-gnu.so+0x67810b) (BuildId: 46434a56828b14a5896a23eb3c80b267e178a68c) [ DEATH ] #2 std::sys::fs::unix::File::write::hf489ad007e4249a9 /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/sys/fs/unix.rs:1570:16 (_native.cpython-314-x86_64-linux-gnu.so+0x67810b) [ DEATH ] #3 _$LT$$RF$std..fs..File$u20$as$u20$std..io..Write$GT$::write::hddaf57c9e49ca7b2 /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/fs.rs:1364:20 (_native.cpython-314-x86_64-linux-gnu.so+0x67810b) [ DEATH ] #4 Datadog::Uploader::prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader.cpp:197:9 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13856) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #5 ddup_prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:124:5 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x9904) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #6 __run_prefork_handlers posix/register-atfork.c:141:11 (libc.so.6+0xf834e) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] #7 sample_in_threads_and_fork(unsigned int, unsigned int) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:112:17 (test_forking+0xd0e0) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #8 ForkDeathTest_SampleInThreadsAndForkMany_Test::TestBody() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137:5 (test_forking+0xdb3c) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2638:10 (test_forking+0x47478) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2674:14 (test_forking+0x47478) [ DEATH ] #11 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29ca7) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] [ DEATH ] Previous write of size 8 at 0x72b0000100e0 by thread T1 (mutexes: write M0): [ DEATH ] #0 eventfd <null> (libclang_rt.tsan-x86_64.so+0x79e84) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 new_unregistered /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/mod.rs:8:28 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) (BuildId: 46434a56828b14a5896a23eb3c80b267e178a68c) [ DEATH ] #2 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/waker/eventfd.rs:27:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #3 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/waker.rs:87:9 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #4 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/io/driver.rs:120:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #5 create_io_stack /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:144:42 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #6 tokio::runtime::driver::Driver::new::h35527ab9443fd0a3 /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:47:52 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #7 ddup_upload /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:480:28 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0xa636) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #8 upload_in_thread(void*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:12:5 (test_forking+0xcbd4) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #9 <null> <null> (libclang_rt.tsan-x86_64.so+0x7495b) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] [ DEATH ] As if synchronized via sleep: [ DEATH ] #0 nanosleep <null> (libclang_rt.tsan-x86_64.so+0x71c34) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l>>(std::chrono::duration<long, std::ratio<1l, 1000000l>> const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/this_thread_sleep.h:80:9 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13879) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #2 Datadog::Uploader::prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader.cpp:198:9 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13879) [ DEATH ] #3 ddup_prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:124:5 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x9904) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #4 __run_prefork_handlers posix/register-atfork.c:141:11 (libc.so.6+0xf834e) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] #5 sample_in_threads_and_fork(unsigned int, unsigned int) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:112:17 (test_forking+0xd0e0) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #6 ForkDeathTest_SampleInThreadsAndForkMany_Test::TestBody() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137:5 (test_forking+0xdb3c) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2638:10 (test_forking+0x47478) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #8 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2674:14 (test_forking+0x47478) [ DEATH ] #9 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29ca7) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] [ DEATH ] Location is file descriptor 7 created by thread T1 at: [ DEATH ] #0 eventfd <null> (libclang_rt.tsan-x86_64.so+0x79e84) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 new_unregistered /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/mod.rs:8:28 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) (BuildId: 46434a56828b14a5896a23eb3c80b267e178a68c) [ DEATH ] #2 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/waker/eventfd.rs:27:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #3 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/waker.rs:87:9 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #4 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/io/driver.rs:120:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #5 create_io_stack /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:144:42 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #6 tokio::runtime::driver::Driver::new::h35527ab9443fd0a3 /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:47:52 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #7 ddup_upload /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:480:28 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0xa636) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #8 upload_in_thread(void*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:12:5 (test_forking+0xcbd4) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #9 <null> <null> (libclang_rt.tsan-x86_64.so+0x7495b) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] [ DEATH ] Mutex M0 (0x7ffff6b117e8) created at: [ DEATH ] #0 pthread_mutex_lock <null> (libclang_rt.tsan-x86_64.so+0x76782) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 __gthread_mutex_lock(pthread_mutex_t*) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/x86_64-linux-gnu/c++/14/bits/gthr-default.h:762:12 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13766) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #2 std::mutex::lock() /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_mutex.h:113:17 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13766) [ DEATH ] #3 Datadog::Uploader::lock() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader.cpp:160:17 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13766) [ DEATH ] #4 ddup_upload /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:455:5 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0xa4d2) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #5 upload_in_thread(void*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:12:5 (test_forking+0xcbd4) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #6 <null> <null> (libclang_rt.tsan-x86_64.so+0x7495b) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] [ DEATH ] Thread T1 (tid=10744, running) created by main thread at: [ DEATH ] #0 pthread_create <null> (libclang_rt.tsan-x86_64.so+0x74a05) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 sample_in_threads_and_fork(unsigned int, unsigned int) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:98:9 (test_forking+0xd082) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #2 ForkDeathTest_SampleInThreadsAndForkMany_Test::TestBody() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137:5 (test_forking+0xdb3c) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #3 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2638:10 (test_forking+0x47478) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2674:14 (test_forking+0x47478) [ DEATH ] #5 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29ca7) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] [ DEATH ] SUMMARY: ThreadSanitizer: data race /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/sys/fd/unix.rs:346:13 in std::sys::fd::unix::FileDesc::write::h03a0f593da82637a [ DEATH ] ================== [ DEATH ] Error uploading (ddog_prof_Exporter_send_blocking failed: Failed to send HTTP request: error sending request for url (https://127.0.0.1:9126/profiling/v1/input): client error (Connect): tcp connect error: Connection refused (os error 111)) [ DEATH ] Error uploading (ddog_prof_Exporter_send_blocking failed: Failed to send HTTP request: error sending request for url (https://127.0.0.1:9126/profiling/v1/input): client error (Connect): tcp connect error: Connection refused (os error 111)) [ DEATH ] ThreadSanitizer: reported 1 warnings [ DEATH ] Error uploading (ddog_prof_Exporter_send_blocking failed: Failed to send HTTP request: error sending request for url (https://127.0.0.1:9126/profiling/v1/input): client error (Connect): tcp connect error: Connection refused (os error 111)) [ DEATH ] ThreadSanitizer: reported 1 warnings [ DEATH ] [ FAILED ] ForkDeathTest.SampleInThreadsAndForkMany (1759 ms) [----------] 1 test from ForkDeathTest (1759 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (1762 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] ForkDeathTest.SampleInThreadsAndForkMany ``` The problem was that in some cases, we could have a thread trying to use a cancellation token to cancel an ongoing upload before the associated `tokio` runtime had finished initialising, leading to that race condition. This same race condition was also responsible for some crashes reported by Crash Tracking. **Note** this PR currently points to a specific commit SHA in `libdatadog` -- before merging it we'll have to update that SHA and upgrade `libdatadog` in `dd-trace-py` to be able to use that new API. Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes a crash coming from IAST due to an inconsistent reference count contract between `new_pyobject_id` and its callers, where the callers would expect a new owned reference like it [already does today](https://github.com/DataDog/dd-trace-py/blob/c02775f9db03c05f90356181323d000b86aba7da/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.cpp#L169-L171) but some code paths were missing the `Py_INCREF`, causing segmentation faults (see [example usage](https://github.com/DataDog/dd-trace-py/blob/c02775f9db03c05f90356181323d000b86aba7da/ddtrace/appsec/_iast/_taint_tracking/aspects/aspect_operator_add.cpp#L30-L31)). This error has been around at least since 3.11.0 and is currently causing approximately [50k errors per week](https://app.datadoghq.com/error-tracking/issue/01522162-6bf3-11f0-b96b-da7ad0900002?query=%28%40tags.severity%3Acrash%20OR%20severity%3Acrash%20OR%20signum%3A%2A%20OR%20%40error.is_crash%3Atrue%29%20%40lib_language%3Apython&index=&tb=%40org_id&from_ts=1775841064700&to_ts=1776445864700&live=true). ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x000061335a8b72d4 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2126:1) #1 0x000061335a89e11c PyObject_TypeCheck (/usr/src/python/./Include/object.h:381:36) #2 0x000061335a89e11c object_isinstance (/usr/src/python/Objects/abstract.c:2571:18) #3 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2606:16) #4 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2628:17) #5 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2602:1) #6 0x000061335a89cbeb PyObject_IsInstance (/usr/src/python/Objects/abstract.c:2670:12) #7 0x000061335a8c89ed _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3036:26) #8 0x000061335a98fd11 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #9 0x000061335a884945 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:267:11) #10 0x000061335a8a0bf4 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #11 0x000061335a8a0bf4 object_vacall (/usr/src/python/Objects/call.c:850:14) #12 0x000061335a8fdf8e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:957:14) #13 0x000074f48affeb28 WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024:18) #14 0x000061335a8a12e2 PyObject_Call #15 0x000061335a8cb1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #16 0x000061335a8a3ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #17 0x000061335a8a3ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #18 0x000074f48d48bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22) #19 0x000074f48d48c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11) #20 0x000061335a8af877 cfunction_vectorcall_O (/usr/src/python/Objects/methodobject.c:509:24) #21 0x000074f48a434f69 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15) #22 0x000074f48a434f69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25) #23 0x000074f48a43a96b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25) #24 0x000074f48a434e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24) #25 0x000074f48a436c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19) #26 0x000074f48a452311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1) #27 0x000074f48a44f647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5) #28 0x000074f48a370db5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23) #29 0x000074f48a3d8e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18) #30 0x000074f48a3e9cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18) #31 0x000074f48a3e9cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13) #32 0x000061335a8a159c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #33 0x000061335a8a159c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887:24) #34 0x000074f48a3edd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33768:23) #35 0x000074f48a3ef591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33318:13) #36 0x000061335a8a0a18 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #37 0x000061335a8a0a18 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12) #38 0x000061335a8c7807 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #39 0x000061335a8a3ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #40 0x000061335a8a3ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #41 0x000074f48d48bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22) #42 0x000074f48d48c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11) #43 0x000061335a8a06fe _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18) #44 0x000061335a82380c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:90:16) #45 0x000061335a82380c context_run (/usr/src/python/Python/context.c:668:29) #46 0x000061335a912d7b cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:438:24) #47 0x000061335a8cb1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #48 0x000061335a94a4b9 PyEval_EvalCode (/usr/src/python/Python/ceval.c:578:21) #49 0x000061335a96852c run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1722:9) #50 0x000061335a9684a4 run_mod (/usr/src/python/Python/pythonrun.c:1743:19) #51 0x000061335a968061 pyrun_file (/usr/src/python/Python/pythonrun.c:1643:15) #52 0x000061335a967ea7 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433:13) #53 0x000061335a967cc7 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78:15) #54 0x000061335a972230 pymain_run_file_obj (/usr/src/python/Modules/main.c:360:15) #55 0x000061335a972230 pymain_run_file (/usr/src/python/Modules/main.c:379:15) #56 0x000061335a972230 pymain_run_python (/usr/src/python/Modules/main.c:633:21) #57 0x000061335a972230 Py_RunMain (/usr/src/python/Modules/main.c:713:5) #58 0x000061335a971dbd Py_BytesMain (/usr/src/python/Modules/main.c:767:12) #59 0x000074f48e000e40 __libc_start_main #60 0x000061335a8ea2d5 _start ``` Co-authored-by: thomas.kowalski <[email protected]>

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes the following crash. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00005a87adf31055 dictkeys_get_index (/usr/src/python/Objects/dictobject.c:344:13) #1 0x00005a87adf31055 unicodekeys_lookup_generic (/usr/src/python/Objects/dictobject.c:876:14) #2 0x00005a87adf31055 _Py_dict_lookup (/usr/src/python/Objects/dictobject.c:1056:18) #3 0x00005a87adf31055 PyDict_GetItemWithError (/usr/src/python/Objects/dictobject.c:1789:10) #4 0x00007b2e6fc1a8e0 __pyx_pw_7ddtrace_8internal_9telemetry_18metrics_namespaces_15MetricNamespace_5add_metric #5 0x00005a87ae00c955 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #6 0x00005a87ae00c955 PyObject_Vectorcall (/usr/src/python/Objects/call.c:299:12) #7 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #8 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #9 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #10 0x00005a87ae00d587 PyObject_Call #11 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #12 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #13 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #14 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #15 0x00005a87ae00e0ba _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #16 0x00005a87ae00e0ba method_vectorcall (/usr/src/python/Objects/classobject.c:59:18) #17 0x00005a87ae00c955 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #18 0x00005a87ae00c955 PyObject_Vectorcall (/usr/src/python/Objects/call.c:299:12) #19 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #20 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #21 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #22 0x00005a87ae00d02f _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:393:16) #23 0x00005a87ae00d02f _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:141:15) #24 0x00005a87ae00d02f _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:482:24) #25 0x00005a87ae0bafd9 slot_tp_call (/usr/src/python/Objects/typeobject.c:7623:15) #26 0x00005a87ae00c4d9 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:214:18) #27 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #28 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #29 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #30 0x00005a87ae00d02f _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:393:16) #31 0x00005a87ae00d02f _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:141:15) #32 0x00005a87ae00d02f _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:482:24) #33 0x00005a87ae041f7b slot_tp_init (/usr/src/python/Objects/typeobject.c:7854:15) #34 0x00005a87ae040731 type_call (/usr/src/python/Objects/typeobject.c:1103:19) #35 0x00005a87ae00c4d9 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:214:18) #36 0x00005a87ae05ee1d _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:4760:23) #37 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #38 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #39 0x00005a87ae00e134 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #40 0x00005a87ae00e134 method_vectorcall (/usr/src/python/Objects/classobject.c:89:18) #41 0x00005a87ae00d587 PyObject_Call #42 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #43 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #44 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #45 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #46 0x00005a87ae00d587 PyObject_Call #47 0x00005a87ae061fd8 do_call_core (/usr/src/python/Python/ceval.c:7343:12) #48 0x00005a87ae061fd8 _PyEval_EvalFrameDefault (/usr/src/python/Python/ceval.c:5367:22) #49 0x00005a87ae05e060 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:73:16) #50 0x00005a87ae05e060 _PyEval_Vector (/usr/src/python/Python/ceval.c:6425:24) #51 0x00005a87ae00e04b _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #52 0x00005a87ae00e04b method_vectorcall (/usr/src/python/Objects/classobject.c:67:20) #53 0x00005a87ae00d587 PyObject_Call #54 0x00005a87ae10e9b3 thread_run (/usr/src/python/./Modules/_threadmodule.c:1124:21) #55 0x00005a87ae0f1098 pythread_wrapper (/usr/src/python/Python/thread_pthread.h:241:5) #56 0x00007b2e71c49609 start_thread #57 0x00007b2e71a14353 clone ``` Co-authored-by: brettlangdon <[email protected]>

## Description This PR fixes a crash in the Profiler that occurred when `fork` was called while the Sampling Thread was actively modifying the `LRUCache` for Frames (about 100 crashes per week). ``` Error UnixSignal: Process terminated with SI_TKILL (SIGABRT) #0 0x0000ffffb293cb3c raise #1 0x0000ffffb2927e00 abort #2 0x0000ffffb2996f98 free #3 0x0000ffffa3f47cc4 std::_List_base<std::pair<unsigned long, std::unique_ptr<Frame, std::default_delete<Frame> > >, std::allocator<std::pair<unsigned long, std::unique_ptr<Frame, std::default_delete<Frame> > > > >::_M_clear #4 0x0000ffffa3f50ea4 Datadog::Sampler::postfork_child #5 0x0000ffffa3f51cc0 _stack_atfork_child #6 0x0000ffffb29c1cd4 __libc_fork #7 0x0000aaaabbd4b998 os_fork_impl #8 0x0000aaaabbdaff58 cfunction_vectorcall_NOARGS.llvm.1085662745629047260 #9 0x0000aaaabbe197d4 _PyEval_EvalFrameDefault #10 0x0000aaaabbd87548 _PyObject_VectorcallTstate.llvm.15733451206353458062 #11 0x0000aaaabbd89df8 object_vacall.llvm.15733451206353458062 #12 0x0000aaaabbe7f378 PyObject_CallFunctionObjArgs #13 0x0000ffffb1a4453c WraptFunctionWrapperBase_call (/project/src/wrapt/_wrappers.c:2455:14) #14 0x0000aaaabbe15fe0 _PyEval_EvalFrameDefault #15 0x0000aaaabbdcaa78 slot_tp_init #16 0x0000aaaabbdc32b0 type_call #17 0x0000aaaabbe15fe0 _PyEval_EvalFrameDefault #18 0x0000aaaabbd8a464 method_vectorcall.llvm.15917642511948773313 #19 0x0000aaaabbe197d4 _PyEval_EvalFrameDefault #20 0x0000aaaabbd8e354 gen_iternext #21 0x0000aaaabbde9a58 builtin_next #22 0x0000aaaabbe187d0 _PyEval_EvalFrameDefault #23 0x0000aaaabbd8a4c4 method_vectorcall.llvm.15917642511948773313 #24 0x0000aaaabbe1a02c _PyEval_EvalFrameDefault #25 0x0000aaaabbd8a4c4 method_vectorcall.llvm.15917642511948773313 #26 0x0000aaaabbe1a02c _PyEval_EvalFrameDefault #27 0x0000aaaabbd89480 _PyObject_Call_Prepend #28 0x0000aaaabbebba50 slot_tp_call #29 0x0000aaaabbe15fe0 _PyEval_EvalFrameDefault #30 0x0000aaaabbf15ea4 PyEval_EvalCode #31 0x0000aaaabbf48b8c run_mod.llvm.17615948296688877498 #32 0x0000aaaabbcb34b4 pyrun_file #33 0x0000aaaabbcb288c _PyRun_SimpleFileObject #34 0x0000aaaabbcb2488 _PyRun_AnyFileObject #35 0x0000aaaabbcbfc1c pymain_run_file_obj #36 0x0000aaaabbcbf994 pymain_run_file #37 0x0000aaaabbf60d2c Py_RunMain #38 0x0000aaaabbf611f0 pymain_main.llvm.144270552765144348 #39 0x0000aaaabbe589d0 main #40 0x0000ffffb2928598 __libc_start_main ``` The Sampling Thread is not stopped before forking. This means when `fork` is called, the Sampling Thread could be mutating `frame_cache_` (backed by `std::list` + `std::unordered_map`) via `lookup` (splice) and `store` (`emplace_front`/`pop_back`). As a result, those data structures could end up in a corrupt state at the moment the child process is created. When the child process resumes work, calling `frame_cache_.clear()` post-fork, the crash can happen. The most natural thing to do to avoid this would be to ask the Sampling Thread to stop at fork time, then making sure it is restarted both in the parent and the child post-fork (if it was running). However, this approach has a significant caveat: depending on the current state of adaptive sampling, doing this may result in blocking the fork for up to 100ms (worst case scenario) which is not an acceptable side effect for the user. By not stopping the Thread but cleaning up in the child process, we keep state consistency without having to wait, at the cost of a small one-time leak (the data in the containers that we placement-new). Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR fixes the [following crash](https://app.datadoghq.com/error-tracking/issue/a7910426-126f-11f1-8982-da7ad0900002) that is still happening in recent versions of `ddtrace` and that we attribute to small max stack sizes in certain circumstances. The updated version has the same behaviour without needing recursion, at the cost of having to reverse the container before returning. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007fc46c0e8b5a GenInfo::create_impl #1 0x00007fc46c0e8c28 GenInfo::create_impl #2 0x00007fc46c0e8c28 GenInfo::create_impl #3 0x00007fc46c0e8c28 GenInfo::create_impl #4 0x00007fc46c0e8c28 GenInfo::create_impl #5 0x00007fc46c0e8d2e GenInfo::create #6 0x00007fc46c0e8da3 TaskInfo::create_impl #7 0x00007fc46c0e8f90 TaskInfo::create #8 0x00007fc46c0e911c ThreadInfo::get_all_tasks #9 0x00007fc46c0e97e2 ThreadInfo::unwind_tasks #10 0x00007fc46c0edb21 ThreadInfo::sample #11 0x00007fc46c0edc50 std::_Function_handler<void (_ts*, ThreadInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}::operator()(InterpreterInfo&) const::{lambda(_ts*, ThreadInfo&)#1}>::_M_invoke #12 0x00007fc46c0ea8d0 for_each_thread #13 0x00007fc46c0ea968 std::_Function_handler<void (InterpreterInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}>::_M_invoke #14 0x00007fc46c0e76ad for_each_interp #15 0x00007fc46c0ead36 Datadog::Sampler::sampling_thread #16 0x00007fc46c0eaf9d call_sampling_thread #17 0x00007fc46eef0ea7 start_thread #18 0x00007fc46f006adf clone ``` Co-authored-by: thomas.kowalski <[email protected]>

## What does this PR do? This PR changes the Profiler codebase to use the new two-time upload API (1. create the `tokio` runtime; 2. actually start the upload) to avoid a race condition often reported by TSan the `profiling_native` job -- see DataDog/libdatadog#1733. pr ``` [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from ForkDeathTest [ RUN ] ForkDeathTest.SampleInThreadsAndForkMany /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137: Failure Death test: sample_in_threads_and_fork(16, 10e3) Result: died but not with expected exit code: Exited with exit status 66 Actual msg: [ DEATH ] ================== [ DEATH ] WARNING: ThreadSanitizer: data race (pid=10692) [ DEATH ] Read of size 8 at 0x72b0000100e0 by main thread: [ DEATH ] #0 write <null> (libclang_rt.tsan-x86_64.so+0x82976) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 std::sys::fd::unix::FileDesc::write::h03a0f593da82637a /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/sys/fd/unix.rs:346:13 (_native.cpython-314-x86_64-linux-gnu.so+0x67810b) (BuildId: 46434a56828b14a5896a23eb3c80b267e178a68c) [ DEATH ] #2 std::sys::fs::unix::File::write::hf489ad007e4249a9 /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/sys/fs/unix.rs:1570:16 (_native.cpython-314-x86_64-linux-gnu.so+0x67810b) [ DEATH ] #3 _$LT$$RF$std..fs..File$u20$as$u20$std..io..Write$GT$::write::hddaf57c9e49ca7b2 /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/fs.rs:1364:20 (_native.cpython-314-x86_64-linux-gnu.so+0x67810b) [ DEATH ] #4 Datadog::Uploader::prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader.cpp:197:9 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13856) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #5 ddup_prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:124:5 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x9904) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #6 __run_prefork_handlers posix/register-atfork.c:141:11 (libc.so.6+0xf834e) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] #7 sample_in_threads_and_fork(unsigned int, unsigned int) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:112:17 (test_forking+0xd0e0) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #8 ForkDeathTest_SampleInThreadsAndForkMany_Test::TestBody() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137:5 (test_forking+0xdb3c) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2638:10 (test_forking+0x47478) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2674:14 (test_forking+0x47478) [ DEATH ] #11 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29ca7) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] [ DEATH ] Previous write of size 8 at 0x72b0000100e0 by thread T1 (mutexes: write M0): [ DEATH ] #0 eventfd <null> (libclang_rt.tsan-x86_64.so+0x79e84) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 new_unregistered /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/mod.rs:8:28 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) (BuildId: 46434a56828b14a5896a23eb3c80b267e178a68c) [ DEATH ] #2 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/waker/eventfd.rs:27:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #3 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/waker.rs:87:9 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #4 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/io/driver.rs:120:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #5 create_io_stack /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:144:42 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #6 tokio::runtime::driver::Driver::new::h35527ab9443fd0a3 /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:47:52 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #7 ddup_upload /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:480:28 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0xa636) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #8 upload_in_thread(void*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:12:5 (test_forking+0xcbd4) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #9 <null> <null> (libclang_rt.tsan-x86_64.so+0x7495b) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] [ DEATH ] As if synchronized via sleep: [ DEATH ] #0 nanosleep <null> (libclang_rt.tsan-x86_64.so+0x71c34) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l>>(std::chrono::duration<long, std::ratio<1l, 1000000l>> const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/this_thread_sleep.h:80:9 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13879) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #2 Datadog::Uploader::prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader.cpp:198:9 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13879) [ DEATH ] #3 ddup_prefork() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:124:5 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x9904) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #4 __run_prefork_handlers posix/register-atfork.c:141:11 (libc.so.6+0xf834e) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] #5 sample_in_threads_and_fork(unsigned int, unsigned int) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:112:17 (test_forking+0xd0e0) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #6 ForkDeathTest_SampleInThreadsAndForkMany_Test::TestBody() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137:5 (test_forking+0xdb3c) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2638:10 (test_forking+0x47478) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #8 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2674:14 (test_forking+0x47478) [ DEATH ] #9 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29ca7) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] [ DEATH ] Location is file descriptor 7 created by thread T1 at: [ DEATH ] #0 eventfd <null> (libclang_rt.tsan-x86_64.so+0x79e84) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 new_unregistered /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/mod.rs:8:28 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) (BuildId: 46434a56828b14a5896a23eb3c80b267e178a68c) [ DEATH ] #2 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/sys/unix/waker/eventfd.rs:27:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #3 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mio-1.1.0/src/waker.rs:87:9 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #4 new /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/io/driver.rs:120:21 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #5 create_io_stack /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:144:42 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #6 tokio::runtime::driver::Driver::new::h35527ab9443fd0a3 /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/driver.rs:47:52 (_native.cpython-314-x86_64-linux-gnu.so+0x6957ad) [ DEATH ] #7 ddup_upload /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:480:28 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0xa636) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #8 upload_in_thread(void*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:12:5 (test_forking+0xcbd4) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #9 <null> <null> (libclang_rt.tsan-x86_64.so+0x7495b) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] [ DEATH ] Mutex M0 (0x7ffff6b117e8) created at: [ DEATH ] #0 pthread_mutex_lock <null> (libclang_rt.tsan-x86_64.so+0x76782) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 __gthread_mutex_lock(pthread_mutex_t*) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/x86_64-linux-gnu/c++/14/bits/gthr-default.h:762:12 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13766) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #2 std::mutex::lock() /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_mutex.h:113:17 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13766) [ DEATH ] #3 Datadog::Uploader::lock() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader.cpp:160:17 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0x13766) [ DEATH ] #4 ddup_upload /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/src/ddup_interface.cpp:455:5 (libdd_wrapper.cpython-314-x86_64-linux-gnu.so+0xa4d2) (BuildId: d897612444f657ca175c65879a25c92ca5723dd7) [ DEATH ] #5 upload_in_thread(void*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:12:5 (test_forking+0xcbd4) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #6 <null> <null> (libclang_rt.tsan-x86_64.so+0x7495b) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] [ DEATH ] Thread T1 (tid=10744, running) created by main thread at: [ DEATH ] #0 pthread_create <null> (libclang_rt.tsan-x86_64.so+0x74a05) (BuildId: 229b67347f9d16547489a41e6cdd38bf904a70b2) [ DEATH ] #1 sample_in_threads_and_fork(unsigned int, unsigned int) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:98:9 (test_forking+0xd082) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #2 ForkDeathTest_SampleInThreadsAndForkMany_Test::TestBody() /go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/dd_wrapper/test/test_forking.cpp:137:5 (test_forking+0xdb3c) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #3 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2638:10 (test_forking+0x47478) (BuildId: 8619e3219ddfb766594b8d595ec0c44feb82c6b0) [ DEATH ] #4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /go/src/github.com/DataDog/apm-reliability/dd-trace-py/build/dd_wrapper/_deps/googletest-src/googletest/src/gtest.cc:2674:14 (test_forking+0x47478) [ DEATH ] #5 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29ca7) (BuildId: fce446c9d4ad48e2b0c90cce1a11722897805281) [ DEATH ] [ DEATH ] SUMMARY: ThreadSanitizer: data race /rustc/01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf/library/std/src/sys/fd/unix.rs:346:13 in std::sys::fd::unix::FileDesc::write::h03a0f593da82637a [ DEATH ] ================== [ DEATH ] Error uploading (ddog_prof_Exporter_send_blocking failed: Failed to send HTTP request: error sending request for url (https://127.0.0.1:9126/profiling/v1/input): client error (Connect): tcp connect error: Connection refused (os error 111)) [ DEATH ] Error uploading (ddog_prof_Exporter_send_blocking failed: Failed to send HTTP request: error sending request for url (https://127.0.0.1:9126/profiling/v1/input): client error (Connect): tcp connect error: Connection refused (os error 111)) [ DEATH ] ThreadSanitizer: reported 1 warnings [ DEATH ] Error uploading (ddog_prof_Exporter_send_blocking failed: Failed to send HTTP request: error sending request for url (https://127.0.0.1:9126/profiling/v1/input): client error (Connect): tcp connect error: Connection refused (os error 111)) [ DEATH ] ThreadSanitizer: reported 1 warnings [ DEATH ] [ FAILED ] ForkDeathTest.SampleInThreadsAndForkMany (1759 ms) [----------] 1 test from ForkDeathTest (1759 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (1762 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] ForkDeathTest.SampleInThreadsAndForkMany ``` The problem was that in some cases, we could have a thread trying to use a cancellation token to cancel an ongoing upload before the associated `tokio` runtime had finished initialising, leading to that race condition. This same race condition was also responsible for some crashes reported by Crash Tracking. **Note** this PR currently points to a specific commit SHA in `libdatadog` -- before merging it we'll have to update that SHA and upgrade `libdatadog` in `dd-trace-py` to be able to use that new API. Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR ensures we clear the string tables post-fork. This is meant to address a crash happening on (at least) 4.8.0rc5. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x0000000000000000 std::__detail::_Hashtable_base<unsigned long, std::pair<unsigned long const, std::string>, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<false, false, true> >::_M_equals(unsigned long const&, unsigned long, std::__detail::_Hash_node<std::pair<unsigned long const, std::string>, false>*) const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable_policy.h:1804) #1 0x000073afd44ec124 std::__detail::_Hashtable_base<unsigned long, std::pair<unsigned long const, std::string>, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<false, false, true> >::_M_equals(unsigned long const&, unsigned long, std::__detail::_Hash_node<std::pair<unsigned long const, std::string>, false>*) const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable_policy.h:1804) #2 0x000073afd44ec124 std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const [clone .constprop.0] (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:1580) #3 0x000073afd44ece4b std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_node(unsigned long, unsigned long const&, unsigned long) const [clone .isra.0] (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:694) #4 0x0000000000000000 std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:1455) #5 0x0000000000000000 std::unordered_map<unsigned long, std::string, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::string> > >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unordered_map.h:920) #6 0x000073afd44e4e61 std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:1455) #7 0x000073afd44e4e61 std::unordered_map<unsigned long, std::string, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::string> > >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unordered_map.h:920) #8 0x000073afd44e4e61 StringTable::key(_object*, StringTag) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/strings.cc:58) #9 0x000073afd44e517f Frame::create(EchionSampler&, PyCodeObject*, int) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc:24) #10 0x000073afd44e5436 Frame::get(EchionSampler&, PyCodeObject*, int) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc:214) #11 0x000073afd44e55b1 Frame::read(EchionSampler&, _PyInterpreterFrame*, _PyInterpreterFrame**) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc:153) #12 0x000073afd44e56c0 unwind_frame(EchionSampler&, _object*, FrameStack&, unsigned long) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/stacks.cc:56) #13 0x000073afd44e5859 unwind_python_stack(EchionSampler&, _ts*, FrameStack&) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/stacks.cc:103) #14 0x000073afd44e7bbf ThreadInfo::unwind(EchionSampler&, _ts*) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc:15) #15 0x0000000000000000 __gnu_cxx::__normal_iterator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const*, std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > > >::__normal_iterator(std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const* const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_iterator.h:979) #16 0x0000000000000000 std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::end() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:839) #17 0x0000000000000000 std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::empty() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:1008) #18 0x000073afd44eb2eb __gnu_cxx::__normal_iterator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const*, std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > > >::__normal_iterator(std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const* const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_iterator.h:979) #19 0x000073afd44eb2eb std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::end() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:839) #20 0x000073afd44eb2eb std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::empty() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:1008) #21 0x000073afd44eb2eb ThreadInfo::sample(EchionSampler&, _ts*, long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc:698) #22 0x000073afd44eb789 Datadog::Sampler::sampling_thread(unsigned long) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/sampler.cpp:287) #23 0x000073afd44ebad0 call_sampling_thread(void*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/sampler.cpp:45) #24 0x000073afdf06bac3 start_thread (nptl/nptl/pthread_create.c:442) #25 0x000073afdf0fd8d0 __clone3 ``` Co-authored-by: thomas.kowalski <[email protected]>

## Description This fixes a crash that would occur when using the Kafka integration. **Python stacks** ``` Thread 0x00007f2e5affd6c0 (most recent call first): File "/usr/lib/python3.12/threading.py", line 355 in wait File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/_queue.py", line 242 in get File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/worker.py", line 120 in _target File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f2e797fa6c0 (most recent call first): File "/opt/venv/lib/python3.12/site-packages/ddtrace/contrib/internal/kafka/patch.py", line 232 in traced_poll_or_consume File "/opt/app/pkgs/extensions/kafka/confluent_kafka_consumer.py", line 136 in consume File "/opt/app/suma/kafka/consumer/runner.py", line 158 in consume_messages File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 245 in poll_messages File "/opt/app/suma/kafka/consumer/runner.py", line 333 in _run File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 325 in run File "/opt/app/suma/kafka/consumer_cmd.py", line 156 in run_consumer_thread File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f2e7bfff6c0 (most recent call first): File "/opt/venv/lib/python3.12/site-packages/ddtrace/contrib/internal/kafka/patch.py", line 232 in traced_poll_or_consume File "/opt/app/pkgs/extensions/kafka/confluent_kafka_consumer.py", line 136 in consume File "/opt/app/suma/kafka/consumer/runner.py", line 158 in consume_messages File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 245 in poll_messages File "/opt/app/suma/kafka/consumer/runner.py", line 333 in _run File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 325 in run File "/opt/app/suma/kafka/consumer_cmd.py", line 156 in run_consumer_thread File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f2ed9ffb6c0 (most recent call first): File "/opt/venv/lib/python3.12/site-packages/ddtrace/contrib/internal/kafka/patch.py", line 232 in traced_poll_or_consume File "/opt/app/pkgs/extensions/kafka/confluent_kafka_consumer.py", line 136 in consume File "/opt/app/suma/kafka/consumer/runner.py", line 158 in consume_messages File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 245 in poll_messages File "/opt/app/suma/kafka/consumer/runner.py", line 333 in _run File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 325 in run File "/opt/app/suma/kafka/consumer_cmd.py", line 156 in run_consumer_thread File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f2ef0d016c0 (most recent call first): File "/opt/venv/lib/python3.12/site-packages/ddtrace/contrib/internal/kafka/patch.py", line 232 in traced_poll_or_consume File "/opt/app/pkgs/extensions/kafka/confluent_kafka_consumer.py", line 136 in consume File "/opt/app/suma/kafka/consumer/runner.py", line 158 in consume_messages File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 245 in poll_messages File "/opt/app/suma/kafka/consumer/runner.py", line 333 in _run File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 325 in run File "/opt/app/suma/kafka/consumer_cmd.py", line 156 in run_consumer_thread File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f2ed97fa6c0 (most recent call first): File "/opt/venv/lib/python3.12/site-packages/ddtrace/contrib/internal/kafka/patch.py", line 232 in traced_poll_or_consume File "/opt/app/pkgs/extensions/kafka/confluent_kafka_consumer.py", line 136 in consume File "/opt/app/suma/kafka/consumer/runner.py", line 158 in consume_messages File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 245 in poll_messages File "/opt/app/suma/kafka/consumer/runner.py", line 333 in _run File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 325 in run File "/opt/app/suma/kafka/consumer_cmd.py", line 156 in run_consumer_thread File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f2edbfff6c0 (most recent call first): File "/opt/venv/lib/python3.12/site-packages/ddtrace/contrib/internal/kafka/patch.py", line 232 in traced_poll_or_consume File "/opt/app/pkgs/extensions/kafka/confluent_kafka_consumer.py", line 136 in consume File "/opt/app/suma/kafka/consumer/runner.py", line 158 in consume_messages File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 245 in poll_messages File "/opt/app/suma/kafka/consumer/runner.py", line 333 in _run File "/opt/venv/lib/python3.12/site-packages/ddtrace/_trace/tracer.py", line 860 in func_wrapper File "/opt/app/suma/kafka/consumer/runner.py", line 325 in run File "/opt/app/suma/kafka/consumer_cmd.py", line 156 in run_consumer_thread File "/usr/lib/python3.12/threading.py", line 1012 in run File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func File "/opt/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 140 in run File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap ``` **Native stack** ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007f3058686017 rd_kafka_q_serve_rkmessages #1 0x00007f30586480bc rd_kafka_consume_batch_queue #2 0x00007f306d783cda cfunction_call #3 0x00007f306d796f02 _PyObject_Call.localalias #4 0x00007f306d76804a _PyEval_EvalFrameDefault.localalias #5 0x00007f306d783a43 object_vacall #6 0x00007f306d827040 PyObject_CallFunctionObjArgs #7 0x00007f306d757ccc _PyObject_MakeTpCall.localalias #8 0x00007f306d762ec4 _PyEval_EvalFrameDefault.localalias #9 0x00007f306d7b220e _PyObject_VectorcallTstate.lto_priv.14 #10 0x00007f306d7b1d6f method_vectorcall #11 0x00007f306d8a3afd thread_run #12 0x00007f306d89018c pythread_wrapper #13 0x00007f306d3e73c6 start_thread #14 0x00007f306d46c1cc __clone3 ``` Co-authored-by: thomas.kowalski <[email protected]>

## Description This PR ensures we clear the string tables post-fork. This is meant to address a crash happening on (at least) 4.8.0rc5. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x0000000000000000 std::__detail::_Hashtable_base<unsigned long, std::pair<unsigned long const, std::string>, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<false, false, true> >::_M_equals(unsigned long const&, unsigned long, std::__detail::_Hash_node<std::pair<unsigned long const, std::string>, false>*) const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable_policy.h:1804) #1 0x000073afd44ec124 std::__detail::_Hashtable_base<unsigned long, std::pair<unsigned long const, std::string>, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<false, false, true> >::_M_equals(unsigned long const&, unsigned long, std::__detail::_Hash_node<std::pair<unsigned long const, std::string>, false>*) const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable_policy.h:1804) #2 0x000073afd44ec124 std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const [clone .constprop.0] (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:1580) #3 0x000073afd44ece4b std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_node(unsigned long, unsigned long const&, unsigned long) const [clone .isra.0] (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:694) #4 0x0000000000000000 std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:1455) #5 0x0000000000000000 std::unordered_map<unsigned long, std::string, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::string> > >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unordered_map.h:920) #6 0x000073afd44e4e61 std::_Hashtable<unsigned long, std::pair<unsigned long const, std::string>, std::allocator<std::pair<unsigned long const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/hashtable.h:1455) #7 0x000073afd44e4e61 std::unordered_map<unsigned long, std::string, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::string> > >::find(unsigned long const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unordered_map.h:920) #8 0x000073afd44e4e61 StringTable::key(_object*, StringTag) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/strings.cc:58) #9 0x000073afd44e517f Frame::create(EchionSampler&, PyCodeObject*, int) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc:24) #10 0x000073afd44e5436 Frame::get(EchionSampler&, PyCodeObject*, int) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc:214) #11 0x000073afd44e55b1 Frame::read(EchionSampler&, _PyInterpreterFrame*, _PyInterpreterFrame**) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc:153) #12 0x000073afd44e56c0 unwind_frame(EchionSampler&, _object*, FrameStack&, unsigned long) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/stacks.cc:56) #13 0x000073afd44e5859 unwind_python_stack(EchionSampler&, _ts*, FrameStack&) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/stacks.cc:103) #14 0x000073afd44e7bbf ThreadInfo::unwind(EchionSampler&, _ts*) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc:15) #15 0x0000000000000000 __gnu_cxx::__normal_iterator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const*, std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > > >::__normal_iterator(std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const* const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_iterator.h:979) #16 0x0000000000000000 std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::end() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:839) #17 0x0000000000000000 std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::empty() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:1008) #18 0x000073afd44eb2eb __gnu_cxx::__normal_iterator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const*, std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > > >::__normal_iterator(std::unique_ptr<StackInfo, std::default_delete<StackInfo> > const* const&) (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_iterator.h:979) #19 0x000073afd44eb2eb std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::end() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:839) #20 0x000073afd44eb2eb std::vector<std::unique_ptr<StackInfo, std::default_delete<StackInfo> >, std::allocator<std::unique_ptr<StackInfo, std::default_delete<StackInfo> > > >::empty() const (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:1008) #21 0x000073afd44eb2eb ThreadInfo::sample(EchionSampler&, _ts*, long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc:698) #22 0x000073afd44eb789 Datadog::Sampler::sampling_thread(unsigned long) [clone .localalias] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/sampler.cpp:287) #23 0x000073afd44ebad0 call_sampling_thread(void*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/datadog/profiling/stack/src/sampler.cpp:45) #24 0x000073afdf06bac3 start_thread (nptl/nptl/pthread_create.c:442) #25 0x000073afdf0fd8d0 __clone3 ``` Co-authored-by: thomas.kowalski <[email protected]> (cherry picked from commit f1f6901) Co-authored-by: Thomas Kowalski <[email protected]>

## Description This PR fixes crashes (caught by Crash Tracking) caused by IAST trying to access freed memories during interpreter shutdown. `is_text` supposedly safely access objects but can only check whether the addresses make sense, it can't know whether objects have been freed. https://github.com/DataDog/dd-trace-py/blob/160d0fb6ac1a4f8487825d35064acb7489392f7c/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h#L50-L67 When uvloop's `run_until_complete` unwinds (`Py_XDECREF` in frames 57 to 61), custom type objects can be freed by the cyclic GC before all of their instances finish cleanup. An instance can still have a positive refcount, its pointer is non-null and aligned, `ob_type` is non-null, but the type it points to has been deallocated. There's already a `shutting_down` guard in `get_tainted_object_map_from_pyobject`, but `get_tainted_object_map` calls `is_text(obj)` before reaching that. https://github.com/DataDog/dd-trace-py/blob/c641709fa9381957ef114f605b5eadb3fa79ceeb/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp#L140-L147 **Example crashing stack** ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x000061dcfbfffe73 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2137) #1 0x0000000000000000 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #2 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #3 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #4 0x000074d4052a7de1 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #5 0x000074d4052a7de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #6 0x000074d4052a7de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #7 0x000074d4052a7de1 TaintEngineContext::get_tainted_object_map(_object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:141) #8 0x0000000000000000 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #9 0x0000000000000000 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #10 0x0000000000000000 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #11 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #12 0x0000000000000000 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #13 0x0000000000000000 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #14 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #15 0x000074d4052a8a70 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #16 0x000074d4052a8a70 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #17 0x000074d4052a8a70 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #18 0x000074d4052a8a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #19 0x000074d4052a8a70 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #20 0x000074d4052a8a70 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #21 0x000074d4052a8a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #22 0x000074d4052a8a70 pybind11::cpp_function::initialize<pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object)#5}, bool, pybind11::object, pybind11::name, pybind11::scope, pybind11::sibling>(pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object&&)#5}, bool (*)(pybind11::object), pybind11::name const, pybind11::scope&, pybind11::sibling)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .cold] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:400) #23 0x000074d40525fd00 pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:1063) #24 0x000061dcfbff87cf cfunction_call (/usr/src/python/Objects/methodobject.c:551) #25 0x000061dcfbfe95de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #26 0x000061dcfc010357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #27 0x000061dcfc0d8a91 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:93) #28 0x000061dcfbfcd894 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:269) #29 0x000061dcfbfe9ad4 object_vacall (/usr/src/python/Objects/call.c:850) #30 0x000061dcfc046d3e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:961) #31 0x000074d40e7d4d0a WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024) #32 0x000061dcfbfe95de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #33 0x000061dcfc010357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #34 0x000061dcfbfeca86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #35 0x000074d40fdeadc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #36 0x000074d40fdeb5a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #37 0x0000000000000000 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #38 0x000074d3f24978eb __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #39 0x000074d3f24978eb __Pyx_PyObject_FastCallDict (/project/uvloop/loop.c:191552) #40 0x000074d3f2573a69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66873) #41 0x000074d3f257796b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975) #42 0x000074d3f2571e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927) #43 0x000074d3f2573c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335) #44 0x0000000000000000 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #45 0x000074d3f258f311 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #46 0x000074d3f258f311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68) #47 0x000074d3f258c647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:440) #48 0x000074d3f24addb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471) #49 0x000074d3f2515e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876) #50 0x0000000000000000 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #51 0x000074d3f2526cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #52 0x000074d3f2526cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331) #53 0x000061dcfbfea47c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887) #54 0x0000000000000000 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #55 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #56 0x000074d3f252ad60 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #57 0x000074d3f252ad60 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #58 0x000074d3f252ad60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769) #59 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #60 0x000074d3f252c591 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #61 0x000074d3f252c591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33322) #62 0x000061dcfbfe98f8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325) #63 0x000061dcfc010357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #64 0x000061dcfbfeca86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #65 0x000074d40fdeadc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #66 0x000074d40fdeb5a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #67 0x000061dcfbfe95de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #68 0x000061dcfbf6c77a context_run (/usr/src/python/Python/context.c:668) #69 0x000061dcfc05bbeb cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:439) #70 0x000061dcfbf62275 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3227) #71 0x000061dcfc093679 PyEval_EvalCode (/usr/src/python/Python/ceval.c:579) #72 0x000061dcfc0b12bc run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1723) #73 0x000061dcfc0b1234 run_mod (/usr/src/python/Python/pythonrun.c:1744) #74 0x000061dcfc0b0df1 pyrun_file (/usr/src/python/Python/pythonrun.c:1643) #75 0x000061dcfc0b0c37 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433) #76 0x000061dcfc0b0a57 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78) #77 0x000061dcfc0bafb0 Py_RunMain (/usr/src/python/Modules/main.c:713) #78 0x000061dcfc0bab3d Py_BytesMain (/usr/src/python/Modules/main.c:768) #79 0x000074d411000d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58) #80 0x0000000000000000 call_init (csu/libc-start.c:128) #81 0x000074d411000e40 call_init (csu/libc-start.c:128) #82 0x000074d411000e40 __libc_start_main_alias_2 (csu/libc-start.c:379) #83 0x000061dcfc033075 _start ``` Co-authored-by: thomas.kowalski <[email protected]>

## Description This is an attempt to fix a customer-reported crash ([Jira](https://datadoghq.atlassian.net/browse/SCP-1205)). - The customer is using a version that already includes this `ContextVar`-related crash: #17407 - It's likely python/cpython#148928 would fix the crash as well but (1) it would only land in recent versions of CPython -- the customer is using 3.11 -- and (2) I don't have a review/approval yet. This fix addresses the issue in a sort of makeshift way like the CPython upstream fix would. Since we can't bump reference counts directly from CPython, the PR adds a Rust helper which takes care of it. Not great, not terrible. **Note** the bug only happens in versions prior to 3.12 (which made significant changes to the GC and doesn't trigger that HAMT crash anymore), so the fix is gated to them. --- The crash signature is ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00000000006a3e3b #1 0x0000000000690b5b #2 0x00000000004a5e01 #3 0x0000000000690a27 #4 0x0000000000690990 #5 0x0000000000690699 PyContextVar_Set #6 0x0000000000540be3 _PyEval_EvalFrameDefault #7 0x0000000000585937 #8 0x000000000058512e #9 0x000000000054aea5 PyObject_Vectorcall #10 0x000000000053d7a6 _PyEval_EvalFrameDefault #11 0x0000000000585040 #12 0x0000000000541c19 _PyEval_EvalFrameDefault #13 0x0000000000583177 #14 0x0000000000581a14 #15 0x00000000005b418d #16 0x000000000053fb96 _PyEval_EvalFrameDefault #17 0x00000000005afd4e #18 0x00000000005bf334 #19 0x000000000053f6b2 _PyEval_EvalFrameDefault #20 0x0000000000585937 #21 0x000000000058512e #22 0x0000000000570534 PyObject_Call #23 0x0000000000541c19 _PyEval_EvalFrameDefault #24 0x0000000000565f43 _PyFunction_Vectorcall #25 0x00000000005341e9 _PyObject_FastCallDictTstate #26 0x000000000056e0f9 _PyObject_Call_Prepend #27 0x000000000065a02b #28 0x000000000052f59c _PyObject_MakeTpCall #29 0x000000000053d7a6 _PyEval_EvalFrameDefault #30 0x00000000006140f4 #31 0x0000000000613757 PyEval_EvalCode #32 0x00000000006344eb #33 0x0000000000630744 #34 0x0000000000644be5 #35 0x00000000006441f4 _PyRun_SimpleFileObject #36 0x0000000000644007 _PyRun_AnyFileObject #37 0x000000000063ed19 Py_RunMain #38 0x00000000006049cd Py_BytesMain #39 0x00007f3bd11a6d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58) #40 0x0000000000000000 call_init (csu/libc-start.c:128) #41 0x00007f3bd11a6e40 call_init (csu/libc-start.c:128) #42 0x00007f3bd11a6e40 __libc_start_main_alias_2 (csu/libc-start.c:379) #43 0x0000000000604855 _start ``` Co-authored-by: thomas.kowalski <[email protected]>

## Description This fixes a recurring crash where IAST dereferences a Python type object that has already been freed by the cyclic GC. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00006553759b6e73 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2137) #1 0x0000000000000000 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #2 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #3 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #4 0x00007f20d5851de1 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #5 0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #6 0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #7 0x00007f20d5851de1 TaintEngineContext::get_tainted_object_map(_object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:141) #8 0x0000000000000000 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #9 0x0000000000000000 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #10 0x0000000000000000 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #11 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #12 0x0000000000000000 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #13 0x0000000000000000 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #14 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #15 0x00007f20d5852a70 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #16 0x00007f20d5852a70 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #17 0x00007f20d5852a70 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #18 0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #19 0x00007f20d5852a70 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #20 0x00007f20d5852a70 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #21 0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #22 0x00007f20d5852a70 pybind11::cpp_function::initialize<pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object)#5}, bool, pybind11::object, pybind11::name, pybind11::scope, pybind11::sibling>(pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object&&)#5}, bool (*)(pybind11::object), pybind11::name const, pybind11::scope&, pybind11::sibling)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .cold] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:400) #23 0x00007f20d5809d00 pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:1063) #24 0x00006553759af7cf cfunction_call (/usr/src/python/Objects/methodobject.c:551) #25 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #26 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #27 0x0000655375a8fa91 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:93) #28 0x0000655375984894 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:269) #29 0x00006553759a0ad4 object_vacall (/usr/src/python/Objects/call.c:850) #30 0x00006553759fdd3e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:961) #31 0x00007f20ded79d0a WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024) #32 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #33 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #34 0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #35 0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #36 0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #37 0x0000000000000000 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #38 0x00007f20c37988eb __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #39 0x00007f20c37988eb __Pyx_PyObject_FastCallDict (/project/uvloop/loop.c:191552) #40 0x00007f20c3874a69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66873) #41 0x00007f20c387896b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975) #42 0x00007f20c3872e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927) #43 0x00007f20c3874c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335) #44 0x0000000000000000 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #45 0x00007f20c3890311 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #46 0x00007f20c3890311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68) #47 0x00007f20c388d647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:440) #48 0x00007f20c37aedb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471) #49 0x00007f20c3816e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876) #50 0x0000000000000000 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #51 0x00007f20c3827cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #52 0x00007f20c3827cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331) #53 0x00006553759a147c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887) #54 0x0000000000000000 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #55 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #56 0x00007f20c382bd60 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #57 0x00007f20c382bd60 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #58 0x00007f20c382bd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769) #59 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #60 0x00007f20c382d591 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #61 0x00007f20c382d591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33322) #62 0x00006553759a08f8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325) #63 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #64 0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #65 0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #66 0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #67 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #68 0x000065537592377a context_run (/usr/src/python/Python/context.c:668) #69 0x0000655375a12beb cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:439) #70 0x0000655375919275 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3227) #71 0x0000655375a4a679 PyEval_EvalCode (/usr/src/python/Python/ceval.c:579) #72 0x0000655375a682bc run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1723) #73 0x0000655375a68234 run_mod (/usr/src/python/Python/pythonrun.c:1744) #74 0x0000655375a67df1 pyrun_file (/usr/src/python/Python/pythonrun.c:1643) #75 0x0000655375a67c37 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433) #76 0x0000655375a67a57 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78) #77 0x0000655375a71fb0 Py_RunMain (/usr/src/python/Modules/main.c:713) #78 0x0000655375a71b3d Py_BytesMain (/usr/src/python/Modules/main.c:768) #79 0x00007f20e1519d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58) #80 0x0000000000000000 call_init (csu/libc-start.c:128) #81 0x00007f20e1519e40 call_init (csu/libc-start.c:128) #82 0x00007f20e1519e40 __libc_start_main_alias_2 (csu/libc-start.c:379) #83 0x00006553759ea075 _start ``` Root cause (as identified by LLM): - `is_pyobject_tainted` is forwarded to the native `is_in_taint_map(obj, context_id)` - When no IAST request context is active, `context_id` is `None`, so the native binding takes the multi-slot fallback `get_tainted_object_map(obj)`, whose very first step is `is_text(obj)` - `is_text` validates the object pointer (non-null, aligned, non-null `ob_type`) but cannot detect a freed type - `PyUnicode_Check` → `PyObject_TypeCheck` → `PyType_IsSubtype` then walks the dangling type and faults. This happens when the query is invoked from an object finalizer / async-generator teardown run by the GC (`uvloop`'s `run_until_complete` is in the reported stack), i.e. mid-lifecycle rather than at interpreter exit, so the existing `shutting_down` guard (added in #18534, only set from the `atexit` handler) does not cover it. No other code seems to be affected by this specific issue. **Note** the bug affects at least versions 3.19 and 4.8, so I will backport all the way to 4.8 (and we can do 3.19 if AppSec think it's worth doing). ### Changes - `is_pyobject_tainted` now early-returns `False` when `is_iast_request_enabled()` is false, matching `get_tainted_ranges`. - The `get_ranges` public wrapper (`_taint_tracking/__init__.py`): short-circuit to `[]` when the resolved `context_id` is `None`, so no-request callers can't reach the same `is_text`-on-a-freed-type native path. - Added regression tests: no-context query does not consult native; a `__del__`/GC finalizer querying taint with no active context is safe and returns `False`; raw `get_ranges` returns empty without a context.  Co-authored-by: Alberto Vara <[email protected]>

## Description This fixes a recurring crash where IAST dereferences a Python type object that has already been freed by the cyclic GC. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00006553759b6e73 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2137) #1 0x0000000000000000 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #2 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #3 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #4 0x00007f20d5851de1 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #5 0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #6 0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #7 0x00007f20d5851de1 TaintEngineContext::get_tainted_object_map(_object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:141) #8 0x0000000000000000 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #9 0x0000000000000000 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #10 0x0000000000000000 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #11 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #12 0x0000000000000000 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #13 0x0000000000000000 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #14 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #15 0x00007f20d5852a70 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #16 0x00007f20d5852a70 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #17 0x00007f20d5852a70 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #18 0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #19 0x00007f20d5852a70 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #20 0x00007f20d5852a70 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #21 0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #22 0x00007f20d5852a70 pybind11::cpp_function::initialize<pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object)#5}, bool, pybind11::object, pybind11::name, pybind11::scope, pybind11::sibling>(pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object&&)#5}, bool (*)(pybind11::object), pybind11::name const, pybind11::scope&, pybind11::sibling)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .cold] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:400) #23 0x00007f20d5809d00 pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:1063) #24 0x00006553759af7cf cfunction_call (/usr/src/python/Objects/methodobject.c:551) #25 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #26 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #27 0x0000655375a8fa91 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:93) #28 0x0000655375984894 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:269) #29 0x00006553759a0ad4 object_vacall (/usr/src/python/Objects/call.c:850) #30 0x00006553759fdd3e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:961) #31 0x00007f20ded79d0a WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024) #32 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #33 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #34 0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #35 0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #36 0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #37 0x0000000000000000 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #38 0x00007f20c37988eb __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #39 0x00007f20c37988eb __Pyx_PyObject_FastCallDict (/project/uvloop/loop.c:191552) #40 0x00007f20c3874a69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66873) #41 0x00007f20c387896b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975) #42 0x00007f20c3872e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927) #43 0x00007f20c3874c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335) #44 0x0000000000000000 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #45 0x00007f20c3890311 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #46 0x00007f20c3890311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68) #47 0x00007f20c388d647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:440) #48 0x00007f20c37aedb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471) #49 0x00007f20c3816e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876) #50 0x0000000000000000 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #51 0x00007f20c3827cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #52 0x00007f20c3827cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331) #53 0x00006553759a147c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887) #54 0x0000000000000000 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #55 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #56 0x00007f20c382bd60 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #57 0x00007f20c382bd60 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #58 0x00007f20c382bd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769) #59 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #60 0x00007f20c382d591 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #61 0x00007f20c382d591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33322) #62 0x00006553759a08f8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325) #63 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #64 0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #65 0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #66 0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #67 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #68 0x000065537592377a context_run (/usr/src/python/Python/context.c:668) #69 0x0000655375a12beb cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:439) #70 0x0000655375919275 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3227) #71 0x0000655375a4a679 PyEval_EvalCode (/usr/src/python/Python/ceval.c:579) #72 0x0000655375a682bc run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1723) #73 0x0000655375a68234 run_mod (/usr/src/python/Python/pythonrun.c:1744) #74 0x0000655375a67df1 pyrun_file (/usr/src/python/Python/pythonrun.c:1643) #75 0x0000655375a67c37 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433) #76 0x0000655375a67a57 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78) #77 0x0000655375a71fb0 Py_RunMain (/usr/src/python/Modules/main.c:713) #78 0x0000655375a71b3d Py_BytesMain (/usr/src/python/Modules/main.c:768) #79 0x00007f20e1519d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58) #80 0x0000000000000000 call_init (csu/libc-start.c:128) #81 0x00007f20e1519e40 call_init (csu/libc-start.c:128) #82 0x00007f20e1519e40 __libc_start_main_alias_2 (csu/libc-start.c:379) #83 0x00006553759ea075 _start ``` Root cause (as identified by LLM): - `is_pyobject_tainted` is forwarded to the native `is_in_taint_map(obj, context_id)` - When no IAST request context is active, `context_id` is `None`, so the native binding takes the multi-slot fallback `get_tainted_object_map(obj)`, whose very first step is `is_text(obj)` - `is_text` validates the object pointer (non-null, aligned, non-null `ob_type`) but cannot detect a freed type - `PyUnicode_Check` → `PyObject_TypeCheck` → `PyType_IsSubtype` then walks the dangling type and faults. This happens when the query is invoked from an object finalizer / async-generator teardown run by the GC (`uvloop`'s `run_until_complete` is in the reported stack), i.e. mid-lifecycle rather than at interpreter exit, so the existing `shutting_down` guard (added in #18534, only set from the `atexit` handler) does not cover it. No other code seems to be affected by this specific issue. **Note** the bug affects at least versions 3.19 and 4.8, so I will backport all the way to 4.8 (and we can do 3.19 if AppSec think it's worth doing). ### Changes - `is_pyobject_tainted` now early-returns `False` when `is_iast_request_enabled()` is false, matching `get_tainted_ranges`. - The `get_ranges` public wrapper (`_taint_tracking/__init__.py`): short-circuit to `[]` when the resolved `context_id` is `None`, so no-request callers can't reach the same `is_text`-on-a-freed-type native path. - Added regression tests: no-context query does not consult native; a `__del__`/GC finalizer querying taint with no active context is safe and returns `False`; raw `get_ranges` returns empty without a context.  Co-authored-by: Alberto Vara <[email protected]> (cherry picked from commit 3e16e51) Co-authored-by: Thomas Kowalski <[email protected]>

## Description This fixes a recurring crash where IAST dereferences a Python type object that has already been freed by the cyclic GC. ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00006553759b6e73 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2137) #1 0x0000000000000000 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #2 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #3 0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #4 0x00007f20d5851de1 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381) #5 0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66) #6 0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51) #7 0x00007f20d5851de1 TaintEngineContext::get_tainted_object_map(_object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:141) #8 0x0000000000000000 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #9 0x0000000000000000 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #10 0x0000000000000000 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #11 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #12 0x0000000000000000 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #13 0x0000000000000000 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #14 0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #15 0x00007f20d5852a70 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732) #16 0x00007f20d5852a70 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183) #17 0x00007f20d5852a70 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121) #18 0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357) #19 0x00007f20d5852a70 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137) #20 0x00007f20d5852a70 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105) #21 0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430) #22 0x00007f20d5852a70 pybind11::cpp_function::initialize<pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object)#5}, bool, pybind11::object, pybind11::name, pybind11::scope, pybind11::sibling>(pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object&&)#5}, bool (*)(pybind11::object), pybind11::name const, pybind11::scope&, pybind11::sibling)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .cold] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:400) #23 0x00007f20d5809d00 pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:1063) #24 0x00006553759af7cf cfunction_call (/usr/src/python/Objects/methodobject.c:551) #25 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #26 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #27 0x0000655375a8fa91 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:93) #28 0x0000655375984894 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:269) #29 0x00006553759a0ad4 object_vacall (/usr/src/python/Objects/call.c:850) #30 0x00006553759fdd3e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:961) #31 0x00007f20ded79d0a WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024) #32 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #33 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #34 0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #35 0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #36 0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #37 0x0000000000000000 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #38 0x00007f20c37988eb __Pyx_PyObject_Call (/project/uvloop/loop.c:191431) #39 0x00007f20c37988eb __Pyx_PyObject_FastCallDict (/project/uvloop/loop.c:191552) #40 0x00007f20c3874a69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66873) #41 0x00007f20c387896b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975) #42 0x00007f20c3872e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927) #43 0x00007f20c3874c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335) #44 0x0000000000000000 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #45 0x00007f20c3890311 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33) #46 0x00007f20c3890311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68) #47 0x00007f20c388d647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:440) #48 0x00007f20c37aedb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471) #49 0x00007f20c3816e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876) #50 0x0000000000000000 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #51 0x00007f20c3827cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528) #52 0x00007f20c3827cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331) #53 0x00006553759a147c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887) #54 0x0000000000000000 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #55 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #56 0x00007f20c382bd60 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700) #57 0x00007f20c382bd60 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798) #58 0x00007f20c382bd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769) #59 0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #60 0x00007f20c382d591 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797) #61 0x00007f20c382d591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33322) #62 0x00006553759a08f8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325) #63 0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719) #64 0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238) #65 0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869) #66 0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188) #67 0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245) #68 0x000065537592377a context_run (/usr/src/python/Python/context.c:668) #69 0x0000655375a12beb cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:439) #70 0x0000655375919275 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3227) #71 0x0000655375a4a679 PyEval_EvalCode (/usr/src/python/Python/ceval.c:579) #72 0x0000655375a682bc run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1723) #73 0x0000655375a68234 run_mod (/usr/src/python/Python/pythonrun.c:1744) #74 0x0000655375a67df1 pyrun_file (/usr/src/python/Python/pythonrun.c:1643) #75 0x0000655375a67c37 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433) #76 0x0000655375a67a57 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78) #77 0x0000655375a71fb0 Py_RunMain (/usr/src/python/Modules/main.c:713) #78 0x0000655375a71b3d Py_BytesMain (/usr/src/python/Modules/main.c:768) #79 0x00007f20e1519d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58) #80 0x0000000000000000 call_init (csu/libc-start.c:128) #81 0x00007f20e1519e40 call_init (csu/libc-start.c:128) #82 0x00007f20e1519e40 __libc_start_main_alias_2 (csu/libc-start.c:379) #83 0x00006553759ea075 _start ``` Root cause (as identified by LLM): - `is_pyobject_tainted` is forwarded to the native `is_in_taint_map(obj, context_id)` - When no IAST request context is active, `context_id` is `None`, so the native binding takes the multi-slot fallback `get_tainted_object_map(obj)`, whose very first step is `is_text(obj)` - `is_text` validates the object pointer (non-null, aligned, non-null `ob_type`) but cannot detect a freed type - `PyUnicode_Check` → `PyObject_TypeCheck` → `PyType_IsSubtype` then walks the dangling type and faults. This happens when the query is invoked from an object finalizer / async-generator teardown run by the GC (`uvloop`'s `run_until_complete` is in the reported stack), i.e. mid-lifecycle rather than at interpreter exit, so the existing `shutting_down` guard (added in #18534, only set from the `atexit` handler) does not cover it. No other code seems to be affected by this specific issue. **Note** the bug affects at least versions 3.19 and 4.8, so I will backport all the way to 4.8 (and we can do 3.19 if AppSec think it's worth doing). ### Changes - `is_pyobject_tainted` now early-returns `False` when `is_iast_request_enabled()` is false, matching `get_tainted_ranges`. - The `get_ranges` public wrapper (`_taint_tracking/__init__.py`): short-circuit to `[]` when the resolved `context_id` is `None`, so no-request callers can't reach the same `is_text`-on-a-freed-type native path. - Added regression tests: no-context query does not consult native; a `__del__`/GC finalizer querying taint with no active context is safe and returns `False`; raw `get_ranges` returns empty without a context.  Co-authored-by: Alberto Vara <[email protected]>

LotharSee reviewed Jun 30, 2016
View reviewed changes

LeoCavaille added 5 commits June 30, 2016 14:23

new redis contrib integration

cd519af

stop default circle ci redis

a69d4d2

[redis] import properly module and base classes

4576d1f

[redis] add sampler support

fb1ed5e

[redis] unicode compatible resources

3171956

LeoCavaille force-pushed the leo/redis branch from e617d6d to 3171956 Compare June 30, 2016 15:05

[trace/redis] add meta

143ad44

LeoCavaille merged commit 90caef9 into master Jul 1, 2016

LeoCavaille deleted the leo/redis branch July 1, 2016 12:33

labbati added a commit that referenced this pull request Sep 13, 2018

[tmp] #9

cff96c2

yanglong1010 mentioned this pull request Feb 2, 2026

[BUG]: Allocation profiler can not record allocations larger than 512B #16300

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

new redis contrib integration#9

new redis contrib integration#9
LeoCavaille merged 6 commits into
masterfrom
leo/redis

LeoCavaille commented Jun 30, 2016

Uh oh!

LotharSee Jun 30, 2016

Uh oh!

LeoCavaille commented Jun 30, 2016

Uh oh!

LotharSee commented Jun 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LeoCavaille commented Jun 30, 2016

Uh oh!

LotharSee Jun 30, 2016

Choose a reason for hiding this comment

Uh oh!

LeoCavaille commented Jun 30, 2016

Uh oh!

LotharSee commented Jun 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants