Bug description
Hello, we are using a PHP distributed task manager that uses pcntl_fork()to create child processes to execute tasks.
pcntl_fork() creates an exact copy of a running process, the only difference being the process's PID and PPID.
From my investigation, it looks like the seed is initialized once when the root process is started using GENERATE_SEED() which uses the PID of the process and a random number.
The problem is that any new child processes will not go through the initialization process so it will have the exact same seed as the root process and so it will generate the same sequence of random numbers the root process, this means multiple processes will submit spans with the same span_ids.
When 2 spans with the same span_id are sent to Datadog, the UI appears to show the one that was sent earliest. For example, span_id 684198073586290847 should have been submitted with "myRootSpan" as the resource.
[2020-11-20T06:15:43+00:00] [ddtrace] [debug] - Encoding span '684198073586290847' op: 'myRootSpan' serv: 'cli' res: 'myRootSpan' type ''
[2020-11-20T06:15:43+00:00] [ddtrace] [debug] - Tags for span 684198073586290847 'tag:chars_count' are: env:5,system.pid:4,pid:4,error.msg:31,error.type:9,error.stack:4414
But in Datadog shows a different span https://app.datadoghq.com/apm/trace/684198073586290847?colorBy=service&env=local&graphType=flamegraph&shouldShowLegend=true&sort=time&spanID=684198073586290847&spanViewType=metadata (interestingly, sometimes the APM live trace view will show the one of the spans in the list but then clicking on it will show the other span)
As a side note, we've also disabled sending traces via thread since it also appeared to not work with the forked child processes: DD_TRACE_BETA_SEND_TRACES_VIA_THREAD: "false"
PHP version
PHP 7.4.12 (cli) (built: Nov 5 2020 20:24:10) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.12, Copyright (c), by Zend Technologies
with Xdebug v2.9.8, Copyright (c) 2002-2020, by Derick Rethans
with ddtrace v0.51.0, Copyright Datadog, by Datadog
Installed extensions
[PHP Modules]
Core
ctype
curl
date
ddtrace
dom
fileinfo
filter
ftp
hash
iconv
json
libxml
mbstring
mysqlnd
openssl
pcntl
pcre
PDO
pdo_pgsql
pdo_sqlite
Phar
posix
readline
redis
Reflection
session
SimpleXML
sodium
SPL
sqlite3
standard
tokenizer
xdebug
xml
xmlreader
xmlwriter
Zend OPcache
zlib
[Zend Modules]
Xdebug
Zend OPcache
ddtrace
OS info
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
Diagnostics and configuration
Output of phpinfo() (ddtrace >= 0.47.0)
ddtrace
Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020
Datadog tracing support => enabled
Version => 0.51.0
DATADOG TRACER CONFIGURATION => {"date":"2020-11-20T15:55:26Z","os_name":"Linux 00d0f37e0a22 5.4.39-linuxkit #1 SMP Fri May 8 23:03:06 UTC 2020 x86_64","os_version":"5.4.39-linuxkit","version":"0.51.0","lang":"php","lang_version":"7.4.12","env":"local","enabled":true,"service":"datadog-example","enabled_cli":true,"agent_url":"http://datadog-agent:8126","debug":true,"analytics_enabled":false,"sample_rate":1.000000,"sampling_rules":null,"tags":null,"service_mapping":null,"distributed_tracing_enabled":true,"priority_sampling_enabled":true,"dd_version":null,"architecture":"x86_64","sapi":"cli","ddtrace.request_init_hook":"/usr/local/lib/php/datadog_trace/bridge/dd_wrap_autoloader.php","open_basedir_configured":false,"uri_fragment_regex":null,"uri_mapping_incoming":null,"uri_mapping_outgoing":null,"auto_flush_enabled":true,"generate_root_span":false,"http_client_split_by_domain":false,"measure_compile_time":true,"report_hostname_on_root_span":false,"traced_internal_functions":null,"auto_prepend_file_configured":false,"integrations_disabled":null,"enabled_from_env":true,"opcache.file_cache":null}
Diagnostics
Diagnostic checks => passed
Directive => Local Value => Master Value
ddtrace.disable => Off => Off
ddtrace.request_init_hook => /usr/local/lib/php/datadog_trace/bridge/dd_wrap_autoloader.php => /usr/local/lib/php/datadog_trace/bridge/dd_wrap_autoloader.php
Bug description
Hello, we are using a PHP distributed task manager that uses
pcntl_fork()to create child processes to execute tasks.pcntl_fork()creates an exact copy of a running process, the only difference being the process's PID and PPID.From my investigation, it looks like the seed is initialized once when the root process is started using GENERATE_SEED() which uses the PID of the process and a random number.
The problem is that any new child processes will not go through the initialization process so it will have the exact same seed as the root process and so it will generate the same sequence of random numbers the root process, this means multiple processes will submit spans with the same span_ids.
When 2 spans with the same span_id are sent to Datadog, the UI appears to show the one that was sent earliest. For example, span_id 684198073586290847 should have been submitted with "myRootSpan" as the resource.
But in Datadog shows a different span https://app.datadoghq.com/apm/trace/684198073586290847?colorBy=service&env=local&graphType=flamegraph&shouldShowLegend=true&sort=time&spanID=684198073586290847&spanViewType=metadata (interestingly, sometimes the APM live trace view will show the one of the spans in the list but then clicking on it will show the other span)
As a side note, we've also disabled sending traces via thread since it also appeared to not work with the forked child processes:
DD_TRACE_BETA_SEND_TRACES_VIA_THREAD: "false"PHP version
Installed extensions
OS info
Diagnostics and configuration
Output of phpinfo() (ddtrace >= 0.47.0)