Skip to content

Split the tracer parts off from ext directory#3912

Merged
bwoebi merged 2 commits into
masterfrom
bob/common-ext
Jun 2, 2026
Merged

Split the tracer parts off from ext directory#3912
bwoebi merged 2 commits into
masterfrom
bob/common-ext

Conversation

@bwoebi

@bwoebi bwoebi commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Move all the things.
Drop a few obsolete things.
Change nearly no code.

  • git data gets collected in ext/, but git object is collected in tracer/
  • the sidecar is now always spawned, unconditionally

@bwoebi
bwoebi requested review from a team as code owners May 23, 2026 00:10
@bwoebi
bwoebi requested review from LobeTia and removed request for a team May 23, 2026 00:10
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented May 23, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 7 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | ASAN test_c with multiple observers: [8.4]   View in Datadog   GitLab

🔧 Fix in code (Fix with Cursor). Failed test: 'Read telemetry via composer' in tmp/build_extension/tests/ext/telemetry/composer.phpt with 29 out of 664 tests passing.

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [8.1]   View in Datadog   GitLab

🔧 Fix in code (Fix with Cursor). Assertion failed in zend_hash.c:1012: `(zend_gc_refcount(&(ht)->gc) == 1) || ((ht)->u.flags & (1<<6))`

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [8.4]   View in Datadog   GitLab

🔧 Fix in code (Fix with Cursor). Assertion failed in zend_hash.c:1098: (zend_gc_refcount(&(ht)->gc) == 1) || ((ht)->u.flags & (1<<6)) while processing debugger_log_probe_process_tags.

View all 7 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 73.53%
Overall Coverage: 54.12% (-0.04%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b0051dc | Docs | Datadog PR Page | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

#include "compatibility.h"

P1 Badge Fix weakrefs include path after move to tracer/

After moving weakrefs.c to tracer/, it still includes "compatibility.h" as if it were in ext/. In this commit config.m4 also drops PHP_ADD_INCLUDE([$ext_srcdir/ext]), so for PHP 8.0/8.1 builds (where weakrefs.c is compiled) the header is no longer resolvable and the build fails. Use #include <ext/compatibility.h> (or restore/include the ext include path) to keep those builds working.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread config.m4 Outdated
@bwoebi
bwoebi force-pushed the bob/common-ext branch 6 times, most recently from 1bc2038 to 4a29a16 Compare May 23, 2026 13:28
@pr-commenter

pr-commenter Bot commented May 23, 2026

Copy link
Copy Markdown

Benchmarks [ profiler ]

Benchmark execution time: 2026-06-02 15:20:16

Comparing candidate commit b0051dc in PR branch bob/common-ext with baseline commit cd4e9ba in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 27 metrics, 8 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:php-profiler-timeline-memory-with-profiler-and-timeline

  • 🟩 cpu_system_time [-35.803ms; -8.998ms] or [-8.756%; -2.201%]

@bwoebi
bwoebi force-pushed the bob/common-ext branch 5 times, most recently from bdedead to 0dbe6ac Compare May 25, 2026 13:20
@pr-commenter

pr-commenter Bot commented May 25, 2026

Copy link
Copy Markdown

Benchmarks [ appsec ]

Benchmark execution time: 2026-06-02 15:42:08

Comparing candidate commit b0051dc in PR branch bob/common-ext with baseline commit cd4e9ba in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@pr-commenter

pr-commenter Bot commented May 25, 2026

Copy link
Copy Markdown

Benchmarks [ tracer ]

Benchmark execution time: 2026-06-02 16:19:04

Comparing candidate commit b0051dc in PR branch bob/common-ext with baseline commit cd4e9ba in branch master.

Found 1 performance improvements and 6 performance regressions! Performance is the same for 187 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:MessagePackSerializationBench/benchMessagePackSerialization

  • 🟥 execution_time [+2.998µs; +5.182µs] or [+2.962%; +5.120%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

  • 🟥 execution_time [+2.408µs; +3.952µs] or [+2.267%; +3.719%]

scenario:SamplingRuleMatchingBench/benchRegexMatching1

  • 🟥 execution_time [+38.917ns; +126.683ns] or [+2.651%; +8.628%]

scenario:SamplingRuleMatchingBench/benchRegexMatching2

  • 🟥 execution_time [+60.053ns; +143.347ns] or [+4.109%; +9.808%]

scenario:SamplingRuleMatchingBench/benchRegexMatching3

  • 🟥 execution_time [+33.133ns; +130.867ns] or [+2.224%; +8.785%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4

  • 🟥 execution_time [+35.730ns; +119.270ns] or [+2.420%; +8.079%]

scenario:TraceSerializationBench/benchSerializeTrace

  • 🟩 execution_time [-23.988µs; -8.512µs] or [-5.658%; -2.008%]

@bwoebi
bwoebi force-pushed the bob/common-ext branch 6 times, most recently from 99f2b5b to fdffb6e Compare May 28, 2026 18:26
@bwoebi
bwoebi requested a review from a team as a code owner May 29, 2026 19:39
@bwoebi
bwoebi requested review from dd-oleksii and leoromanovsky and removed request for a team May 29, 2026 19:39
@bwoebi
bwoebi force-pushed the bob/common-ext branch 2 times, most recently from 6b80a13 to c94e0c3 Compare June 1, 2026 15:08
@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

If you need to update snapshots, please refer to CONTRIBUTING.md

@bwoebi
bwoebi force-pushed the bob/common-ext branch 4 times, most recently from 437f6e6 to ce6fc63 Compare June 1, 2026 23:30

@Leiyks Leiyks left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 👌

Comment thread ext/sidecar.c
Comment thread tracer/tracer_api.h Outdated
#endif
extern ddog_LiveDebuggerSetup ddtrace_live_debugger_setup;

ddtrace_span_data *ddtrace_active_span(void);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shoudn't this one be in a #ifdef DDTRACE guard as well ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, let's fully ifdef this, gives compile warnings rather than runtime link failures.

@bwoebi
bwoebi merged commit b026bfc into master Jun 2, 2026
2117 of 2135 checks passed
@bwoebi
bwoebi deleted the bob/common-ext branch June 2, 2026 15:50
@github-actions github-actions Bot added this to the 1.21.0 milestone Jun 2, 2026
Leiyks added a commit that referenced this pull request Jul 16, 2026
…#4035)

* Consolidate Claude config: shared vs personal split + project.md

Add a portable, repo-side convention so each developer can keep personal
Claude config alongside the shared team config without committing it, and
fold project-level knowledge into the committed common config.

- .claude/.gitignore: ignore per-developer paths (personal/, commands/,
  scripts/, skills/omc-reference/, settings.local.json, worktrees/, .omc/)
  in place; keep personal/README.md committed. Previously this relied on
  each developer's global gitignore, which isn't portable and silently
  hid new shared files.
- .claude/personal/: git-ignored drop zone for per-developer config;
  CLAUDE.md optionally imports personal/CLAUDE.md if present.
- .claude/project.md: project knowledge (overview, layout, architecture,
  INI, conventions) in the terse house style, cross-referencing the
  existing build/CI/debug docs instead of duplicating them. Reflects the
  ext/ -> tracer/ split (#3912); versions reference VERSION / Cargo.toml
  / rust-toolchain.toml rather than being pinned inline.
- CLAUDE.md: link general.md, project.md, ci/index.md and import personal.

* Auto-load common config via @import (general.md, project.md)

Plain markdown links in CLAUDE.md are not injected into context by Claude
Code — only @-imports are. Import general.md and project.md so the shared
operating rules and project knowledge are always loaded alongside each
developer's personal config. ci/index.md stays a link (large on-demand CI
reference).

* Restructure shared config: scripts/ and debugging/ folders; shareable commands

Addresses PR review feedback:

- .claude/.gitignore: commands/ is now personal-by-default via `/commands/*`
  (instead of ignoring the whole dir), so the team CAN commit shared commands
  by un-ignoring them while personal ones stay ignored. Claude Code discovers
  commands by filesystem regardless of git status, so both coexist.
- Scripts: shared helper scripts moved into .claude/scripts/ (dd_php_release_url,
  find_map_region.py, parse_ucontext.py); personal scripts go in
  .claude/scripts/local/ (git-ignored). crash-analysis skill refs updated.
- Debugging docs grouped under .claude/debugging/ with an index.md (like ci/):
  gdb.md, appsec-integration.md (was debugging.md), system-tests.md (was
  debugging-system-tests.md). All inbound/outbound links updated.

Note: ci/ keeps its own co-located tooling (dockerh, check-ci, ...) since those
are referenced throughout the ci/ docs as a self-contained module.

* Keep personal skill ignores out of shared gitignore

Personal skills live in .claude/skills/ alongside shared ones (Claude Code
discovers by filesystem, ignoring git status). Their dir name is the skill
name, so there's no generic pattern; document that each dev ignores their own
personal skill dirs in .git/info/exclude (per-clone) instead of publishing
personal skill names into the committed shared config.

* Drop in-repo personal mechanism; personal config lives in ~/.claude

Personal Claude config now lives entirely in ~/.claude (global), so the repo's
.claude/ is purely shared team config:

- Remove .claude/personal/ (README) and the @.claude/personal/CLAUDE.md import
  from CLAUDE.md.
- Slim .claude/.gitignore to just local/runtime artifacts (settings.local.json,
  worktrees/, .omc/); document that personal config goes in ~/.claude.

* Remove .claude/.gitignore; personal tool artifacts ignored globally

The repo .claude/ carries no ignore file now — it's purely shared config.
Local/runtime artifacts are handled outside the shared config: settings.local.json
via the repo root .gitignore, and per-developer tooling output (OMC .omc/, Claude
Code agent worktrees/checkpoints/etc.) via each dev's own global ~/.gitignore.

* Expand project.md into a project/ code-guide folder

Turn the single project.md into a ci/-style folder: a concise always-imported
index.md (overview, layout, subsystem map, architecture summary, config,
conventions, pointers) plus one terse orientation file per subsystem —
ext, tracer, userland (src/), components (+components-rs/ZAI), sidecar, appsec,
profiling. Each covers what it is / key files & dirs / how it fits / gotchas,
cross-linked, with build/CI/debug content linked out (not duplicated).

Produced via an analyze→write→review pass over the actual source: per-subsystem
code analysis, drafting, then a source-accuracy + house-style review. Versions
are referenced by source-of-truth (VERSION, Cargo.toml, rust-toolchain.toml);
no hardcoded versions or source line numbers. CLAUDE.md now imports
project/index.md.

* project/: add data-flow to tracer/ext/sidecar; link architecture & CONTRIBUTING

Targeted depth-increase (on-demand area files only; index stays lean):
- tracer.md: ordered span→upload data-flow (RINIT → headers → span create →
  uhook → close → sampling → serializer → auto_flush → sidecar/coms → agent).
- ext.md: fix the incorrect "MSHUTDOWN tears down in reverse" claim — it isn't
  (sidecar shuts down late, after config is freed); tighten MINIT/RINIT/RSHUTDOWN
  ordering (logging-first, zend_extension registration, dd_rinit_once).
- sidecar.md: submit→upload flow (serialize → send_traces_to_sidecar → IPC →
  TraceFlusher batches ~5s/~1MB → agent upload; fork preserves session ID).
- index.md: link architecture.md (background sender / components / version code)
  and CONTRIBUTING.md (setup/testing) from Pointers.

Produced via the OMC analyze→write→review pipeline over live source; verified
accurate (flush thresholds, ordering, function anchors). No line numbers.

* project/: address Codex review — fix 4 doc inaccuracies

All four verified against live source:
- appsec.md: C++ helper client/runner/engine/service are .cpp/.hpp files, not
  dirs (only network/, remote_config/, subscriber/ are dirs).
- appsec.md: the helper is enabled via the sidecar — ext/sidecar.c calls the
  appsec module's dd_appsec_maybe_enable_helper -> ddog_sidecar_enable_appsec
  (resolved Rust/C++ path) before ddappsec's first RINIT — not spawned by
  ddappsec MINIT/RINIT.
- tracer.md + index.md: the sidecar sender is only the default on PHP 8.3+ /
  Windows (DD_SIDECAR_TRACE_SENDER_DEFAULT); on PHP 7.0-8.2 the in-process
  coms.c sender is the default.
- components.md: drop container_id from the component list — it's a stale
  build-artifact dir with no source; container ID is in the Rust bridge
  (ddtrace_get_container_id).

* project/: address review — drop FFI-surface note; fix cbindgen regen trigger

Per @bwoebi: the "keep FFI surface small" line isn't actionable (dropped), and cbindgen headers regenerate when any components-rs/libdatadog source changes, not just lib.rs.

* Update .claude/project/tracer.md

Co-authored-by: Bob Weinand <[email protected]>

* feat(ci): add --jobs named-subset filter to check-ci

Add an opt-in --jobs "pat1,pat2" option (comma-separated, case-insensitive
substring patterns) to the check-ci monitor. When set, blocking-wait
completion and failure/timeout accounting consider only jobs whose name
matches any pattern, so the monitor finishes as soon as the matched subset
is terminal instead of waiting for the whole pipeline. --list-jobs shows
only matched jobs. Applies to both GitLab jobs and GitHub Actions jobs.
Default (no --jobs) keeps whole-pipeline monitoring unchanged.

* docs(claude): add CI flakiness & debugging gotchas to shared config

---------

Co-authored-by: Bob Weinand <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants