Push llm event address by estringana · Pull Request #3664 · DataDog/dd-trace-php

estringana · 2026-02-19T15:18:18Z

Description

This PR enables appsec capabilities when using openai-php client. The implementation push appsec addresses to the waf and then eventually they are reported to the backend.

PHP implementation of the cross-AppSec LLM usage detection RFC for detecting and monitoring OpenAI SDK usage per endpoint.

The instrumentation wraps the OpenAI PHP SDK's request methods and captures LLM-related signals on each call: the model being used, input and output token counts, the request type (completion, chat, embedding, etc) and whether the call succeeded or failed. These are emitted as appsec events.

More info on RFC: API Endpoints: AI usage

Reviewer checklist

Test coverage seems ok.
Appropriate labels assigned.

datadog-official · 2026-02-19T15:27:23Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)
Risky Test
phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:97
testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69a56aef00000000be1ac22698f60f2a
tid: 69a56aef00000000
hexProcessTraceId: be1ac22698f60f2a
hexProcessSpanId: 097e709869854503
processTraceId: 13698474687631593258
processSpanId: 684107993314575619

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
View all

ℹ️ Info

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ba5e2ab | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

codecov-commenter · 2026-02-19T15:33:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.08%. Comparing base (6733f8f) to head (ba5e2ab).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3664      +/-   ##
==========================================
- Coverage   62.20%   62.08%   -0.12%     
==========================================
  Files         141      141              
  Lines       13352    13352              
  Branches     1746     1746              
==========================================
- Hits         8305     8290      -15     
- Misses       4256     4269      +13     
- Partials      791      793       +2

see 4 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6733f8f...ba5e2ab. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2026-02-19T16:40:33Z

Benchmarks [ tracer ]

Benchmark execution time: 2026-03-02 11:56:00

Comparing candidate commit ba5e2ab in PR branch estringana/add-openai-integration with baseline commit 6733f8f in branch master.

Found 3 performance improvements and 27 performance regressions! Performance is the same for 163 metrics, 1 unstable metrics.

scenario:ComposerTelemetryBench/benchTelemetryParsing

🟥 mem_peak [+86.936KB; +86.936KB] or [+2.176%; +2.176%]

scenario:ContextPropagationBench/benchExtractHeaders128Bit

🟥 mem_peak [+98.648KB; +98.648KB] or [+2.469%; +2.469%]

scenario:ContextPropagationBench/benchExtractHeaders64Bit

🟥 mem_peak [+98.648KB; +98.648KB] or [+2.469%; +2.469%]

scenario:ContextPropagationBench/benchExtractTraceContext128Bit

🟥 mem_peak [+98.648KB; +98.648KB] or [+2.469%; +2.469%]

scenario:ContextPropagationBench/benchExtractTraceContext64Bit

🟥 mem_peak [+98.648KB; +98.648KB] or [+2.469%; +2.469%]

scenario:ContextPropagationBench/benchInject128Bit

🟥 mem_peak [+98.656KB; +98.656KB] or [+2.469%; +2.469%]

scenario:ContextPropagationBench/benchInject64Bit

🟥 mem_peak [+98.656KB; +98.656KB] or [+2.469%; +2.469%]

scenario:HookBench/benchHookOverheadInstallHookOnFunction

🟥 mem_peak [+98.520KB; +98.520KB] or [+2.466%; +2.466%]

scenario:HookBench/benchHookOverheadInstallHookOnMethod

🟥 mem_peak [+98.520KB; +98.520KB] or [+2.466%; +2.466%]

scenario:HookBench/benchHookOverheadTraceFunction

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.417%; +2.417%]

scenario:HookBench/benchHookOverheadTraceMethod

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.383%; +2.383%]

scenario:HookBench/benchWithoutHook

🟥 mem_peak [+98.536KB; +98.536KB] or [+2.466%; +2.466%]

scenario:LogsInjectionBench/benchLogsInfoInjection-opcache

🟩 execution_time [-525.508ns; -234.892ns] or [-5.965%; -2.666%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.524%; +2.524%]
🟩 execution_time [-4.997µs; -3.843µs] or [-4.728%; -3.636%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

🟩 execution_time [-6.778µs; -5.322µs] or [-6.083%; -4.777%]

scenario:PDOBench/benchPDOBaseline

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.662%; +2.662%]

scenario:PHPRedisBench/benchRedisBaseline

🟥 mem_peak [+97.776KB; +97.776KB] or [+2.447%; +2.447%]

scenario:SamplingRuleMatchingBench/benchGlobMatching1

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.548%; +2.548%]

scenario:SamplingRuleMatchingBench/benchGlobMatching2

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.548%; +2.548%]

scenario:SamplingRuleMatchingBench/benchGlobMatching3

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.548%; +2.548%]

scenario:SamplingRuleMatchingBench/benchGlobMatching4

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.548%; +2.548%]

scenario:SamplingRuleMatchingBench/benchRegexMatching1

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.547%; +2.547%]

scenario:SamplingRuleMatchingBench/benchRegexMatching2

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.547%; +2.547%]

scenario:SamplingRuleMatchingBench/benchRegexMatching3

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.547%; +2.547%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4

🟥 mem_peak [+101.776KB; +101.776KB] or [+2.547%; +2.547%]

scenario:SpanBench/benchDatadogAPI

🟥 mem_peak [+100.008KB; +100.008KB] or [+2.503%; +2.503%]

scenario:TraceAnnotationsBench/benchTraceAnnotationOverhead

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.378%; +2.378%]

scenario:TraceFlushBench/benchFlushTrace

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.633%; +2.633%]

scenario:TraceSerializationBench/benchSerializeTrace

🟥 mem_peak [+107.360KB; +107.360KB] or [+2.573%; +2.573%]

cataphract

I'd wait for a production rule to be ready (if it ain't already), update the recommended.json files and then also write an integration test. This would validate the correctness of the address and its parameters.

estringana · 2026-02-24T12:28:29Z

I'd wait for a production rule to be ready (if it ain't already), update the recommended.json files and then also write an integration test. This would validate the correctness of the address and its parameters.

I have the rule https://github.com/DataDog/appsec-event-rules/pull/265 but it's not merged yet. Also we would need to mock the openai library http call. Do we have a system for that already on integration?
@cataphract

cataphract · 2026-02-25T22:27:54Z

+    int getPort() {
+        PORT
+    }


same effect could be gotten in groovy by not making PORT private (with no access modifiers, it generates setters/getters)

cataphract · 2026-02-25T22:36:40Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c92b6746e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cataphract

Seems better now, pending test success, but see some new comments

estringana force-pushed the estringana/add-openai-integration branch 2 times, most recently from cff1eed to 1ae39aa Compare February 20, 2026 12:06

estringana marked this pull request as ready for review February 22, 2026 14:57

estringana requested review from a team as code owners February 22, 2026 14:57

estringana force-pushed the estringana/add-openai-integration branch from f363215 to 3fadba0 Compare February 23, 2026 09:55

estringana added 8 commits February 23, 2026 10:55

Push llm event address

0fe616d

Add missing operations

ddb5115

Add tests

c826cf7

Make appsec mock to work in memory

70dac6e

Move push address to prehook

0b41651

Create fake response

123295c

Update snapshots

b5e2c9a

Fix Mysql implementation of mock

6028af2

estringana force-pushed the estringana/add-openai-integration branch from 3fadba0 to 6028af2 Compare February 23, 2026 09:55

Fix addEvent signature

b64379c

estringana requested review from bwoebi and cataphract February 23, 2026 14:16

cataphract reviewed Feb 24, 2026

View reviewed changes

Comment thread src/DDTrace/Integrations/OpenAI/OpenAIIntegration.php

Explain the values of the array

86bde1a

Add integration tests for llm events

905ba6a

estringana requested a review from a team as a code owner February 25, 2026 16:57

estringana added 2 commits February 25, 2026 21:28

Fix port

0eb5a02

Make it compatible with php 7.0

3c92b67

cataphract requested changes Feb 25, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Feb 25, 2026

View reviewed changes

Comment thread src/DDTrace/Integrations/OpenAI/OpenAIIntegration.php Outdated

estringana added 2 commits February 27, 2026 12:33

Amend comments from PR

193aafa

Avoid creating not required span

277957f

estringana force-pushed the estringana/add-openai-integration branch from e713572 to 277957f Compare February 27, 2026 16:21

Fix tests

e618728

cataphract approved these changes Feb 27, 2026

View reviewed changes

Comment thread src/DDTrace/Integrations/OpenAI/OpenAIIntegration.php Outdated

Comment thread src/DDTrace/Integrations/OpenAI/OpenAIIntegration.php Outdated

Comment thread ...c/tests/integration/src/test/groovy/com/datadog/appsec/php/integration/LlmEventsTests.groovy

estringana added 2 commits March 2, 2026 10:52

Amend pr comments

dc468ff

Merge branch 'master' into estringana/add-openai-integration

ba5e2ab

estringana merged commit c9f888d into master Mar 2, 2026
2060 of 2063 checks passed

estringana deleted the estringana/add-openai-integration branch March 2, 2026 12:35

github-actions Bot added this to the 1.17.0 milestone Mar 2, 2026

Conversation

estringana commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer checklist

Uh oh!

datadog-official Bot commented Feb 19, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

codecov-commenter commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter Bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [ tracer ]

scenario:ComposerTelemetryBench/benchTelemetryParsing

scenario:ContextPropagationBench/benchExtractHeaders128Bit

scenario:ContextPropagationBench/benchExtractHeaders64Bit

scenario:ContextPropagationBench/benchExtractTraceContext128Bit

scenario:ContextPropagationBench/benchExtractTraceContext64Bit

scenario:ContextPropagationBench/benchInject128Bit

scenario:ContextPropagationBench/benchInject64Bit

scenario:HookBench/benchHookOverheadInstallHookOnFunction

scenario:HookBench/benchHookOverheadInstallHookOnMethod

scenario:HookBench/benchHookOverheadTraceFunction

scenario:HookBench/benchHookOverheadTraceMethod

scenario:HookBench/benchWithoutHook

scenario:LogsInjectionBench/benchLogsInfoInjection-opcache

scenario:MessagePackSerializationBench/benchMessagePackSerialization

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

scenario:PDOBench/benchPDOBaseline

scenario:PHPRedisBench/benchRedisBaseline

scenario:SamplingRuleMatchingBench/benchGlobMatching1

scenario:SamplingRuleMatchingBench/benchGlobMatching2

scenario:SamplingRuleMatchingBench/benchGlobMatching3

scenario:SamplingRuleMatchingBench/benchGlobMatching4

scenario:SamplingRuleMatchingBench/benchRegexMatching1

scenario:SamplingRuleMatchingBench/benchRegexMatching2

scenario:SamplingRuleMatchingBench/benchRegexMatching3

scenario:SamplingRuleMatchingBench/benchRegexMatching4

scenario:SpanBench/benchDatadogAPI

scenario:TraceAnnotationsBench/benchTraceAnnotationOverhead

scenario:TraceFlushBench/benchFlushTrace

scenario:TraceSerializationBench/benchSerializeTrace

Uh oh!

cataphract left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

estringana commented Feb 24, 2026

Uh oh!

Uh oh!

cataphract Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cataphract commented Feb 25, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cataphract left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

estringana commented Feb 19, 2026 •

edited

Loading

datadog-official Bot commented Feb 19, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

codecov-commenter commented Feb 19, 2026 •

edited

Loading

pr-commenter Bot commented Feb 19, 2026 •

edited

Loading