feat(claude-agent-sdk): add instrumentation for @anthropic-ai/claude-agent-sdk#7974
feat(claude-agent-sdk): add instrumentation for @anthropic-ai/claude-agent-sdk#7974mr-lee wants to merge 52 commits into
Conversation
Overall package sizeSelf size: 6.55 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #7974 +/- ##
==========================================
- Coverage 93.64% 92.90% -0.74%
==========================================
Files 898 904 +6
Lines 52365 52720 +355
Branches 12324 12415 +91
==========================================
- Hits 49038 48982 -56
- Misses 3327 3738 +411
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: b421293 | Docs | Datadog PR Page | Give us feedback! |
33b112c to
68c716f
Compare
52e6727 to
e497246
Compare
e497246 to
045d54a
Compare
BenchmarksBenchmark execution time: 2026-07-02 20:41:39 Comparing candidate commit b421293 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2255 metrics, 31 unstable metrics.
|
8ed7a6b to
04accb7
Compare
04accb7 to
e98f416
Compare
sabrenner
left a comment
sabrenner
left a comment
There was a problem hiding this comment.
high-level comments, two main callouts:
- we should still be able to use orchestrion, even when doing argument mutations. orchestrion is great for tracing the top-level things like
querydirectly with lifecycle events, but otherwise we can intercept arguments and patch them there. ideally, we could also patch the code where hooks are called and do a tracing channel on that function. - we should be able to actually test against the library vs using hooks & mocking calls to the library. i was able to
requireit in a CJS file, and even if it doesn't work here, we can write some tests in ourintegration-testsuite to run ESM files. - looks like there are a couple of
debuggerchanges in this PR that are unrelated and are probs just from a bad rebase. for some reason i couldnt comment on them directly
i think these points will change a decent chunk of the code, so i'll do a further in-depth review after!
975e1b2 to
9fb8c74
Compare
|
|
||
| const assert = require('node:assert') | ||
| const { describe, before, after, it } = require('mocha') | ||
| const { tracingChannel, channel } = require('dc-polyfill') |
There was a problem hiding this comment.
Tried setting ANTHROPIC_BASE_URL=http://127.0.0.1:19126 and running query() — the subprocess spawns and returns real messages, but zero HTTP requests hit the proxy. The CLI binary (cli.js) reads ANTHROPIC_BASE_URL but has a host validation (function cH) that checks if the host is api.anthropic.com. Non-standard hosts get filtered. How did you get it working locally? Were you using the bridge entry point or the default query() path?
There was a problem hiding this comment.
you can also tried setting CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST: true and see if that helps, that's what i also had to include locally as otherwise i ran into what you're describing
|
|
||
| const assert = require('node:assert') | ||
| const { describe, before, after, it } = require('mocha') | ||
| const { tracingChannel, channel } = require('dc-polyfill') |
There was a problem hiding this comment.
That worked! CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST=true makes the subprocess send HTTP through the configured ANTHROPIC_BASE_URL. Proxy captured POST /v1/messages?beta=true requests.
One issue for cassette recording: the CLI uses Vault OIDC auth (not raw ANTHROPIC_API_KEY), so the requests 401 when proxied. Would be great to pair on getting a successful recording for the cassette — do you have a working cassette from your local testing?
9fb8c74 to
f6d9ca1
Compare
| - VCR_PROVIDER_MAP=claude-agent-sdk=https://api.anthropic.com | ||
| - VCR_JSON_BODY_NORMALIZERS=metadata.user_id,output_config.effort | ||
| - >- | ||
| VCR_BODY_REGEX_NORMALIZERS=agentId: [a-f0-9]+,SendMessage with to: '[a-f0-9]+',<usage>[\s\S]*?</usage>,"events":\[\{"event_type":"ClaudeCodeInternalEvent"[^\]]*\],cc_version=[^;]+,cch=[^;]+,<system-reminder>[\s\S]*?</system-reminder>,<env>[\s\S]*?</env>,(?<="input":\{"description":")[^"]*,(?:You are an agent for Claude Code|Write the title|Report the result concisely\.|Async agent launched successfully)(?:\\.|[^"\\])* |
There was a problem hiding this comment.
lots of weird normalization had to happen between the linux GHA runners, local macos development, etc., but these normalizations on things that are non-query specific seem to do the trick. this might be finicky for versions going forward, but work here
|
@codex review |
|
was there a strong reason to swap away from hook-based instrumentation? |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff1240fee8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
we would not get LLM spans while using hooks, and parentage was still very finicky. Parsing the event stream seems to be more stable to me |
This makes sense for the LLM spans - but also - the hooks are a direct ancestor to the event stream, in a way. IMO it's a more stable surface to instrument to get reliable span creation than the event stream in isolation. Perhaps consider a joint approach? |
i had thought of that, but parentage gets really tricky and i'd rather not be storing contexts and re-activating them when we can "reconstruct" the lifecycle events with tracing channels and make it a bit cleaner. this is sorta similar to our approach with tracing the python version as well. |
…g/dd-trace-js into feat/claude-agent-sdk-integration
Wait - does this need to be stored in memory? My argument is to have this be the trigger (e.g. the "Stop" hook) because that's the cleanest injection point. As and when progressive tracing (i.e. be able to publish incomplete traces) becomes a requirement - additional hook points can be included. |
|
superseded by #9202 |
3 participants
What does this PR do?
Adds support for tracing top-level query, tool, LLM, and subagent calls from the Claude Agent SDK.
This is accomplished by parsing the event stream into tracing-lifecycle events after the fact.
Motivation
Provide support for the Claude Agent SDK package.