[MLOB-1858] feat(llmobs): langchain submits llmobs span events by sabrenner · Pull Request #4923 · DataDog/dd-trace-js

sabrenner · 2024-11-21T20:08:04Z

What does this PR do?

Adds an additional LangChainLLMObs plugin to handle sending LLMObs span events for the LangChain integration. This plugin is registered as part of the existing LangChain plugin, but is not enabled when LLMObs is not enabled.

Context Management

This PR makes some core logic changes to context management for LLMObs plugins.

Initially, spans from LLMObs plugins (only OpenAI) were considered endpoints for that branch of the trace - once the OpenAI SDK was called, the execution flow effectively left userland code, so it was not necessary to preserve context in the plugin. The context store would remain the same, and would resume after the call was finished.

However, it is possible for calls through the LangChain SDK to remain in userland code, as they could have custom steps of their chain wrapped, and we need the proper state in our context store in order to maintain its parentage correctly. Thus, I added some logic to the LLMObsPlugin class to enter with the newly created span on execution start, store the parent span on the execution context, and then restore the parent as the current state when the execution finishes.

There's an important distinction for this: context is only maintained while the traced function executes, not when the promise resolves. This is fine for 99% of cases, where promises can create other promises (i.e., the chain creating its steps, or those steps creating steps), but does not hold for our streamed implementation, specifically for OpenAI. In that case, the parentage only holds while we create the promise for the stream. After its settled, and we consume the stream, our parentage has been restored to what it was before the OpenAI call. This is the general behavior for our APM integration as well (should someone have done tracer.trace prior to LLMObs).

Motivation

Furthering auto-instrumentation for the LLMObs Node.js SDK. This integration will be expanded upon to be in parity with the Python SDK in future PRs.

github-actions · 2024-11-21T20:09:01Z

Overall package size

Self size: 8.5 MB
Deduped: 94.84 MB
No deduping: 95.36 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.3.0 | 29.43 MB | 29.43 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.2.0 | 13.9 MB | 13.91 MB | | @datadog/pprof | 5.4.1 | 9.76 MB | 10.13 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.6.1 | 2.59 MB | 2.73 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 826.22 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | semver | 7.6.3 | 95.82 kB | 95.82 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.1 | 51.46 kB | 51.46 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

pr-commenter · 2024-11-21T21:10:10Z

Benchmarks

Benchmark execution time: 2025-01-09 15:38:21

Comparing candidate commit ebbcd4a in PR branch sabrenner/llmobs-langchain with baseline commit 844d623 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 259 metrics, 7 unstable metrics.

codecov · 2024-11-21T22:16:05Z

Codecov Report

Attention: Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.

Project coverage is 62.20%. Comparing base (c8ab3e4) to head (fbfa26e).
Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
packages/dd-trace/src/llmobs/tagger.js	0.00%	3 Missing ⚠️
packages/dd-trace/src/llmobs/util.js	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4923      +/-   ##
==========================================
- Coverage   66.51%   62.20%   -4.32%     
==========================================
  Files         143      281     +138     
  Lines        4784    13125    +8341     
==========================================
+ Hits         3182     8164    +4982     
- Misses       1602     4961    +3359

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

…r/llmobs-langchain

Kyle-Verhoog · 2024-12-03T15:14:24Z


-  getLLMObsSPanRegisterOptions (ctx) {
+  getLLMObsSpanRegisterOptions (ctx) {
    throw new Error('getLLMObsSPanRegisterOptions must be implemented by the subclass')


Suggested change

throw new Error('getLLMObsSPanRegisterOptions must be implemented by the subclass')

throw new Error('getLLMObsSpanRegisterOptions must be implemented by the subclass')

🤫

tlhunter · 2025-01-14T23:08:35Z

+const BEDROCK_PROVIDER_NAME = 'amazon_bedrock'
+const OPENAI_PROVIDER_NAME = 'openai'
+
+const SUPPORTED_INTEGRATIONS = ['openai']


Suggested change

const SUPPORTED_INTEGRATIONS = ['openai']

const SUPPORTED_INTEGRATIONS = new Set(['openai'])

will be following up on this work, will address these comments in another PR 😄

tlhunter · 2025-01-14T23:08:58Z

+  }
+
+  isLLMIntegrationEnabled (integration) {
+    return SUPPORTED_INTEGRATIONS.includes(integration) && pluginManager?._pluginsByName[integration]?.llmobs?._enabled


Suggested change

return SUPPORTED_INTEGRATIONS.includes(integration) && pluginManager?._pluginsByName[integration]?.llmobs?._enabled

return SUPPORTED_INTEGRATIONS.has(integration) && pluginManager?._pluginsByName[integration]?.llmobs?._enabled

tlhunter

Optional changes

* llmobs langchain plugin * starting test changes * refactor llmobs langchain plugin to use handlers * wip * finish adding most tests * higher timeout ts tests * add missing tests * change lowerbound node version for langchain test to 18 * add cohere test for newer versions * add externals * add error tests * more consistent parentage behavior * review comments

sabrenner added 5 commits November 20, 2024 14:19

llmobs langchain plugin

8ac8fc0

starting test changes

b395cb8

refactor llmobs langchain plugin to use handlers

9726f49

wip

600c209

finish adding most tests

a38fc22

sabrenner added the semver-minor label Nov 21, 2024

sabrenner added 2 commits November 21, 2024 16:03

higher timeout ts tests

4d635e6

add missing tests

ad4824c

datadog-datadog-prod-us1 Bot reviewed Nov 21, 2024

View reviewed changes

Comment thread .github/workflows/llmobs.yml

Comment thread .github/workflows/llmobs.yml

change lowerbound node version for langchain test to 18

1fcbf15

sabrenner added 2 commits November 21, 2024 16:44

add cohere test for newer versions

aec77bb

add externals

fbfa26e

sabrenner and others added 4 commits November 22, 2024 11:13

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

3ef7db3

…r/llmobs-langchain

add error tests

265527d

more consistent parentage behavior

a14f2f2

Merge branch 'master' into sabrenner/llmobs-langchain

ef1a333

sabrenner marked this pull request as ready for review January 8, 2025 15:46

sabrenner requested review from a team as code owners January 8, 2025 15:46

Kyle-Verhoog reviewed Jan 8, 2025

View reviewed changes

review comments

ebbcd4a

Kyle-Verhoog approved these changes Jan 10, 2025

View reviewed changes

tlhunter reviewed Jan 14, 2025

View reviewed changes

tlhunter approved these changes Jan 14, 2025

View reviewed changes

sabrenner merged commit 6523d94 into master Jan 15, 2025

sabrenner deleted the sabrenner/llmobs-langchain branch January 15, 2025 15:03

watson mentioned this pull request Jan 22, 2025

v4.56.0 proposal #5139

Closed

watson mentioned this pull request Jan 22, 2025

v5.32.0 proposal #5140

Merged

	throw new Error('getLLMObsSPanRegisterOptions must be implemented by the subclass')
	throw new Error('getLLMObsSpanRegisterOptions must be implemented by the subclass')

	const SUPPORTED_INTEGRATIONS = ['openai']
	const SUPPORTED_INTEGRATIONS = new Set(['openai'])

	return SUPPORTED_INTEGRATIONS.includes(integration) && pluginManager?._pluginsByName[integration]?.llmobs?._enabled
	return SUPPORTED_INTEGRATIONS.has(integration) && pluginManager?._pluginsByName[integration]?.llmobs?._enabled

Uh oh!

Conversation

sabrenner commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Context Management

Motivation

Uh oh!

github-actions Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

Uh oh!

Uh oh!

pr-commenter Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

codecov Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Kyle-Verhoog Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tlhunter Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

sabrenner Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

tlhunter Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

tlhunter left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sabrenner commented Nov 21, 2024 •

edited

Loading

github-actions Bot commented Nov 21, 2024 •

edited

Loading

pr-commenter Bot commented Nov 21, 2024 •

edited

Loading

codecov Bot commented Nov 21, 2024 •

edited

Loading