Bug Fixes

• bootstrap: keep yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.

• Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted text containing literal IAST evidence marker delimiters.

• Code Security (IAST): Resolve a weak-hash false positive reported on ddtrace's own code when IAST and the live debugger (Symbol Database) are both enabled.

• LLM Observability: This fix resolves an issue where annotating a span with a metric key containing a dot (e.g. anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.

• LLM Observability: This fixes an issue where cache write input tokens were not captured by the OpenAI integration when using newer models that include cache write token charges.

Bug Fixes

• bootstrap: keep yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.

• IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted data. A concurrent or still-open request holding a tainted value could leak its taint into the current request's checks. Queries are now scoped to the calling request's slot.

Bug Fixes

• bootstrap: keep yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.

• IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted data. A concurrent or still-open request holding a tainted value could leak its taint into the current request's checks. Queries are now scoped to the calling request's slot.

• Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted text containing literal IAST evidence marker delimiters.

• Code Security (IAST): Resolve a weak-hash false positive reported on ddtrace's own code when IAST and the live debugger (Symbol Database) are both enabled.

• profiling: crash fix for stack sampler in case where another component installs its own SIGSEGV/SIGBUS handler that the profiler cannot wrap (e.g. CUDA, PyTorch, etc.). The sampler upgrades to the faster fault-recovery copy if it still owns both fault handlers afterwards. Otherwise, it permanently falls back to the syscall-based copy.

Bug Fixes

• bootstrap: keep yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.

• IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted data. A concurrent or still-open request holding a tainted value could leak its taint into the current request's checks. Queries are now scoped to the calling request's slot.

• Code Security (IAST): Resolve a weak-hash false positive reported on ddtrace's own code when IAST and the live debugger (Symbol Database) are both enabled.

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.

Upgrade Notes

• flask: Requests served under a non-empty WSGI SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.

• flask: API endpoint discovery now reports every HTTP method the framework serves — including Werkzeug's auto-added HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].

• LLM Observability: when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.

New Features

• tracing: collect the x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.

• Database Monitoring (DBM) propagation supports dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.

• ai_guard: add AI Guard evaluation support to the Anthropic SDK Messages instrumentation. Both non-streaming and streaming requests and non-streaming responses are evaluated through the configured AI Guard client (covering Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.

• aiokafka: Adds kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.

• ASM: This introduces the _dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.

• ASM: Extends _dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).

• LLM Observability: The claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.

• aws_durable_execution_sdk_python: Adds distributed tracing for durable workflows across suspend/resume cycles, so a workflow that pauses and resumes in a later invocation appears as a single connected trace. See aws_durable_execution_sdk_python for details and opt-out configuration.

• AAP: API endpoint discovery now covers Flask sub-applications mounted via werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.

• google_cloud_pubsub: This introduces Data Streams Monitoring (DSM) context propagation for the Google Cloud Pub/Sub integration. Producer publish operations inject the DSM pathway context into message attributes, and subscriber callbacks extract it and record a consume checkpoint. To enable, set DD_DATA_STREAMS_ENABLED=true.

• LLM Observability: automatically tags spans and experiments with git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.

• LLM Observability: LLM span data can now be exported to the Datadog Agent over the APM trace, and is reliably delivered even when the APM trace is not sampled. This has no effect on APM sampling decisions or billing.

• LLM Observability: experiment evaluators and summary evaluators can now return a MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.

• LLM Observability: Add LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.

• LLM Observability: task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.

• LLM Observability: Adds support for DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.

• profiling: The fast stack profiler optimisation is now enabled by default.

• Profiling: The heap profiler can now track allocations made through PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.

• ray: Added DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.

• This change adds the ability to override a manually installed tracer for Single Step Instrumentation. When this is set, all ddtrace packages will be preferred over the user installed packages. For example, wrapt, bytecode, and others. To ensure the injected library takes precedence, DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.

• CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).

  Example:

  from pydantic_evals.evaluators import EqualsExpected import logging

  from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter

  handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)

  while True:
  job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)

  handler.close()

Bug Fixes

• AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, c...

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Upgrade Notes

• LLM Observability
  • when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.
• ASM
  • This upgrades libddwaf to 2.0.0.

New Features

• AI Guard
  • This introduces the DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.
  • add AI Guard support for LangChain 1.0+ agents built with create_agent.
  • Added opt-in evaluation of streamed OpenAI Chat Completions and Responses via DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.
  • This introduces AI Guard standalone mode. When APM tracing is disabled (DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.
  • Add opt-in buffered stream response evaluation for AI Guard on Anthropic streaming calls. When DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged.
• AAP
  • Adds the normalized HTTP route span tag (_dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398
• AppSec:
  • Detect OpenAI LLM usage within API endpoints. When AppSec is enabled and an OpenAI chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).
• aws_durable_execution_sdk_python
  • Add aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.
• ray_serve
  • Adds instrumentation support for ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076
• LLM Observability
  • This introduces support for attaching audio to messages on LLM spans. Input and output messages annotated via LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.
  • Added a sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.
  • Add SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.
  • LLM span data can now be exported to the Datadog Agent over the APM trace, and is reliably delivered even when the APM trace is not sampled. This has no effect on APM sampling decisions or billing.
  • The OpenAI integration now captures input and output audio on audio chat completions (any audio-capable Chat Completions model, e.g. gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.
  • Adds support for DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.
  • Adds span metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.
  • Adds support for submitting LLM Observability spans for mistralai client chat and embedding calls.
• profiling
  • implement heap live count data collection
  • The adaptive sampling mechanism has been revamped to better match the needs of real-life workloads.
  • The PyTorch profiler now properly shows nested frames in flame graphs.
• mistralai
  • Adds tracing support for the mistralai client chat and embedding calls.
• pytorch
  • Adds a pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.
• This change adds the ability to override a manually installed tracer for Single Step Instrumentation. When this is set, all ddtrace packages will be preferred over the user installed packages. For example, wrapt, bytecode, and others. To ensure the injected library takes precedence, DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.
• CI Visibility
  • The DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.

Bug Fixes

• LLM Observability:
  • Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).
  • Resolves an issue in the openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk.
  • Fixes an issue where the span writer ignored its configured request timeout (_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.
  • This fix resolves an issue where the OpenAI and Anthropic integrations did not record the model output on the LLM Observability span when AI Guard blocked the request after the model call completed. The model response is now captured even though the span is marked as errored by the block.
  • Fixes an issue where spans with very large JSON depth nested fields were being submitted but dropped by Datadog. The LLM Observability integration now detects nested fields that exceed the allowed depth and stringifies them, ensuring spans will not be dropped due to JSON depth limits in Datadog.
  • Resolves an issue where Azure OpenAI calls through the LiteLLM integration tagged model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.
  • Resolves an issue where spans annotated with metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.
  • Resolves an issue where the OpenAI integration recorded unset request parameters (OpenAI SDK's Omit/NotGiven sentinel values) as...

Bug Fixes

• AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.

• tracing: This fix resolves a rare issue that could cause an application to hang after forking a child process.

• LLM Observability: This fix resolves an issue where the google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.

• LLM Observability: This fix resolves an issue where the Google GenAI and google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content.

• CI Visibility: This fix resolves an issue where quarantined tests were not retried by Auto Test Retry (ATR).

Bug Fixes

• tracing: This fix resolves a rare issue that could cause an application to hang after forking a child process.

Bug Fixes

• AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.

• database monitoring: Resolved a memory leak that grew with query volume when DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.

• botocore: Fixes a bug in Data Streams Monitoring where the pathway context of an SQS message was not extracted when the message body was valid JSON, silently breaking the data stream for non-SNS messages with a JSON payload. The _datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.

• Django: This fix resolves an issue where async function-based middleware attributes, such as process_exception, were not preserved, causing Django to skip exception handlers under ASGI.

• tracing: This fix resolves a rare issue that could cause an application to hang after forking a child process.

• CI Visibility: This fix resolves an issue where quarantined tests were not retried by Auto Test Retry (ATR).