Skip to content

OTEL: Update Operation Name to Match Frontend Usage#1033

Merged
zarirhamza merged 6 commits into
mainfrom
zarir/OTLP-operation-name-change
Apr 30, 2026
Merged

OTEL: Update Operation Name to Match Frontend Usage#1033
zarirhamza merged 6 commits into
mainfrom
zarir/OTLP-operation-name-change

Conversation

@zarirhamza

@zarirhamza zarirhamza commented Feb 17, 2026

Copy link
Copy Markdown
Contributor

Overview

OTEL Lambda spans emitted by the opentelemetry.instrumentation.aws_lambda Python instrumentation come through the Extension with name: server.request — the OTEL semantic-convention generic-server fallback. The Datadog Serverless page invocation table filters by:

operation_name:("aws.lambda" OR "aws.lambda.function"
              OR "opentelemetry_instrumentation_aws_lambda.server"
              OR "opentelemetry.instrumentation.aws_lambda.server")

server.request matches none of those, so OTEL Lambda functions show 0 traces on the Serverless page even though the spans land in APM correctly.

This PR overrides the operation name to aws.lambda for OTEL Lambda root-invocation spans on the Extension side. Detection is gated on the instrumentation scope being opentelemetry.instrumentation.aws_lambda AND span kind being Server, so other spans in the trace (HTTP clients, internal spans, custom spans) are not touched.

What Changed

bottlecap/src/otlp/transform.rsget_otel_operation_name_v2:

  • Threads the OTEL InstrumentationScope reference through to the operation-name resolver
  • Adds an early branch: lib.name == "opentelemetry.instrumentation.aws_lambda" AND span.kind == Server"aws.lambda"
  • All other spans continue to resolve via the existing HTTP / DB / messaging / RPC / FAAS / generic fallback chain
  • The lib parameter was already available at the call site in otel_span_to_dd_span; the change just plumbs it into get_otel_operation_name_v2

Why on the Extension (vs. logs-backend)

The complementary path — spans arriving through the OTLP endpoint without going through the Extension — is handled by https://github.com/DataDog/logs-backend/pull/134974, which performs the same server.request → aws.lambda override in SpansOTLPProtobufPayloadParser along with the broader OTEL → Datadog field remap (cloud.resource_id → function_arn, faas.invocation_id → request_id, etc.).

Doing the override on the Extension side for the Extension-forwarded path is preferred because:

  1. The override is closer to the source of truth — the Extension is the only component that knows the span came from the OTEL aws_lambda instrumentation
  2. APM stats / error tracking / rule-based sampling all see the same aws.lambda name, avoiding "named differently in different places" bugs
  3. Mirrors how the Extension already overrides other OTEL-side fields with Datadog conventions

Testing

Existing test_otel_operation_name_* extended to assert server.request is still produced when the lib name does not match the AWS Lambda scope (regression coverage for the threading change).

New test_otel_operation_name_aws_lambda covers three cases:

  1. AWS Lambda scope + Server kind → aws.lambda
  2. AWS Lambda scope + Internal kind (handler child spans) → unchanged (SPAN_KIND_INTERNAL)
  3. Different scope (e.g., handler) + Server kind → unchanged (server.request)

Companion PRs

  • logs-backend #134974 — same operation-name override for the OTLP-endpoint path, plus the OTEL → Datadog field remap
  • web-ui #300855 — defensive request_id fallback in the trace list hook + widened log queries that match OTLP-ingested logs alongside Datadog Forwarder logs

Together with this PR, OTEL-instrumented AWS Lambda functions (both endpoint-only and Extension-forwarded) work end-to-end on the Datadog Serverless page: trace discovery, invocation correlation, function-level log discovery, error filter, log count.

@zarirhamza
zarirhamza marked this pull request as ready for review April 29, 2026 17:42
@zarirhamza
zarirhamza requested a review from a team as a code owner April 29, 2026 17:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the OTLP → Datadog span name resolution so OTEL-instrumented AWS Lambda invocation spans are reported with an operation name the Datadog Serverless UI recognizes, preventing “0 traces” in the invocation table for Extension-forwarded OTEL traces.

Changes:

  • Thread the OTEL InstrumentationScope into get_otel_operation_name_v2.
  • Override operation name to aws.lambda for SpanKind::Server spans emitted by opentelemetry.instrumentation.aws_lambda.
  • Extend/add unit tests covering the new AWS Lambda-specific naming branch and regressions.

Comment thread bottlecap/src/otlp/transform.rs Outdated
The previous comment said "root Lambda invocation span" but the code only
checks SpanKind::Server + instrumentation scope name, which does not verify
root-ness — a Lambda span can have a remote parent when the upstream caller
(e.g. API Gateway, Step Functions, another Lambda) propagates trace context.

That is still the Lambda invocation span we want to rename to aws.lambda, so
the behavior is correct; just clarify the comment to avoid implying a root
guarantee that the predicate doesn't enforce.

No code change.
Comment thread bottlecap/src/otlp/transform.rs

@duncanista duncanista left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, left a comment

@zarirhamza
zarirhamza merged commit b86a291 into main Apr 30, 2026
56 of 59 checks passed
@zarirhamza
zarirhamza deleted the zarir/OTLP-operation-name-change branch April 30, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants