Skip to content

[OpenTelemetry] Document and align cross-SDK behavioral differences #23

New issue
New issue
Closed
Closed
[OpenTelemetry] Document and align cross-SDK behavioral differences#23
Assignees
zhongkechen

Description

@zhongkechen
zhongkechen
opened on Jul 23, 2026

Context

The current OpenTelemetry conformance tests were initially derived from the Python SDK's telemetry behavior. Python therefore appears most conformant by construction. This issue documents the differences observed across the three SDKs so that we can decide which behaviors belong in a language-neutral contract and which are SDK defects.

The Python and TypeScript evidence below is unchanged from the previous runs because those SDK versions have not changed. Java was refreshed against SDK main at fd3505b (2.1.1-SNAPSHOT). Each ADOT + X-Ray job ran the same 19 OpenTelemetry scenarios:

SDK Run Result Scenarios with correlated telemetry
Python 29985063295 0/19 passed 19/19
Java (main) 30027042347 0/19 passed 19/19
TypeScript 29986336955 0/19 passed 16/19

All three listed jobs failed their current conformance assertions. In the refreshed Java job, the SDK checkout, sdk and otel-plugin installation, and handler compilation all succeeded; the job failed while evaluating the telemetry scenarios. The Python community-layer/Dash0/Datadog jobs were skipped, so those exporters are outside this comparison.

Python

Python's durable operation and attempt spans match the current test model: expected attributes are present, operation and attempt spans use the expected parent, and failed attempts are exported as OpenTelemetry ERROR with durable.attempt.outcome=FAILED.

Its failures reduce to invocation-span semantics:

  • All 28 invocation spans use span.kind=INTERNAL; the tests expect SERVER.
  • All 28 invocation spans have OpenTelemetry status OK, including failed handler invocations; the tests expect ERROR for failures.
  • The ambiguous selections in otel-17 and otel-18 are downstream effects of failed and successful invocations all having status OK.

Aside from those two invocation properties, the Python telemetry satisfies the current assertions.

Java

The SDK main run did not resolve the differences previously observed with the Maven Central 2.1.0 release. Java exports correlated telemetry for all 19 scenarios, and its invocation spans behave better than Python in two respects: all 28 are SERVER, and failed invocations are exported as ERROR (9 ERROR, 19 OK).

The remaining shape differs substantially from the Python-derived model:

  • Invocation spans add durable.invocation.first and durable.invocation.status. The current exact-key assertion permits only durable.execution.arn.
  • In-progress operations use durable.operation.status=PENDING rather than STARTED.
  • Attempt spans are children of operation spans. The current model expects operation and attempt spans to share the enclosing invocation/context parent.
  • All 44 operation lifecycle spans omit durable.attempt.number.
  • All 17 attempt spans omit durable.operation.subtype and durable.operation.status.
  • All 17 attempt spans have OpenTelemetry status OK; 14 emit durable.attempt.outcome=SUCCEEDED, 3 omit the outcome, and none emit FAILED.
  • In the retry scenario (otel-3), attempt 1 is exported as OK/SUCCEEDED even though it failed, while attempt 2 omits durable.attempt.outcome.
  • Similar parent and attribute differences occur for child contexts, parallel branches, map iterations, callbacks, invokes, and wait-for-condition attempts.

Some Java differences may be intentional extensions or modeling choices. In particular, the invocation attributes could be valid additions if the cross-SDK contract permits extensible durable.* attributes.

TypeScript

TypeScript has both telemetry-shape differences and execution/export failures:

  • The exported invocation wrapper is SERVER, but span.name is the Lambda function name rather than invocation.
  • Invocation spans have no durable.execution.arn, so the harness reports zero matching SDK invocation spans.
  • All observed invocation spans have OpenTelemetry status OK, including failure paths.
  • Operation spans never include durable.operation.status.
  • Attempt numbering uses durable.operation.attempt rather than durable.attempt.number.
  • Attempt spans are children of operation spans rather than siblings under the enclosing invocation/context.
  • otel-10 and otel-17 do not complete because the harness cannot match the anonymous callback action; both time out.
  • otel-19 produces no correlated X-Ray trace when handler execution fails.

Only 16 of the 19 scenarios produced normalized telemetry artifacts. The callback failures and missing failed-handler trace are functional gaps independent of the exact telemetry schema.

Contract decisions needed

  1. Define the language-neutral invocation contract: logical span name, span kind, required attributes, and failure status.
  2. Decide whether additional durable.* attributes are allowed or exact-key matching remains required.
  3. Define operation/attempt parentage and whether both spans repeat operation subtype, status, and attempt number.
  4. Standardize lifecycle vocabulary such as STARTED versus PENDING.
  5. Standardize failure mapping for invocation, operation, and attempt spans.
  6. Update the conformance requirements to describe the agreed contract independently of any SDK implementation.
  7. Track SDK-specific fixes only after intentional differences have been separated from contract violations.

Completion criteria

  • The language-neutral span contract is documented.
  • Tests distinguish required fields from permitted extensions.
  • Intentional SDK differences are represented explicitly in the requirements.
  • Remaining Python, Java, and TypeScript deviations are tracked as implementation defects.

Activity

  1. zhongkechen
    self-assigned this
    on Jul 23, 2026

  2. SilanHe commented on Jul 23, 2026

    @SilanHe
    SilanHe
    on Jul 23, 2026
    Contributor
    1. Invocation spans should use span.kind=INTERNAL.
    2. All invocations have span status UNSET when Execution not completed. If Execution is failed, the last Invocation span will have status ERROR. We need to fix this behaviour in Python SDK and Typescript SDK.
    3. Add durable.invocation.first and durable.invocation.status to Python and Javascript Otel plugins. Suspends = PENDING. Fails = RETRYING. Succeeded and failed for execution status.
    4. PluginOperationStatus needs to be implemented for all operations (durable.operation.status). Javascript is missing the field for certain onOperationEnd calls. Java Otel plugin doesn't need to explicitly update spans with PluginOperationStatus.PENDING in onInvocationEnd for operations that have not yet completed. We're assuming that they would just be PluginOperationStatus.STARTED. Double check for Wait-for-condition.
    5. We think there might be a bug in python Otel plugin/plugin interface relating to attempts of operations. Currrently, Python operation attempts share the same parent as the operation. Instead, the attempt's parent span should be the operation span.
    6. In Python, STEP operation span and WAIT-FOR-CONDITION operation span exports an attribute to the span dubbed durable.attempt.number to denote the total number of attempts that happened. Should only be emitted after the operation is completed.
    7. All attributes from operation spans are available in associated attempt spans. Attempt spans additionally have durable.attempt.number and durable.attempt.outcome attributes.
    8. Java otel plugin attempts should include durable.attempt.outcome attribute. Specifically, we seem to be missing the FAILED outcomes.
    9. For invocation view using ADOT layer (specifically Typescript), create the "invocation" span regardless and include durable.execution.arn.
    10. For Typescript, rename durable.operation.attempt attribute to durable.attempt.number in attempts span.

  3. zhongkechen commented on Jul 23, 2026

    @zhongkechen
    zhongkechen
    on Jul 23, 2026
    ContributorAuthor

    Fix of invocation span kind and attributes in Python SDK: aws/aws-durable-execution-sdk-python#577

  4. zhongkechen commented on Jul 24, 2026

    @zhongkechen
    zhongkechen
    on Jul 24, 2026
    ContributorAuthor

    Detailed community layer + S3 collector comparison

    This compares the three community-layer/S3 jobs at the same PR head, f7c17c7. That revision includes the canonical invocation-counting fix and runs the original 19 OpenTelemetry scenarios.

    SDK Result Scenarios with correlated telemetry Run
    Python 18/19 passed 19/19 Community layer + S3 collector
    Java 0/19 passed 19/19 Community layer + S3 collector
    TypeScript 0/19 passed 16/19 Community layer + S3 collector

    All three jobs built and deployed the same custom OpenTelemetry collector with the S3 exporter. The differing results are therefore comparisons of the SDK/community-layer telemetry carried by the same raw OTLP path.

    Python: 18/19

    otel-1 through otel-18 all pass. This covers successful and failed operations, multiple physical invocations, retries, child/parallel/map contexts, callbacks, wait-for-condition, and chained invokes. The invocation-counting fix also eliminates all count mismatches, including when the community layer does not provide a faas.invocation_id.

    The sole failure is otel-19:

    • The expected invocation span is present with kind=INTERNAL, OpenTelemetry status ERROR, the execution ARN, and durable.invocation.first=true.
    • Python emits durable.invocation.status=FAILED.
    • The requirement selects durable.invocation.status=RETRY, so the otherwise matching span is not selected and is subsequently reported as uncovered.

    This is a lifecycle-value mismatch between the requirement and the Python plugin, not a collector, correlation, or invocation-count problem.

    Java: 0/19, with correlated telemetry for every case

    Java produces correlated telemetry for all 19 scenarios and does not have any invocation-count failures. Across the uploaded traces, all 26 invocation spans have unique invocation IDs plus durable.invocation.first and durable.invocation.status.

    The failures come from a shared telemetry shape that differs from the current Python-derived contract:

    • All Java invocation spans are SERVER, while the requirements expect INTERNAL.
    • Failed invocations are correctly ERROR/FAILED, but successful and pending invocation spans use OpenTelemetry status UNSET rather than OK.
    • In-progress operation lifecycle spans use durable.operation.status=PENDING rather than STARTED.
    • Operation lifecycle spans generally omit durable.attempt.number.
    • Attempt spans are children of operation spans rather than siblings under the enclosing invocation/context span.
    • All 17 attempt spans are UNSET; 12 report durable.attempt.outcome=SUCCEEDED, five omit the outcome, and none report a failed outcome.
    • Attempt spans omit durable.operation.subtype and durable.operation.status.
    • Some parallel/map/callback traces are missing expected correlated spans, producing minimum-span and parent-resolution failures.

    Thus Java's 0/19 result does not mean telemetry is absent. Its invocation lifecycle is complete enough to count correctly, but invocation kind/status, operation/attempt attributes, failure outcomes, hierarchy, and some multi-context correlation differ from the current assertions.

    TypeScript: 0/19, with operation telemetry but no canonical invocation view

    TypeScript produces normalized telemetry artifacts for 16 scenarios, but none contains either a canonical invocation span or an invocation-ID attribute:

    • The available Lambda wrapper is named after the function, is SERVER, and carries no durable execution or invocation lifecycle attributes.
    • The five scenarios requiring multiple physical invocations (otel-2, otel-3, otel-9, otel-11, and otel-18) consequently report zero recognized invocations.
    • Durable operation spans are present and INTERNAL, but all 28 omit durable.operation.status.
    • Attempt spans are children of operation spans and use durable.operation.attempt rather than durable.attempt.number.
    • All 12 attempt spans carry one span link and report outcomes: seven SUCCEEDED and five FAILED. Four failed attempts are ERROR; the otel-9 condition attempt remains UNSET.
    • Multi-invocation execution traces are fragmented, reducing correlated span counts.
    • otel-10 and otel-17 time out because the anonymous callback action cannot be matched.
    • otel-19 produces no execution-correlated trace within the 90-second lookup window.

    TypeScript therefore has a useful operation/attempt model, including correct failed-attempt status and outcome, but it lacks the logical invocation model and cross-invocation correlation required by the current contract.

    Cross-SDK comparison

    • Python most closely matches the current contract: 18/19 cases pass, with only the FAILED versus RETRY lifecycle value in otel-19.
    • Java has the strongest physical-invocation coverage and complete invocation identities, but differs broadly in canonical span kind/status, operation/attempt duplication, parentage, and attempt failure modeling.
    • TypeScript models operation attempts and their outcomes better than Java, but has no conforming logical invocation spans or invocation identities and loses correlation across replays/invokes.
    • Collector/S3 path is functioning across all three languages. It preserves raw span kind, status, attributes, parent IDs, and links; the observed differences originate in the emitted telemetry shape or the current language-neutral contract, not in S3 normalization.

    The main contract decisions remain invocation kind/status, FAILED versus RETRY, operation/attempt parentage, repeated attributes across operation and attempt spans, lifecycle vocabulary, and whether linked physical invocation spans must join the execution-correlated trace.

  5. zhongkechen
    added this to aws-durable-executionon Jul 24, 2026
  6. github-project-automation
    moved this to Backlog in aws-durable-executionon Jul 24, 2026
  7. SilanHe
    added a commit that references this issue on Jul 24, 2026

    fix(otel): cross sdk behaviour alignment (#756)

    254f2a3

  8. SilanHe commented on Jul 27, 2026

    @SilanHe
    SilanHe
    on Jul 27, 2026
    Contributor

    Want to document the following requirement as well:

    In execution view, execution/workflow span status can use this mapping from PluginInvocationStatus to span Status:

    FAILED -> ERROR
    SUCCEEDED -> OK
    RETRY/PENDING -> UNSET

    which reflects the execution status

    For invocation span status, we want to use the following mapping from PluginInvocationStatus to span Status:

    SUCCEEDED/PENDING -> OK
    RETRY -> UNSET
    FAILED -> ERROR

    We also want to document that the plugin interface doesn't know if an invocation.workflow was STOPPED or TIMED_OUT, which is also why RETRY maps to UNSET. The workflow span should be INTERNAL as well (span.kind=INTERNAL).

  9. SilanHe
    added a commit that references this issue on Jul 28, 2026

    feat(otel): workflow span status (#764)

    b4a1f54

  10. SilanHe commented on Jul 28, 2026

    @SilanHe
    SilanHe
    on Jul 28, 2026
    Contributor

    Both the InvocationOtelPlugin and the ExecutionOtelPlugin should create a "Workflow" span. Capitalize first letter of the names for the "Invocation" and "Workflow" spans

  11. SilanHe
    added a commit that references this issue on Jul 29, 2026

    feat(otel): capitalize InvocationOtelPlugin invocation span name (#769)

    d2f0174

  12. SilanHe commented on Jul 31, 2026

    @SilanHe
    SilanHe
    on Jul 31, 2026
    Contributor

    I think we should align on Java behaviour here:

    Case Python (invocation_plugin.py) Java (InvocationOtelPlugin, #572) #774 (JS) Conformance yaml (PR #40)
    1a. Container CONTEXT (succeeds) STARTED SUCCEEDED STARTED expects STARTED
    1b. Suspended op (started, unresumed) STARTED attribute absent STARTED expects STARTED
    3. Attempt span operation.status=STARTED + attempt.outcome no operation.status; only attempt.outcome STARTED + attempt.outcome expects STARTED on attempt?

    This will require an update to conformance tests, Python and JS

  13. SilanHe commented on Aug 4, 2026

    @SilanHe
    SilanHe
    on Aug 4, 2026
    Contributor

    Operation start and end time should be constrained within Invocation spans start and end times

  14. SilanHe commented on Aug 4, 2026

    @SilanHe
    SilanHe
    on Aug 4, 2026
    Contributor

    Plugin configuration differences:

    Option JS (both) Py Execution Py Invocation Java (both)
    Custom provider tracerProvider (built TracerProvider) tracer_provider trace_provider tracerProviderBuilder (a builder; plugin calls .setIdGenerator().build())
    Use global/default provider useDefaultTracerProvider (bool) use_default_tracer_provider implicit — always falls back to get_tracer_provider() separate no-arg ctor (ADOT global)
    contextExtractor ✅ ✅ ✅ ✅
    Instrumentation scope name ✅ instrumentationName ✅ instrument_name ✅ instrument_name ❌ hardcoded "aws-durable-execution-sdk-java"
    Auto HTTP instrumentation toggle ✅ enableHttpInstrumentation ✅ ❌ ❌
    OTLP exporter config (endpoint/headers) ✅ exporterConfig ✅ exporter_config ❌ ❌ (BYO builder / no auto-OTLP)
    Custom propagators ✅ propagators ✅ ❌ ❌
    workflowSpanName ✅ ✅ ignored (not configurable) ✅ workflowSpanName
    Log correlation toggle ❌ none ✅ enrich_logger ✅ enrich_logger ✅ enableMdc
    Auto-creates OTLP provider when nothing supplied ✅ both ✅ ❌ (explicit-or-global only) ❌ neither

    I've also decided to use an enum for the TracerProvider source. Having multiple variables and order determine what TracerProvider the plugin was going to use is too confusing.

  15. SilanHe
    added a commit that references this issue on Aug 4, 2026

    feat(otel): add enrichLogger config toggle to gate log context enrich…

    954abb3

  16. SilanHe commented on Aug 4, 2026

    @SilanHe
    SilanHe
    on Aug 4, 2026
    Contributor
    Plugin Sampled field key ID keys Value type
    JS Execution otelTraceSampled traceId / spanId boolean
    JS Invocation otelTraceSampled traceId / spanId boolean
    Python Execution otelTraceSampled traceId / spanId boolean
    Python Invocation otelTraceSampled traceId / spanId boolean
    Java Execution traceSampled trace_id / span_id string ("true"/"false")
    Java Invocation traceSampled trace_id / span_id string ("true"/"false")

  17. SilanHe commented on Aug 7, 2026

    @SilanHe
    SilanHe
    on Aug 7, 2026
    Contributor

    There could be a gap between the behaviour in the ExecutionOtelPlugin when we're not using the ProviderSource.GLOBAL.

    I believe in JS, the Invocation span is a child of the Workflow span. In Python, the Invocation spans are root spans. We'll have to double check.

  18. SilanHe
    added a commit that references this issue on Aug 7, 2026

    refactor(otel): resolve provider source via enum (#792)

    31d7dbe
  19. zhongkechen
    closed this as completedon Aug 21, 2026
  20. github-project-automation
    moved this from Ready to Done in aws-durable-executionon Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions