Skip to content

[Bug]: synthetic execution root is never exported in fallback traces #934

Closed

Description

@zhongkechen

Follow-up to the synthetic-parent problem acknowledged in #929.

Expected Behavior

When a sampled execution uses an SDK-created fallback ancestor, its completed trace should have a connected exported hierarchy. The SDK should supply the ancestor it invents, or resolve a hierarchy that does not reference a permanently unexported SDK-created parent.

Root ownership must account for multiple executions sharing one trace ID. Simply exporting a different root for each execution would introduce multiple distinct roots into that trace.

Actual Behavior

When no complete remote parent can be constructed, resolveExecutionTraceContext creates only a SpanContext whose ID is deriveExecutionRootSpanId(executionArn). Neither ExecutionOtelPlugin nor InvocationOtelPlugin creates a recording span for that ID.

At terminal completion, the plugin exports Workflow under this synthetic parent. With no ambient parent, Invocation also references it. The parent therefore never reaches the exporter, even when sampling is enabled and all flushes finish.

Unlike a propagated parent whose span another component can export, this ancestor is invented by the SDK. Received Workflow and operation spans remain available, but their common parent is permanently absent; backends can display an incomplete hierarchy or missing-root metadata.

Source at OTel 1.1.0:

Steps to Reproduce

This local plugin-level reproducer uses the published packages and a real OpenTelemetry provider/exporter. It deliberately removes extracted context to exercise the SDK-owned fallback; no Lambda account is required.

npm init -y
npm install @aws/[email protected] @aws/[email protected] @opentelemetry/[email protected] @opentelemetry/[email protected]

Save the following as repro.cjs and run node repro.cjs:

const assert = require('node:assert/strict');
const {
  NodeTracerProvider, InMemorySpanExporter, SimpleSpanProcessor, AlwaysOnSampler,
} = require('@opentelemetry/sdk-trace-node');
const {
  ExecutionOtelPlugin, deriveExecutionRootSpanId,
} = require('@aws/durable-execution-sdk-js-otel');

(async () => {
  const exporter = new InMemorySpanExporter();
  const provider = new NodeTracerProvider({
    sampler: new AlwaysOnSampler(),
    spanProcessors: [new SimpleSpanProcessor(exporter)],
  });
  provider.register();
  try {
    // Deliberately exercise the SDK-owned fallback, independently of Lambda.
    const plugin = new ExecutionOtelPlugin({ contextExtractor: () => undefined });
    const info = {
      executionArn: 'arn:aws:lambda:us-east-1:123456789012:function:root-repro:1/durable-execution/root-repro/00000000-0000-4000-8000-000000000001',
      requestId: 'request-1', executionInput: {}, operations: {}, updatedOperations: {},
      isFirstInvocation: true,
      executionStartTimestamp: new Date('2026-09-29T00:00:00Z'),
    };
    await plugin.onInvocationStart(info);
    await plugin.onInvocationEnd({ ...info, status: 'SUCCEEDED' });
    await provider.forceFlush();
    const spans = exporter.getFinishedSpans();
    const workflow = spans.find(span => span.name === 'Workflow');
    const syntheticId = deriveExecutionRootSpanId(info.executionArn);
    const result = {
      exportedSpanNames: spans.map(span => span.name).sort(),
      workflowUsesSyntheticParent: workflow?.parentSpanContext?.spanId === syntheticId,
      syntheticParentExported: spans.some(span => span.spanContext().spanId === syntheticId),
      exportedRootCount: spans.filter(span => !span.parentSpanContext).length,
    };
    assert.deepEqual(result, {
      exportedSpanNames: ['Invocation', 'Workflow'],
      workflowUsesSyntheticParent: true,
      syntheticParentExported: false,
      exportedRootCount: 0,
    });
    console.log(JSON.stringify(result, null, 2));
  } finally {
    await provider.shutdown();
  }
})().catch(error => { console.error(error); process.exitCode = 1; });

Observed output:

{
  "exportedSpanNames": ["Invocation", "Workflow"],
  "workflowUsesSyntheticParent": true,
  "syntheticParentExported": false,
  "exportedRootCount": 0
}

The same fallback occurs when a custom extractor returns a valid traceId and sampling: 'UNDECIDED' but omits parentSpanId.

SDK Version

  • @aws/durable-execution-sdk-js: 2.4.0
  • @aws/durable-execution-sdk-js-otel: 1.1.0
  • @opentelemetry/sdk-trace-node: 2.8.0
  • @opentelemetry/api: 1.9.1

The relevant fallback behavior is also present in OTel 1.0.0.

Node.js Version

24.x; the reproducer above was run on v24.21.0.

Is this a regression?

No previously published working version has been established. The shared-ancestor topology was introduced by #854; the earlier parentless Workflow topology existed on main before its release.

Last Working Version

None identified.

Additional Context

The live Lambda tests discussed in #929 verified the propagated-parent path, including preservation of Root, Parent, and Sampled across wait/resume. They did not exercise this synthetic fallback. The evidence here is the local runtime reproduction and source inspection.

Source inspection found the same behavior in the other SDKs:

Coordinate the fallback contract and validation with aws/aws-durable-execution-conformance-tests#99.

Fix acceptance should cover:

  • A sampled fallback execution has an exported, connected hierarchy after terminal success or failure, in both plugin views.
  • Missing/invalid remote context and a valid trace ID without a usable parent are both covered.
  • Wait/resume and retries preserve trace identity, deterministic span IDs, and links without conflicting exports for one span identity.
  • Multiple executions sharing a propagated Root do not produce multiple distinct roots.
  • Explicit upstream sampling decisions remain authoritative; undecided sampling still uses the configured sampler, and unsampled executions do not emit a stray root.
  • Existing remote-parent propagation remains supported. This fix addresses the SDK-created fallback ancestor.

Activity

  1. added
    bugSomething isn't working
    pkg:otelPackage: aws-durable-execution-sdk-js-otel
    on Sep 29, 2026
  2. blackmore-technology-group commented on Sep 29, 2026

    @blackmore-technology-group
    Contributor

    I’ve opened PR #935 with a fix for this issue:

    #935

    The change exports the SDK-created synthetic execution root for sampled fallback traces, keeps Workflow and Invocation connected under that root, preserves remote-parent behavior and sampling decisions, and includes coverage for terminal success/failure, wait/resume, unsampled fallback, and shared propagated Root identity.

    Local validation is green: 352/352 OTel tests, core + OTel builds, and Biome all pass.

    The AWS Build and OTel Conformance workflows are currently awaiting maintainer approval before jobs can run.

  3. jmsundar commented on Sep 29, 2026

    @jmsundar

    Related: this fix doesn't reach functions with X-Ray disabled and no caller header, because the Parent Lambda generates keeps them off this fallback path. The ask to support no-X-Ray deployments with minimal configuration is in #929 (comment).

  4. blackmore-technology-group commented on Sep 30, 2026

    @blackmore-technology-group
    Contributor

    PR #935 review update: I pushed adfa3633d594ea74b30d351f31a4e668874b2954 to address the reviewer blocker.

    The synthetic execution root is now ARN-scoped again, so two executions sharing one propagated trace cannot export conflicting spans with the same (traceId, spanId). The regression now uses distinct caller/callee provider resources and execution start times and verifies distinct execution-owned root identities.

    Local validation is green: targeted regressions, full OTel tests, core + OTel ESM/CJS/types builds, and Biome.

    The X-Ray-disabled/no-caller-header path noted above remains separate and tracked through #929; this revision stays focused on #934.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageIssue needs triagepkg:otelPackage: aws-durable-execution-sdk-js-otel

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions