Repository navigation
Description
Follow-up to the synthetic-parent problem acknowledged in #929.
Expected Behavior
When a sampled execution uses an SDK-created fallback ancestor, its completed trace should have a connected exported hierarchy. The SDK should supply the ancestor it invents, or resolve a hierarchy that does not reference a permanently unexported SDK-created parent.
Root ownership must account for multiple executions sharing one trace ID. Simply exporting a different root for each execution would introduce multiple distinct roots into that trace.
Actual Behavior
When no complete remote parent can be constructed, resolveExecutionTraceContext creates only a SpanContext whose ID is deriveExecutionRootSpanId(executionArn). Neither ExecutionOtelPlugin nor InvocationOtelPlugin creates a recording span for that ID.
At terminal completion, the plugin exports Workflow under this synthetic parent. With no ambient parent, Invocation also references it. The parent therefore never reaches the exporter, even when sampling is enabled and all flushes finish.
Unlike a propagated parent whose span another component can export, this ancestor is invented by the SDK. Received Workflow and operation spans remain available, but their common parent is permanently absent; backends can display an incomplete hierarchy or missing-root metadata.
Source at OTel 1.1.0:
- Resolver creates the synthetic context.
- Execution plugin materializes Workflow under that context.
- Invocation plugin uses the same parent when materializing Workflow.
Steps to Reproduce
This local plugin-level reproducer uses the published packages and a real OpenTelemetry provider/exporter. It deliberately removes extracted context to exercise the SDK-owned fallback; no Lambda account is required.
npm init -y
npm install @aws/[email protected] @aws/[email protected] @opentelemetry/[email protected] @opentelemetry/[email protected]Save the following as repro.cjs and run node repro.cjs:
const assert = require('node:assert/strict');
const {
NodeTracerProvider, InMemorySpanExporter, SimpleSpanProcessor, AlwaysOnSampler,
} = require('@opentelemetry/sdk-trace-node');
const {
ExecutionOtelPlugin, deriveExecutionRootSpanId,
} = require('@aws/durable-execution-sdk-js-otel');
(async () => {
const exporter = new InMemorySpanExporter();
const provider = new NodeTracerProvider({
sampler: new AlwaysOnSampler(),
spanProcessors: [new SimpleSpanProcessor(exporter)],
});
provider.register();
try {
// Deliberately exercise the SDK-owned fallback, independently of Lambda.
const plugin = new ExecutionOtelPlugin({ contextExtractor: () => undefined });
const info = {
executionArn: 'arn:aws:lambda:us-east-1:123456789012:function:root-repro:1/durable-execution/root-repro/00000000-0000-4000-8000-000000000001',
requestId: 'request-1', executionInput: {}, operations: {}, updatedOperations: {},
isFirstInvocation: true,
executionStartTimestamp: new Date('2026-09-29T00:00:00Z'),
};
await plugin.onInvocationStart(info);
await plugin.onInvocationEnd({ ...info, status: 'SUCCEEDED' });
await provider.forceFlush();
const spans = exporter.getFinishedSpans();
const workflow = spans.find(span => span.name === 'Workflow');
const syntheticId = deriveExecutionRootSpanId(info.executionArn);
const result = {
exportedSpanNames: spans.map(span => span.name).sort(),
workflowUsesSyntheticParent: workflow?.parentSpanContext?.spanId === syntheticId,
syntheticParentExported: spans.some(span => span.spanContext().spanId === syntheticId),
exportedRootCount: spans.filter(span => !span.parentSpanContext).length,
};
assert.deepEqual(result, {
exportedSpanNames: ['Invocation', 'Workflow'],
workflowUsesSyntheticParent: true,
syntheticParentExported: false,
exportedRootCount: 0,
});
console.log(JSON.stringify(result, null, 2));
} finally {
await provider.shutdown();
}
})().catch(error => { console.error(error); process.exitCode = 1; });Observed output:
{
"exportedSpanNames": ["Invocation", "Workflow"],
"workflowUsesSyntheticParent": true,
"syntheticParentExported": false,
"exportedRootCount": 0
}The same fallback occurs when a custom extractor returns a valid traceId and sampling: 'UNDECIDED' but omits parentSpanId.
SDK Version
@aws/durable-execution-sdk-js: 2.4.0@aws/durable-execution-sdk-js-otel: 1.1.0@opentelemetry/sdk-trace-node: 2.8.0@opentelemetry/api: 1.9.1
The relevant fallback behavior is also present in OTel 1.0.0.
Node.js Version
24.x; the reproducer above was run on v24.21.0.
Is this a regression?
No previously published working version has been established. The shared-ancestor topology was introduced by #854; the earlier parentless Workflow topology existed on main before its release.
Last Working Version
None identified.
Additional Context
The live Lambda tests discussed in #929 verified the propagated-parent path, including preservation of Root, Parent, and Sampled across wait/resume. They did not exercise this synthetic fallback. The evidence here is the local runtime reproduction and source inspection.
Source inspection found the same behavior in the other SDKs:
- Java 2.2.1: creates a synthetic SpanContext without an exported counterpart.
- Python OTel 1.0.0: creates the synthetic context, represented by DurableParentSpan, which never records or exports.
Coordinate the fallback contract and validation with aws/aws-durable-execution-conformance-tests#99.
Fix acceptance should cover:
- A sampled fallback execution has an exported, connected hierarchy after terminal success or failure, in both plugin views.
- Missing/invalid remote context and a valid trace ID without a usable parent are both covered.
- Wait/resume and retries preserve trace identity, deterministic span IDs, and links without conflicting exports for one span identity.
- Multiple executions sharing a propagated
Rootdo not produce multiple distinct roots. - Explicit upstream sampling decisions remain authoritative; undecided sampling still uses the configured sampler, and unsampled executions do not emit a stray root.
- Existing remote-parent propagation remains supported. This fix addresses the SDK-created fallback ancestor.
Activity
blackmore-technology-group commentedon Sep 29, 2026
I’ve opened PR #935 with a fix for this issue:
#935
The change exports the SDK-created synthetic execution root for sampled fallback traces, keeps Workflow and Invocation connected under that root, preserves remote-parent behavior and sampling decisions, and includes coverage for terminal success/failure, wait/resume, unsampled fallback, and shared propagated Root identity.
Local validation is green: 352/352 OTel tests, core + OTel builds, and Biome all pass.
The AWS Build and OTel Conformance workflows are currently awaiting maintainer approval before jobs can run.
jmsundar commentedon Sep 29, 2026
Related: this fix doesn't reach functions with X-Ray disabled and no caller header, because the
ParentLambda generates keeps them off this fallback path. The ask to support no-X-Ray deployments with minimal configuration is in #929 (comment).blackmore-technology-group commentedon Sep 30, 2026
PR #935 review update: I pushed
adfa3633d594ea74b30d351f31a4e668874b2954to address the reviewer blocker.The synthetic execution root is now ARN-scoped again, so two executions sharing one propagated trace cannot export conflicting spans with the same
(traceId, spanId). The regression now uses distinctcaller/calleeprovider resources and execution start times and verifies distinct execution-owned root identities.Local validation is green: targeted regressions, full OTel tests, core + OTel ESM/CJS/types builds, and Biome.
The X-Ray-disabled/no-caller-header path noted above remains separate and tracked through #929; this revision stays focused on #934.
fix(sdk): export stable synthetic roots before suspension (#966)