otel: export stable synthetic root anchor during first fallback invocation #938

Activity

1. jmsundar commented on Oct 2, 2026

   

   jmsundar

   on Oct 2, 2026

   More actions

   Thanks for writing this up. The direction looks right to me. A few points to settle in the design, from reading the current plugin source and #935's head:

   1. Replace #935's terminal root rather than add a second export. #935 ends the root at terminal completion (syntheticRootSpan?.end(endTime) in InvocationOtelPlugin, .end() in ExecutionOtelPlugin), while the anchor here ends at executionStartTimestamp. If both ship, the same (traceId, spanId) is exported twice with different end times, which is the conflict #935 just fixed. Either the first-invocation anchor becomes the only materialization, or any later export emits the identical zero-duration shape.

   2. The anchor needs a stable timestamp source, not the wall-clock fallback. Both plugins resolve info.executionStartTimestamp ?? this.executionStartTimestamp ?? new Date(). When the platform doesn't supply a start timestamp, the anchor's start and end become the current time, so a retried first invocation exports a conflicting copy. I'd emit the anchor only when info.executionStartTimestamp is present, and otherwise fall back to terminal export.

   3. Pick, and document, how many times the anchor is exported. Two options with different failure modes:

   • First invocation only: if that invocation is killed before its flush (timeout, out of memory), the anchor is never exported, and with terminal export removed even a completed execution has no root.
   • First invocation, plus the identical anchor again at terminal completion: a completed execution always has its root, at the cost of a possible second copy. That copy matches in every span field, but the resource can still differ when the invocations ran in different execution environments, since per-environment resource attributes (for example faas.instance from a Lambda resource detector) aren't stable across them. Backends that deduplicate by span identity keep whichever copy arrives.

   I'd lean towards the second, since losing the root of a completed execution is worse than a duplicate zero-duration anchor, but either is fine if it's stated.

   4. The anchor and Workflow take their sampling decisions in different invocations. The decision is re-resolved on every invocation, so the anchor follows the first invocation's decision and Workflow follows the terminal one's. They agree for an explicit upstream decision, which is preserved across invocations, and for root policies that are a deterministic function of the trace ID, such as ratio-based samplers. A non-deterministic custom sampler could produce an anchor without a Workflow, or a Workflow whose parent never exports. Worth stating as a requirement in the docs.

   Additional regression cases:

   • the cases above in both plugin views (ExecutionOtelPlugin and InvocationOtelPlugin);
   • a first invocation with no executionStartTimestamp;
   • no two exported copies of the anchor that differ in any span field, across first-invocation retries and terminal completion.
2. blackmore-technology-group commented on Oct 4, 2026

   

   blackmore-technology-group

   on Oct 4, 2026

   ContributorAuthor

   More actions

   Thanks — these constraints make sense, and I agree they should be treated as acceptance criteria for this follow-up.

   The key invariant should be: the same (traceId, spanId) must never be exported with conflicting span fields. So this cannot be implemented as an additive early export on top of #935's terminal materialization.

   I also agree with the other points:

   • an early anchor needs a stable executionStartTimestamp; if that is unavailable, the terminal path should remain the fallback rather than using a wall-clock-derived anchor;
   • both ExecutionOtelPlugin and InvocationOtelPlugin need equivalent coverage;
   • retry/terminal paths must prove that no duplicate anchor differs in span content;
   • sampling behavior across invocations needs to be explicitly documented/tested rather than assumed.

   I’m leaving this issue as the design follow-up and would not consider a patch complete until those invariants are enforced by regression coverage.
3. zhongkechen

   closed this as completedin #966on Oct 9, 2026
4. zhongkechen

   added a commit that references this issue on Oct 9, 2026

   fix(sdk): export stable synthetic roots before suspension (#966)

   ...

   This issue will close once commit 4ce0ae5 is merged into the 'main' branch.

   4ce0ae5
5. github-actions

   added

   needs-triageIssue needs triageIssue needs triage

   on Oct 30, 2026