Skip to content

Workflow: continue_as_new should reset trace context to prevent unbounded traces #10064

Closed

Description

@ayackel

In what area(s)?

/area runtime
/area workflows

What version of Dapr?

1.17.x (msft fork: daprd 1.17.2-msft.1)

Expected Behavior

When a workflow calls continue_as_new, the new orchestration iteration should start with a fresh trace ID. Each iteration of an eternal/polling workflow should produce an independent trace, allowing distributed tracing backends (Geneva, Azure Monitor, Jaeger, etc.) to display bounded, queryable traces per iteration.

Actual Behavior

The daprd sidecar propagates the same trace context across continue_as_new boundaries. All iterations of an eternal workflow share a single trace ID indefinitely. Over hours/days, this produces a single unbounded trace containing thousands of spans — making it:

  1. Unusable in tracing UIs — the waterfall view becomes impossibly long
  2. Unqueryable — you cannot filter to "what happened in this specific polling cycle"
  3. A potential storage/performance concern — trace backends may not expect traces spanning days/weeks

Evidence

  • OrchestratorRequest protobuf has no parentTraceContext field — trace context propagates via gRPC metadata from the sidecar
  • The Python durabletask worker cannot override this — it receives the trace context, it does not create it
  • Activities receive parentTraceContext from ActivityRequest proto, which inherits from the same long-lived trace
  • Screenshot from Geneva showing a single trace spanning the full init_customer_workflows orchestration (5+ minute iterations, same trace_id across all):
Trace ID: 527ee9323b3743020b2f045ce4dde4f0
├── orchestration|init_customer_workflows    5.0 mins
│   ├── activity|init_list_cell_namespaces   187 ms
│   ├── activity|update_start_time           361 ms
│   ├── activity|update_start_time           439 ms
│   └── activity|fetch_report_ids            1.3 s
│       └── ddms.request                     1.3 s  ← our custom span
│           ├── POST (200)                   378 ms
│           └── POST (200)                   902 ms
└── (next iteration reuses same trace_id...)

Proposed Solution

In durabletask-go, when processing a ORCHESTRATION_STATUS_CONTINUED_AS_NEW completion:

  1. Generate a new trace ID for the re-dispatched orchestration instance
  2. Optionally, add a link from the new trace to the previous iteration's trace (preserving correlation without parent-child)
  3. Set the new traceParent in gRPC metadata when dispatching the next OrchestratorRequest

This would give each iteration its own trace while still allowing backends that support trace links to navigate between iterations.

Workaround

Currently adding custom span attributes (iteration_start timestamp) to child spans so we can filter within the mega-trace, but this does not solve the unbounded trace growth problem.

Steps to Reproduce the Problem

  1. Create an eternal workflow using ctx.continue_as_new(...)
  2. Enable distributed tracing (OTLP exporter to any backend)
  3. Let the workflow run 2+ iterations
  4. Observe that all iterations share the same trace_id

Related Issues

Release Note

RELEASE NOTE: ADD Workflow continue_as_new resets trace context, producing bounded per-iteration traces.

Activity

  1. SarthakB11 commented on Jul 1, 2026

    @SarthakB11

    Hi @ayackel, took a look at where the trace context propagates. The carryover happens in dapr/durabletask-go, not in this repo.

    On CONTINUE_AS_NEW, backend/runtimestate/applier.go (in the Actions CAN branch) builds the new generation's ExecutionStartedEvent and copies the old context directly:

    ParentTraceContext: s.StartEvent.ParentTraceContext,

    Each run parents its span off that field, so every generation re-joins the original trace. That's the unbounded growth you're seeing.

    Fix shape, in durabletask-go: mint a fresh root trace context for the new generation instead of copying the old one. Two options:

    1. reset-only (bounded per-iteration traces)
    2. reset + an OTel span link back to the prior iteration (your preferred behavior; needs a small link-carrying addition)

    Before I open a PR: is durabletask-go the repo you want this in, and do you prefer reset-only or reset-with-link?

  2. ayackel commented on Jul 1, 2026

    @ayackel
    Author

    Reset-only is fine with me. I don't know that anyone would optionally want unbounded traces .

  3. SarthakB11 commented on Jul 4, 2026

    @SarthakB11

    Opened dapr/durabletask-go#113 (reset-only, per your ack). Each generation now gets a fresh W3C root TraceContext when the workflow was traced, and stays untraced when it was not. Existing Test_ContinueAsNew end-to-end still green; added a unit test in tests/runtimestate_test.go for both branches.

  4. added a commit that references this issue on Aug 4, 2026
    56dd69e
  5. added 2 commits that reference this issue on Aug 4, 2026
    ea9dfba
    c0a2913
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions