Skip to content

[BUG] gRPC CLIENT spans are incorrectly suppressed in async callbacks when outer CLIENT span has ended but SpanKey remains in Context #16093

@huange7

Description

@huange7

Describe the bug

When using gRPC streaming calls with async task submission in StreamObserver.onNext() callback, new gRPC CLIENT spans initiated from the async task are incorrectly suppressed, even though the outer gRPC CLIENT span has already ended.

Steps to reproduce

  1. Service A receives a gRPC request (SERVER span created)
  2. Service A makes a streaming gRPC call to Service B (CLIENT span 1 created)
  3. In StreamObserver.onNext() callback, an async task is submitted to a thread pool
  4. The executor instrumentation captures current Context (which contains CLIENT span 1's SpanKey)
  5. CLIENT span 1 ends (stream completes)
  6. Async task executes, Context is restored (still contains the ended CLIENT span 1)
  7. Async task makes a new gRPC call to Service C
  8. New CLIENT span 2 is suppressed ❌ (because SpanKey for rpc-client exists in Context)

Expected behavior

The new gRPC CLIENT span should be created normally since the outer CLIENT span has already ended.

Actual behavior

The new gRPC CLIENT span is suppressed because SpanSuppressors$BySpanKey.shouldSuppress() returns true

Javaagent or library instrumentation version

2.24.0

Environment

JDK: 17
OS: mac

Additional context

No response

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds triageNew issue that requires triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions