Skip to content

Fix gRPC client interceptor breaking bidirectional streaming (#1180)#4259

Merged
xrmx merged 9 commits intoopen-telemetry:mainfrom
juandelacruz-calvo:fix/grpc-bidi-stream-1180
Mar 19, 2026
Merged

Fix gRPC client interceptor breaking bidirectional streaming (#1180)#4259
xrmx merged 9 commits intoopen-telemetry:mainfrom
juandelacruz-calvo:fix/grpc-bidi-stream-1180

Conversation

@juandelacruz-calvo
Copy link
Copy Markdown
Contributor

Summary

Root Cause

In _client.py, intercept_stream() routed all RPCs with is_server_stream=True into _intercept_server_stream(), including bidirectional streams (where both is_client_stream and is_server_stream are True). _intercept_server_stream is a generator function (uses yield from), so calling it returns a bare Python generator that lacks the grpc.Call interface (add_done_callback, cancel, is_active, etc.).

Downstream code — notably google.api_core.bidi.BidiRpc.open() used by Google Cloud Pub/Sub — expects a real gRPC call object and crashes:

AttributeError: 'generator' object has no attribute 'add_done_callback'

The Fix

One condition change in intercept_stream():

# Before:
if client_info.is_server_stream:

# After:
if client_info.is_server_stream and not client_info.is_client_stream:

This ensures only unary-stream RPCs use the generator-based _intercept_server_stream. Bidi streams fall through to _intercept, which preserves the grpc.Call/grpc.Future interface.

RPC Type Before After
Unary-Stream _intercept_server_stream _intercept_server_stream (unchanged)
Stream-Unary _intercept _intercept (unchanged)
Bidi (Stream-Stream) _intercept_server_stream (broken) _intercept (fixed)

Files Changed

File Change
instrumentation/.../grpc/_client.py One-line condition fix
instrumentation/.../tests/test_client_interceptor.py Add test_stream_stream_preserves_call_interface
CHANGELOG.md Add bugfix entry under Unreleased/Fixed

Test plan

  • Existing test_stream_stream continues to pass (span correctness)
  • New test_stream_stream_preserves_call_interface passes — verifies grpc.Call attributes on bidi response
  • Existing unary-stream tests pass (no regression from condition change)
  • Existing stream-unary tests pass

Made with Cursor

…lemetry#1180)

Route bidi (stream-stream) RPCs through `_intercept` instead of the
generator-based `_intercept_server_stream`. The generator wrapper strips
the grpc.Call/grpc.Future interface, causing downstream code (e.g.
google.api_core.bidi.BidiRpc) to crash with:

    AttributeError: 'generator' object has no attribute 'add_done_callback'

The fix adds `and not client_info.is_client_stream` to the condition in
`intercept_stream()` so only unary-stream RPCs use the generator path.

Includes a regression test verifying the bidi stream response preserves
the grpc.Call interface (add_done_callback, cancel, is_active).

Co-authored-by: Cursor <[email protected]>
@juandelacruz-calvo
Copy link
Copy Markdown
Contributor Author

@xrmx

I have created an even simpler PR to fix the issues with PubSub that hopefully should be simpler to get approval for 🙏

@juandelacruz-calvo
Copy link
Copy Markdown
Contributor Author

juandelacruz-calvo commented Feb 25, 2026

For anyone needing a fix, this monkey patch should do:

    from opentelemetry.instrumentation.grpc._client import OpenTelemetryClientInterceptor

    _original_intercept_stream = OpenTelemetryClientInterceptor.intercept_stream

    def _patched_intercept_stream(self, request_or_iterator, metadata, client_info, invoker):
        if client_info.is_server_stream and client_info.is_client_stream:
            return self._intercept(request_or_iterator, metadata, client_info, invoker)
        return _original_intercept_stream(self, request_or_iterator, metadata, client_info, invoker)

    OpenTelemetryClientInterceptor.intercept_stream = _patched_intercept_stream

@tammy-baylis-swi
Copy link
Copy Markdown
Contributor

Thanks for contributing! Please could you make sure the existing tests all still pass or are adjusted. For example when tox -e py313-test-instrumentation-grpc-1 there are 2 failures at status code asserts.

Copy link
Copy Markdown
Contributor

@tammy-baylis-swi tammy-baylis-swi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, lgtm! We'll also need some other Approvers to have a look.

Copy link
Copy Markdown
Member

@lmolkova lmolkova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @juandelacruz-calvo !

I'm working on stabilizing OTel RPC semantic conventions (which involves a bunch of changes to this instrumentation in opt-in more).

Would you be interested in taking a look at the prototype #4279 ? Would really appreciate your thoughts.

@linux-foundation-easycla
Copy link
Copy Markdown

linux-foundation-easycla Bot commented Mar 11, 2026

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: juandelacruz-calvo / name: Juan Calvo Pozo (de627c4, e8139f6)

The gRPC status code is already captured in the RPC_GRPC_STATUS_CODE
attribute, making the description in span status unnecessary.
@juandelacruz-calvo juandelacruz-calvo force-pushed the fix/grpc-bidi-stream-1180 branch from 2f72933 to 82bffea Compare March 13, 2026 09:52
@juandelacruz-calvo juandelacruz-calvo marked this pull request as draft March 18, 2026 11:24
Verify that error spans include the correct gRPC status code attribute
(INVALID_ARGUMENT) in both sync and async client interceptor tests.
@juandelacruz-calvo juandelacruz-calvo force-pushed the fix/grpc-bidi-stream-1180 branch from c8c9a54 to e8139f6 Compare March 18, 2026 11:25
@juandelacruz-calvo juandelacruz-calvo marked this pull request as ready for review March 18, 2026 11:26
@xrmx xrmx merged commit bd01dcd into open-telemetry:main Mar 19, 2026
848 checks passed
@github-project-automation github-project-automation Bot moved this from Ready for review to Done in Python PR digest Mar 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants