Fix gRPC client interceptor breaking bidirectional streaming (#1180)#4259
Conversation
…lemetry#1180) Route bidi (stream-stream) RPCs through `_intercept` instead of the generator-based `_intercept_server_stream`. The generator wrapper strips the grpc.Call/grpc.Future interface, causing downstream code (e.g. google.api_core.bidi.BidiRpc) to crash with: AttributeError: 'generator' object has no attribute 'add_done_callback' The fix adds `and not client_info.is_client_stream` to the condition in `intercept_stream()` so only unary-stream RPCs use the generator path. Includes a regression test verifying the bidi stream response preserves the grpc.Call interface (add_done_callback, cancel, is_active). Co-authored-by: Cursor <[email protected]>
|
I have created an even simpler PR to fix the issues with PubSub that hopefully should be simpler to get approval for 🙏 |
|
For anyone needing a fix, this monkey patch should do: |
|
Thanks for contributing! Please could you make sure the existing tests all still pass or are adjusted. For example when |
tammy-baylis-swi
left a comment
There was a problem hiding this comment.
Thanks, lgtm! We'll also need some other Approvers to have a look.
lmolkova
left a comment
There was a problem hiding this comment.
LGTM @juandelacruz-calvo !
I'm working on stabilizing OTel RPC semantic conventions (which involves a bunch of changes to this instrumentation in opt-in more).
Would you be interested in taking a look at the prototype #4279 ? Would really appreciate your thoughts.
The gRPC status code is already captured in the RPC_GRPC_STATUS_CODE attribute, making the description in span status unnecessary.
2f72933 to
82bffea
Compare
Verify that error spans include the correct gRPC status code attribute (INVALID_ARGUMENT) in both sync and async client interceptor tests.
c8c9a54 to
e8139f6
Compare
Summary
AttributeError: 'generator' object has no attribute 'add_done_callback'by routing bidi streams through_interceptinstead of the generator-based_intercept_server_stream(issue Thread-ConsumeBidirectionalStream caught unexpected exception 'generator' object has no attribute 'add_done_callback' and will exit. #1180)test_stream_stream_preserves_call_interfaceverifying the bidi stream response preserves thegrpc.Callinterface (add_done_callback,cancel,is_active)Root Cause
In
_client.py,intercept_stream()routed all RPCs withis_server_stream=Trueinto_intercept_server_stream(), including bidirectional streams (where bothis_client_streamandis_server_streamareTrue)._intercept_server_streamis a generator function (usesyield from), so calling it returns a bare Python generator that lacks thegrpc.Callinterface (add_done_callback,cancel,is_active, etc.).Downstream code — notably
google.api_core.bidi.BidiRpc.open()used by Google Cloud Pub/Sub — expects a real gRPC call object and crashes:The Fix
One condition change in
intercept_stream():This ensures only unary-stream RPCs use the generator-based
_intercept_server_stream. Bidi streams fall through to_intercept, which preserves thegrpc.Call/grpc.Futureinterface._intercept_server_stream_intercept_server_stream(unchanged)_intercept_intercept(unchanged)_intercept_server_stream(broken)_intercept(fixed)Files Changed
instrumentation/.../grpc/_client.pyinstrumentation/.../tests/test_client_interceptor.pytest_stream_stream_preserves_call_interfaceCHANGELOG.mdTest plan
test_stream_streamcontinues to pass (span correctness)test_stream_stream_preserves_call_interfacepasses — verifiesgrpc.Callattributes on bidi responseMade with Cursor