-
Notifications
You must be signed in to change notification settings - Fork 113
OpenTelemetry spans being ended multiple times causing "Operation attempted on ended Span" errors #2320
Description
Please make sure you have searched for information in the following guides.
- Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
- Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
- Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
- Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
- Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
- Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
- Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples
A screenshot that you have tested with "Try this API".
Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.
A step-by-step description of how to reproduce the issue, based on the linked reproduction.
Environment
- @google-cloud/spanner version: 8.x.x (latest)
- Node.js version: 20.x
- express": "^5.1.0"
- OpenTelemetry SDK version:
@opentelemetry/api: ^1.9.0@opentelemetry/sdk-node: ^0.202.0@opentelemetry/auto-instrumentations-node: ^0.60.0
A complete sample application demonstrating this bug is available at:
https://github.com/hsojod/otel-spanner-sample
To reproduce the issue:
-
Clone and run the sample app
-
Make a test request to trigger the bug:
curlhttp://localhost:8080/ -
Inspect the logs:
The application outputs OpenTelemetry traces to the console. You will see the span lifecycle errors mixed in with the trace output, specifically the "Operation attempted on ended Span" errors described above.
The sample application is configured with the OpenTelemetry NodeSDK including auto-instrumentation and demonstrates the bug occurring during basic Spanner operations. Each request generates traces showing the problematic span management within the Spanner client library.
A clear and concise description of what the bug is, and what you expected to happen.
Bug Description
The Google Cloud Spanner Node.js client library has OpenTelemetry instrumentation that attempts to operate on spans that have already been ended, resulting in runtime errors. This appears to be a span lifecycle management issue within the library's tracing implementation.
Error Messages
Cannot execute the operation on ended Span {traceId: 052e2003b73b69c6aca3083d4be34ca0, spanId: 35d90b5b849811f5} Error: Operation attempted on ended Span {traceId: 052e2003b73b69c6aca3083d4be34ca0, spanId: 35d90b5b849811f5}
at SpanImpl._isSpanEnded (node_modules/@opentelemetry/sdk-trace-base/build/src/Span.js:265:27)
at SpanImpl.addEvent (node_modules/@opentelemetry/sdk-trace-base/build/src/Span.js:118:18)
at Snapshot._update (node_modules/@google-cloud/spanner/build/src/transaction.js:986:14)
at PartialResultStream.<anonymous> (node_modules/@google-cloud/spanner/build/src/transaction.js:822:26)
Cannot execute the operation on ended Span {traceId: 052e2003b73b69c6aca3083d4be34ca0, spanId: 4693552b6a7918ee} Error: Operation attempted on ended Span {traceId: 052e2003b73b69c6aca3083d4be34ca0, spanId: 4693552b6a7918ee}
at SpanImpl._isSpanEnded (node_modules/@opentelemetry/sdk-trace-base/build/src/Span.js:265:27)
at SpanImpl.end (node_modules/@opentelemetry/sdk-trace-base/build/src/Span.js:178:18)
at Transform.<anonymous> (node_modules/@google-cloud/spanner/build/src/database.js:1204:22)
Cannot execute the operation on ended Span {traceId: 052e2003b73b69c6aca3083d4be34ca0, spanId: d3c5d5fe4b9e23e4} Error: Operation attempted on ended Span {traceId: 052e2003b73b69c6aca3083d4be34ca0, spanId: d3c5d5fe4b9e23e4}
at SpanImpl._isSpanEnded (node_modules/@opentelemetry/sdk-trace-base/build/src/Span.js:265:27)
at SpanImpl.end (node_modules/@opentelemetry/sdk-trace-base/build/src/Span.js:178:18)
at PartialResultStream.<anonymous> (node_modules/@google-cloud/spanner/build/src/transaction.js:853:26)
Additional warnings:
CloudSpanner.Database.runStream 052e2003b73b69c6aca3083d4be34ca0-4693552b6a7918ee - You can only call end() on a span once.
CloudSpanner.Snapshot.runStream 052e2003b73b69c6aca3083d4be34ca0-d3c5d5fe4b9e23e4 - You can only call end() on a span once.
A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **
Expected Behavior
OpenTelemetry spans should be properly managed with:
- Spans only ended once
- No operations attempted on ended spans
- Proper span lifecycle management across async operations and streams
- Provide configuration options: Add client configuration options to disable built-in tracing:
