[issues/14372] support ListIterator for Kafka consumer client#14707
Closed
patpatpat123 wants to merge 7 commits intoopen-telemetry:mainfrom
patpatpat123:main
Closed
[issues/14372] support ListIterator for Kafka consumer client#14707patpatpat123 wants to merge 7 commits intoopen-telemetry:mainfrom patpatpat123:main
patpatpat123 wants to merge 7 commits intoopen-telemetry:mainfrom
patpatpat123:main
Conversation
Contributor
|
🔧 The result from spotlessApply was committed to the PR branch. |
Contributor
Author
|
Test strategy: As of today, the traces are separated. There are two separate traces, one from the producer, one from the consumer, without a clear relationship. To test, I will create the Java agent containing this new code locally. From there, run the Spark Structured Streaming Kafka Consumer. Expected result: We will be able to see the producer-consumer relationship |
Contributor
Author
|
My bad... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello team,
This PR is to address issue #14372
Problem it is trying to solve:
As of today, Spark Structured Streaming Kafka Consumer, upon consuming a message with a valid W3C header with trace, is not continuing, nor propagating the trace on the Spark side.
Root cause analysis:
It seems Spark Structured Streaming Kafka Consumer is relying on
listIteratorC.f. https://github.com/apache/spark/blob/master/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala#L593As of now, Kafka instrumentation does not support
listIterator.Proposed solution
This solution will add
TracingListIterator<K, V> implements ListIterator<ConsumerRecord<K, V>>(TracingListIterator.java) which will allow the handling oflistIterator(instrumentation/kafka/kafka-clients/kafka-clients-common-0.11/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/common/v0_11/internal/TracingList.java )Thanks
Many thanks Lauri, Trask, and Jay for the help provided