ddtrace/tracer: create debug mode for old, open spans#2149
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
note: tests are very slow. future commit hopefully will fix
hannahkm
changed the title
katiehockman
left a comment
katiehockman
left a comment
There was a problem hiding this comment.
Thanks Hannah! Some first comments. I haven't finished reviewing it yet, but wanted to give a chance to look over these first.
Mostly, I'd like to understand the context around the data structure decision. Why a linked list of generic linked lists, instead of something similar (e.g. a slice of span pointers)?
instead, we print all abandoned spans on tracer stop
Re the above: The main concern I had with creating the new data structure in the tracer was the overhead we'd introduce from adding/removing spans as they are started/finished. A linked list is great for this purpose (especially when removing spans from the middle of the structure), and the outer linked list is for the purposes of bucketing spans together. Spans that have a similar start time are placed in the same "bucket" so the new goroutine hopefully has less to scan through when checking for abandoned spans! |
ajgajg1134
left a comment
ajgajg1134
left a comment
There was a problem hiding this comment.
Looks pretty good! A few questions / changes!
ajgajg1134
left a comment
ajgajg1134
left a comment
There was a problem hiding this comment.
Thanks for being on the team!
This reverts commit 283c0ac.
4 participants
What does this PR do?
This PR introduces a debugging mode that the user can configure for finding and logging old, unfinished spans. Traces/Spans older than some configured
Xamount of time will be printed to the debug logs. When the tracer stops, it will also print all remaining open spans. For example, the debug mode can output log lines like the following:2023/08/14 16:55:45 Datadog Tracer v1.55.0-dev WARN: 4 abandoned spans:2023/08/14 17:15:38 Datadog Tracer v1.55.0-dev WARN: [name: newSpan, span_id: 7578134955455927291, trace_id: 7578134955455927291, age: 84 sec],[name: http.request, span_id: 5293323209049443106, trace_id: 5293323209049443106, age: 84 sec],[name: newSpan, span_id: 284037345165039393, trace_id: 284037345165039393, age: 84 sec],[name: http.request, span_id: 1770903217116959580, trace_id: 1770903217116959580, age: 84 sec],Motivation
Traces with unfinished spans may never finish but never error, which causes confusion when data goes missing.
Fixes #2064
Describe how to test/QA your changes
New and old unit tests should pass. For manual testing, turn on the debug mode with
WithDebugSpansMode(true)or by setting enviornment variableDD_TRACE_DEBUG_OPEN_SPANS=true. Create spans older than the configured amount of time (default value: 1 second, configure using environment variableDD_TRACE_OPEN_SPAN_TIMEOUT) and check that they are printed out in the tracer logs.Call
defer tracer.PrintOpenSpans()to check that a list of open spans is logged when the application ends.Reviewer's Checklist