Overview

This PR adds support for the long running spans feature, which enables periodic snapshots of running spans to be written. It also includes changes to the PendingTrace and PendingTraceBuffer classes to track long running traces and trigger periodic writes.

Long running spans spec

Long running spans are tracked

• priority > 0
• while they last less then 12H

Protocol

• write the span with all tags
• span.Duration = write time
• if running span.Metric["_dd.partial_version"] is set. The highest version is the most recent one
• if previously running a complete span will have span.Metric["_dd.was_longrunning"] = 1

Configuration

• feature is disabled by default
• periodic flushes in [20s, 7.5min] with a default of 5min

Implementation overhead

Feature disabled

• additional if statement at span initialisation and termination with pendingTraceBuffer.runningSpansEnabled()
• no-op on the rest

Feature enabled

all traces

• PendingTrace keeps track of all spans including running ones
• On write we iterate on all spans including running ones
• PendingTrace is referenced up to 1 second
• One insertion to the PendingTraceBuffer queue per trace
• Running spans are also written if the trace lasts longer then the configured runningFlushInterval (default 5 mins, hard coded min 20s)

other

PendingTraceBuffer is waked up every 1s

Breakdown of the required:

• track running spans
• periodically trigger writes
• encode new tags on spans
• avoid computing stats on running spans

Track running spans (PendingTrace)

Writes are triggered by PendingTraces and the PendingTraceBuffer. They only point to completed spans.

current links:
span ---> pendingTrace ---> completedSpans
                     |
                     ------->  trigger writes

New approach when the feature is enabled the pendingTrace will track all spans including runningSpans.
To avoid additional allocations all spans are stored in the pendingTrace.spans list.

new links
span -> pendingTrace -> allSpans (running, completed)
                     |
                     ------->  trigger writes

Periodic write triggers (PendingTraceBuffer)

The PendingTraceBuffer is an existing thread that can trigger partial writes. It listens to a queue where PendingTraces that meet certain conditions are inserted.
This makes it the perfect place to track long running traces and trigger periodic traces.

A 1 second periodic wakeup is added to the PendingTraceBuffer thread. On wakeup runningTraces are re-evaluated: a write can be triggered, and traces already completed are cleaned up.

The runningTraces are stored in a dedicated array LongRunningTracesTracker with a few helpers.

A LongRunningState has been added to the PendingTrace to ensure that the changes of state happen only once. Exemples

• PendingTrace is sent only once to the PendingTraceBuffer to be tracked
• priority is evaluated only once
• the LongRunningTracker does not contain duplicate PendingTrace

PendingTrace state changes

                                  priority <= 0
                    UNDEFINED  --------------------------> NOT_TRACKED
                         |
                         |   priority > 0
                         v
                    TO_TRACK 
                         |
                         |   added to the LongRunningTraceTracker by PendingTraceBuffer
                         v
                    TRACKED 
                         |
                         |   agent feature enabled AND trace is older than flush period
                         v
                    WRITE_RUNNING   //writes will now write running spans
                         |
                         |   trace longer then 12h  
                         v
                    EXPIRED
                      

Encode new tags on spans

The 4 following constraints made this particularly challenging:

• a public CoreSpan interface is passed through tracer write logic and we must not add methods on the public interface
• span.Duration == 0 is used in the tracer to know if a span is finished
• long running span tags should be added in the transport layer
• spans should not be cloned

The proposed solution to meet the constraint is:

1. add a version on DDSpan and pass it through the MetadataConsumer. The MetadataConsumer is used in the v04 and v05 encoders to set tags.
2. fetch the write time from the PendingTrace if duration == 0 at write time. This unfortunately requires to cast CoreSpan to DDSpan. Check getDurationNano(CoreSpan<?> span. Given that the flushed running spans volume is low, this is acceptable.

Avoid computing stats

A helper shouldComputeMetrics is added and running spans are filtered out by checking the span duratio. Note span.isFinished() is not available on the CoresSpan public interface, that's why span.getDuration() == 0 is used.