What does this PR do?

Eliminates race conditions in the tracer's write pipeline by implementing comprehensive thread-safe interfaces and fixing a critical deadlock bug that was introduced during the initial race condition fix.

Motivation

During analysis of the tracer's writer implementation, multiple race conditions were discovered that could cause serious issues under high concurrency:

Original Race Conditions:

• Payload buffer races in agentTraceWriter between add() and flush() operations
• Data races in bytes.Buffer operations detected by Go's race detector
• Buffer corruption causing panics like slice bounds out of range
• Lost or corrupted traces when payload operations interfere with each other

Solution

Phase 1: Fix Writer Race Conditions

• Added mu sync.Mutex to agentTraceWriter struct
• Protected add() operations with mutex to prevent concurrent payload access
• Protected payload swap in flush() to ensure atomic replacement

Phase 2: Comprehensive Payload Interface Redesign

• Added private payload interfaces (payloadWriter, payloadReader, payload)
• Implemented safePayload wrapper with RWMutex for thread-safe operations
• Renamed payload to unsafePayload, replaced with safe interface throughout codebase
• Optimized itemCount() using direct atomic access for performance
• Renamed incrementCount() to recordItem() for clarity
• Extended thread safety to all payload operations across the entire codebase

Key Fixes

1. Deadlock Resolution: Fixed recursive locking in agentTraceWriter.add()
2. Thread-Safe Payload Interface: All payload operations now race-condition free
3. Performance Optimizations: Efficient atomic operations where appropriate
4. Comprehensive Testing: Added extensive concurrent access tests

Files Changed

Core Writer Components:

• ddtrace/tracer/writer.go - Fixed deadlock, added mutex synchronization
• ddtrace/tracer/payload.go - Thread-safe interface redesign
• ddtrace/tracer/transport.go - Updated to use safe payload interface

CI Visibility Components:

• ddtrace/tracer/civisibility_*.go - Updated for thread-safe payload usage

Testing & Configuration:

• ddtrace/tracer/*_test.go - Added comprehensive race condition tests
• ddtrace/tracer/option.go - Updated payload initialization

Testing

** Race Condition Tests:**

• TestAgentWriterRaceCondition: Spawns multiple goroutines doing concurrent add/flush operations
• TestAgentWriterTraceCountAccuracy: Validates trace counting accuracy under concurrency
• TestWorker: Now passes (was timing out due to deadlock)
• Payload concurrency tests: Comprehensive testing of new thread-safe interfaces

Verification:

• Tests consistently fail with race detector on unfixed code (including panics and deadlocks)
• Tests consistently pass with no race detector warnings after all fixes
• All existing tests pass with no regressions
• Performance maintained with optimized atomic operations where possible

Backward Compatibility

• No breaking API changes - all changes are internal implementation details
• All existing functionality preserved
• Performance characteristics maintained or improved

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• New code is free of linting errors. You can check this by running ./scripts/lint.sh locally.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.