What does this PR do?

• Issue Identified:

  • Our storage logic has a race condition issue affecting event processing.
  • This problem is visible only in a specific use case involving concurrent operations and tracking consent management.

• Specific Use Case:

  • The system is operating under "PENDING" tracking consent mode.
  • Events are being written via a featureScope by one thread.
  • At the same time, tracking consent is being changed by another thread.

• Underlying Process:

  • For each event written:
    • The system first resolves the DatadogContext to determine the current environment and configuration.
    • It then uses the DatadogContext to resolve the FileOrchestrator, which decides where events should be stored.
    • Subsequently, a task is added to the Executor queue to handle the actual writing of the event.

override fun writeCurrentBatch(
       datadogContext: DatadogContext,
       forceNewBatch: Boolean,
       callback: (EventBatchWriter) -> Unit
   ) {
       val orchestrator = when (datadogContext.trackingConsent) {
           TrackingConsent.GRANTED -> grantedOrchestrator
           TrackingConsent.PENDING -> pendingOrchestrator
           TrackingConsent.NOT_GRANTED -> null
       }

       val metric = internalLogger.startPerformanceMeasure(
           callerClass = ConsentAwareStorage::class.java.name,
           metric = TelemetryMetricType.MethodCalled,
           samplingRate = MethodCallSamplingRate.RARE.rate,
           operationName = "writeCurrentBatch[${orchestrator?.getRootDirName()}]"
       )

       executorService.submitSafe("Data write", internalLogger) {
           synchronized(writeLock) {
               val batchFile = orchestrator?.getWritableFile(forceNewBatch)
               val metadataFile = if (batchFile != null) {
                   orchestrator.getMetadataFile(batchFile)
                   ...

• Race Condition Trigger:
  • During the execution of setTrackingConsent(Granted):
    • A new task is created to migrate events from the "PENDING" to the "GRANTED" consent state.
    • This task is added to the Executor queue.
  • The race condition occurs when this migration task enters the queue between the resolution of the FileOrchestrator and the addition of the event-writing task.

journey
    title Storge Executor Queue
    section Write Tasks
        event1InPendign: Write Event 1
        event2InPending: Write Event 2
    section SwitchConsent Tasks
        PendingToGranted: Switch Consent
    section Write Tasks
        event3InPending: Write Event 3
        event4InGranted: Write Event 4

• Resulting Issue:

  • The events, initially deemed "PENDING," get stored in the "PENDING" directory.
  • Due to the timing of the migration task:
    • These events may fail to migrate to the "GRANTED" state.
    • Consequently, they remain in the "PENDING" folder indefinitely.
  • This results in the loss of events, as they are not properly sent or processed.

• Implications:

  • This issue could lead to incomplete data capture and analysis.
  • It may also affect downstream processes that rely on accurate and complete event data.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Make sure you discussed the feature or bugfix with the maintaining team in an Issue
• Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)