Skip to content

RUM-10140: Add BackPressured Dump information into Telemetry log#2673

Merged
ambushwork merged 1 commit into
developfrom
yl/add-back-pressued-dump
May 30, 2025
Merged

RUM-10140: Add BackPressured Dump information into Telemetry log#2673
ambushwork merged 1 commit into
developfrom
yl/add-back-pressued-dump

Conversation

@ambushwork

@ambushwork ambushwork commented May 26, 2025

Copy link
Copy Markdown
Member

What does this PR do?

More detailed context can be seen in the ticket.

This PR adds NamedRunnable wrapper class to contain custom name for each runnable, so that in BackPressuredBlockingQueue, the name can be retrieved if needed.
Also for Rum even processing, the operation name changes from "Rum event handling" to "Rum event handling StartAction" to specify the event type that rum-pipeline is handling.

Known issue:

When dumping the blocking queue we may not have the full dump result of the queue due to the concurrency, but my thought is that it's good enough to do the best effort of dumping instead of blocking the thread to have the perfect result.

Motivation

RUM-10140

Result of this PR

in backpressure.dump attribute, we will have the statistic of all the runnable names.

image

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Make sure you discussed the feature or bugfix with the maintaining team in an Issue
  • Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch 2 times, most recently from d794fd8 to dca06d8 Compare May 26, 2025 15:10
@ambushwork
ambushwork marked this pull request as ready for review May 26, 2025 15:18
@ambushwork
ambushwork requested review from a team as code owners May 26, 2025 15:18
@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch 4 times, most recently from 6ec1827 to a16ecd0 Compare May 27, 2025 09:10
@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch from a16ecd0 to 3a95dea Compare May 27, 2025 09:34
@ambushwork

Copy link
Copy Markdown
Member Author

The PR is updated with some major changes:

  • NamedRunnable is no more used in Executor.executeSafe() function to wrap every runnable from our SDK, because it will cause the Executor.remove fail if we want to remove the same instance of runnable. Instead, the caller of executeSafe need to wrap the runnable by itself, and provide the name. In this case, only rum-pipeline runnables are wrapped with it.
  • proguard-rules is added in order to keep the class name of RumRawEvent.

0xnm
0xnm previously approved these changes May 27, 2025

@0xnm 0xnm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! nothing blocking.

Comment on lines +25 to +27
override fun run() {
runnable.run()
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: can be a good use case for the delegate class NamedRunnable(name: String, private val runnable: Runnable): Runnable by runnable, then no need to declare it explicitly.

Comment thread detekt_custom.yml
- "java.util.concurrent.atomic.AtomicReference.set(io.opentracing.Tracer?)"
- "java.util.concurrent.atomic.AtomicReference.set(kotlin.Nothing?)"
- "java.util.concurrent.atomic.AtomicReference.set(kotlin.String?)"
- "java.util.concurrent.LinkedBlockingQueue.toArray()"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, super usage in the call is just for Detekt?

Comment thread features/dd-sdk-android-rum/consumer-rules.pro
@aleksandr-gringauz

aleksandr-gringauz commented May 27, 2025

Copy link
Copy Markdown
Contributor

I discussed it with Yi already, but will duplicate here for everyone else.

I have a concern that if the user calls methods of RumMonitor on main thread one after another and each of these calls leads to the Runnable being dropped due to backpressure, we might have a risk to cause an ANR.

We copy 1024 items from the queue, then compute some Map with statistics.

Especially taking into account the customer case related to backpressure I found this morning.

This needs to be checked, I am not entirely sure, but it looks a bit scary to me.

@0xnm

0xnm commented May 27, 2025

Copy link
Copy Markdown
Member

We copy 1024 items from the queue

Although the copy of the individual item should be cheap, it is just by reference. The main time-consumer here is the iteration over the items.

It can be another approach: maintain statistics at the offer / remove/pull calls and read it when needed, without looping though the items.

@aleksandr-gringauz

Copy link
Copy Markdown
Contributor

It can be another approach: maintain statistics at the offer / remove/pull calls and read it when needed, without looping though the items.

Looks good to me. Performance shouldn't change significantly, it doesn't look heavy. I didn't measure though, but looking at the code, it seems ok.

mariusc83
mariusc83 previously approved these changes May 27, 2025
@ambushwork

Copy link
Copy Markdown
Member Author

We copy 1024 items from the queue

Although the copy of the individual item should be cheap, it is just by reference. The main time-consumer here is the iteration over the items.

It can be another approach: maintain statistics at the offer / remove/pull calls and read it when needed, without looping though the items.

For me a problem of this is that we have to listen all the functions which might cause the size change of the LinkedBlockingQueue, for example we need to deal the exception with put, we have to deal with the result of offer which can be rejected, we have to deal with drainTo and removeAll, and also to handle them properly we need to take care of concurrency issue of each. It's like reinventing the wheel of LinkedBlockingQueue ourselves.

For me maybe copy the queue toArray at the call thread and delegate the map computation on background thread is easier and has less performance overhead?

@ambushwork
ambushwork dismissed stale reviews from mariusc83 and 0xnm via 68ba524 May 27, 2025 15:41
@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch from 3a95dea to 68ba524 Compare May 27, 2025 15:41
@ambushwork

Copy link
Copy Markdown
Member Author

Latest PR update summary:

  • Create ObservableBlockingQueue class to extract the dump logic with time interval (5s) protection, and add unit test for this.
  • BackPressuredBlockingQueue extends from ObservableBlockingQueue, calling dumpQueue function when threshold reached.

@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch from 68ba524 to 21dc2e7 Compare May 27, 2025 15:49
@codecov-commenter

codecov-commenter commented May 27, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.57895% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.00%. Comparing base (5376af1) to head (3709b4f).
⚠️ Report is 219 commits behind head on develop.

Files with missing lines Patch % Lines
...core/internal/thread/BackPressuredBlockingQueue.kt 55.56% 2 Missing and 2 partials ⚠️
...e/internal/thread/ObservableLinkedBlockingQueue.kt 83.33% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2673      +/-   ##
===========================================
+ Coverage    69.95%   70.00%   +0.05%     
===========================================
  Files          820      822       +2     
  Lines        30657    30687      +30     
  Branches      5162     5168       +6     
===========================================
+ Hits         21445    21481      +36     
+ Misses        7769     7763       -6     
  Partials      1443     1443              
Files with missing lines Coverage Δ
...m/datadog/android/internal/thread/NamedRunnable.kt 100.00% <100.00%> (ø)
.../android/rum/internal/monitor/DatadogRumMonitor.kt 85.14% <100.00%> (-0.49%) ⬇️
...e/internal/thread/ObservableLinkedBlockingQueue.kt 83.33% <83.33%> (ø)
...core/internal/thread/BackPressuredBlockingQueue.kt 90.00% <55.56%> (-5.45%) ⬇️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch 2 times, most recently from a1340fd to 7d272cb Compare May 28, 2025 13:15
@ambushwork
ambushwork requested a review from mariusc83 May 28, 2025 14:12
@ambushwork
ambushwork requested a review from 0xnm May 28, 2025 14:12
xgouchet
xgouchet previously approved these changes May 30, 2025
@ambushwork
ambushwork dismissed stale reviews from xgouchet and aleksandr-gringauz via bcdb1ef May 30, 2025 07:26
@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch from 7d272cb to bcdb1ef Compare May 30, 2025 07:26
@ambushwork
ambushwork requested a review from xgouchet May 30, 2025 07:50
0xnm
0xnm previously approved these changes May 30, 2025
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicLong

internal open class ObservableBlockingQueue<E : Any>(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor suggestion

Suggested change
internal open class ObservableBlockingQueue<E : Any>(
internal open class ObservableLinkedBlockingQueue<E : Any>(

Comment on lines +21 to +34
fun dumpQueue(): Map<String, Int>? {
val currentTime = currentTimeProvider.invoke()
val last = lastDumpTimestamp.get()
val timeSinceLastDump = currentTime - last
return if (timeSinceLastDump > DUMPING_TIME_INTERVAL_IN_MS) {
if (lastDumpTimestamp.compareAndSet(last, currentTime)) {
buildDumpMap()
} else {
null
}
} else {
null
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this logic should be on consumer side, because it creates some indeterministic behavior from the caller perspective, but I'm okay keeping it like that, because it is only for the internal use.

@ambushwork
ambushwork dismissed stale reviews from 0xnm and aleksandr-gringauz via 3709b4f May 30, 2025 08:38
@ambushwork
ambushwork force-pushed the yl/add-back-pressued-dump branch from bcdb1ef to 3709b4f Compare May 30, 2025 08:38
@ambushwork
ambushwork requested a review from 0xnm May 30, 2025 08:47
)
@MockitoSettings(strictness = Strictness.LENIENT)
@ForgeConfiguration(Configurator::class)
class ObservableBlockingQueueTest {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be also renamed to ObservableLinkedBlockingQueueTest, but to avoid more PR updates let's keep it like that, it doesn't affect anything really.

@ambushwork
ambushwork merged commit 0933d54 into develop May 30, 2025
@ambushwork
ambushwork deleted the yl/add-back-pressued-dump branch May 30, 2025 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants