Merge feature/continuous-profiling into develop#3622
Conversation
…us-sample-rate-api RUM-15189: Add setContinuousSampleRate API to ProfilingConfiguration
…s-profiling-250326
…s-profiling-250326
…-continuous-profiling-250326 Merge `develop` branch into `feature/continuous-profiling` branch
…aunch-duration RUM-15190: Extend app launch profiling duration for continuous merge
…event-to-profiling Send event to Profiling feature when RUM session is renewed
…us-profiling-scheduler RUM-15191: Implement ContinuousProfilingScheduler
…inuous-profiling RUM-15323: RUM sends ANR & LongTask events to Profiling Feature
…lback RUM-15307: Report failure through ProfilerCallback
…e-profiling-rum-context RUM-15334: Commonize ProfilingRumContext for TTID and Continuous Profiling
…ing-rum-event RUM-15334: Add id to RumAnrEvent and RumLongTaskEvent
…filing-cycle Fix continuous profiling back-to-back with app launch by inserting a cooldown window
…-to-scheduler RUM-15321: Continuous profiling writes are gated on pending RUM events
…s-profiling-030426
…s-profiling-030426
…-continuous-profiling-030426 Merge `develop` branch into `feature/continuous-profiling`
…ndicator RUM-15454: Add Profiler status in RUM debug widget
RUM-16394 Pt.3: Wire Quota Checker with Profiling Feature
…ontext-to-view-events RUM-16696: Attach profiling context to the View events
…rift RUM-16713: Gate profiles with big system clock drift
…ranch Merge develop into feature/continuous-profiling
Add write result telemetry to ProfilingDataWriter
Attach RUM Operation events to profiling
…rofiling Merge develop into feature/continuous-profiling
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0407de65dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sdkCore.getFeature(Feature.PROFILING_FEATURE_NAME)?.sendEvent( | ||
| ProfilerStopEvent.TTID( | ||
| rumContext = TTIDRumContext( | ||
| // startMs is different for different scenarios. For Cold it will be process start, for Warm | ||
| // it will be Activity.onCreate. But we have application launch profiling working for cold starts only anyway. | ||
| ProfilerEvent.RumVitalEvent( |
There was a problem hiding this comment.
Validate TTID before publishing it to profiling
For TTID values over MAX_TTID_DURATION_NS, sendTTIDEvent later drops the RUM vital, but this profiling event has already been queued with the vital id. In that scenario the uploaded profile can reference a TTID vital that was never written to RUM, breaking profile/RUM correlation; perform the same duration validation before sending the profiler event, or only send it after the RUM event is accepted for writing.
Useful? React with 👍 / 👎.
| override fun onAnrDetected(event: ProfilingAnrDetectedEvent) { | ||
| // The ANR event should be forwarded to RUM only when profiling is actually running. | ||
| if (isLaunchProfilingActive || continuousProfilingScheduler?.isActive == true) { | ||
| sdkCore.getFeature(Feature.RUM_FEATURE_NAME)?.sendEvent(event) |
There was a problem hiding this comment.
Honor RUM's non-fatal ANR opt-out
On API 30+ trackNonFatalAnrs is disabled by default (and users can explicitly disable it), but this forwarding path sends every profiling-trigger ANR to the RUM feature whenever a profile is active. RumFeature handles ProfilingAnrDetectedEvent by calling addError unconditionally, bypassing the existing configuration.trackNonFatalAnrs guard used to start the watchdog detector, so enabling profiling can report non-fatal ANR errors for apps that opted out. Gate this event on the RUM ANR setting or carry the setting into this path before sending.
Useful? React with 👍 / 👎.
| if (quotaResult.decision == QuotaResult.Decision.DENIED) { | ||
| logToUser( | ||
| LOG_LAUNCH_PROFILING_DROPPED_QUOTA_DENIED.format( | ||
| Locale.US, | ||
| quotaResult.reason.rawValue | ||
| ) | ||
| ) | ||
| pendingRumEvents.clear() |
There was a problem hiding this comment.
Delete traces dropped by quota
When the launch profile is denied by quota, this branch consumes perfettoResult and never passes result.resultFilePath to ProfilingDataWriter, which is currently the only path that deletes the Perfetto file. Every quota-denied launch profile therefore leaves its trace file behind even though it will never be uploaded; delete the result file in the denied branch before clearing the buffered state.
Useful? React with 👍 / 👎.
| val sampleRate = (context[FeatureContextKeys.RUM_SESSION_SAMPLE_RATE] as? Number)?.toFloat() | ||
| ?: DEFAULT_RUM_SESSION_SAMPLE_RATE | ||
| lastSeenRumSessionId = sessionId |
There was a problem hiding this comment.
Skip sessions RUM did not track
This renewal path samples profiling from the RUM session id and sample rate but ignores the RUM session_state that is also present in the feature context. For a RUM session that sampled out (NOT_TRACKED) while the configured sample rate is still non-zero, profiling can mark the session as sampled, extend the launch profile, and start continuous windows even though RUM is writing through a no-op writer, producing profiles that either have no usable RUM events or reference events RUM never ingested. Check that the RUM session is actually tracked before running quota checks or renewing the profiling scheduler.
Useful? React with 👍 / 👎.
0407de6 to
1371e7d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1371e7d579
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| runnable = { | ||
| val cooldownMs = jitter(CONTINUOUS_COOLDOWN_DURATION_MS) | ||
| logToUser { LOG_ACTIVE_WINDOW_ENDED.format(Locale.US, cooldownMs) } | ||
| scheduleCooldown(cooldownMs) |
There was a problem hiding this comment.
Stop buffering when the active window expires
When a continuous window reaches this scheduled end, the scheduler moves into cooldown but leaves isActive true until the Perfetto result callback later calls onActiveWindowEnded(). If that callback is delayed, ProfilingFeature.isRecordingProfile() continues accepting long tasks, ANRs, and vitals that occurred after the trace interval, so the completed profile can be uploaded with out-of-window RUM metadata; clear the active flag here before entering cooldown, or gate buffering on the actual profiler state.
Useful? React with 👍 / 👎.
| eventType = EventType.DEFAULT | ||
| ) | ||
| } | ||
| safeDelete(profilingResult.resultFilePath) |
There was a problem hiding this comment.
Defer deleting shared traces until all instances read them
When launch profiling was started for multiple SDK instances, PerfettoProfiler fans the same resultFilePath out to every registered callback, but the first ProfilingDataWriter that runs deletes that shared file here. The remaining instance callbacks then try to read a missing file and drop their profile, so only one SDK instance can upload the trace; delete the file only after all callbacks have copied it, or share the bytes before any writer deletes the path.
Useful? React with 👍 / 👎.
| val anrDurationNs = TimeUnit.MILLISECONDS.toNanos(anrDurationMs) | ||
|
|
||
| profilingStatus = resolveErrorProfilingStatus(datadogContext) | ||
| sdkCore.getFeature(Feature.PROFILING_FEATURE_NAME)?.sendEvent( |
There was a problem hiding this comment.
Publish ANR metadata after the error is written
For a non-fatal ANR, this publishes the profiling metadata before the RUM ErrorEvent write operation runs. If the app's errorEventMapper drops the non-crash error or the write fails, the uploaded profile can still reference this error id even though RUM never ingested it; move the profiling send into the write operation's success path using the captured context and id.
Useful? React with 👍 / 👎.
| ) | ||
|
|
||
| val longTaskId = UUID.randomUUID().toString() | ||
| sdkCore.getFeature(Feature.PROFILING_FEATURE_NAME)?.sendEvent( |
There was a problem hiding this comment.
Publish long-task metadata after successful writes
This sends the long-task id to profiling before the corresponding RUM LongTaskEvent is actually accepted by the writer. When a longTaskEventMapper drops the event or storage write fails, the profile can still be uploaded with a long-task id that does not exist in RUM; use the write operation's success callback, as the operation-vital path does, before buffering the profiling metadata.
Useful? React with 👍 / 👎.
| } | ||
| tagsProfiler = buildTags(context, profilingResult.startReason.value), | ||
| application = ProfileEvent.Application(id = rumContext.applicationId), | ||
| session = ProfileEvent.Session(id = rumContext.sessionId), |
There was a problem hiding this comment.
Split profile metadata by RUM session
A continuous window can overlap a RUM session renewal or explicit session reset, but the pending buffer is not drained on renewal and this writes a single profile session.id from the first buffered event while still adding all long-task/error/vital ids from the whole buffer. In that scenario the uploaded profile references events from the later session under the earlier session id, so correlation for those events is wrong; split or drop buffered metadata when the RUM session changes.
Useful? React with 👍 / 👎.
| private fun onTtidEvent() { | ||
| if (isTtidVitalReceived.getAndSet(true)) return | ||
|
|
||
| if (continuousProfilingScheduler?.currentSessionSampled != true) { |
There was a problem hiding this comment.
Stop launch traces once quota denies them
When the first RUM session is sampled for continuous profiling, this TTID path does not stop the launch trace, even if the quota result is already denied or becomes denied just after TTID. The trace then keeps running for the extended launch window only to be dropped in tryWriteProfilingEvent, which wastes profiling overhead for users whose org/session is quota-denied; include the quota decision in this stop condition or stop when a denial arrives.
Useful? React with 👍 / 👎.
| } | ||
| ) | ||
|
|
||
| sdkCore.getFeature(Feature.PROFILING_FEATURE_NAME)?.sendEvent( |
There was a problem hiding this comment.
Publish TTFD metadata after successful writes
This sends the TTFD vital id to profiling before the VitalAppLaunchEvent write is submitted. If a vitalAppLaunchEventMapper drops the TTFD event or storage rejects the write, the buffered profile can still reference a vital id that RUM never ingested; mirror the operation-vital path and send this profiling event from the write operation's success callback.
Useful? React with 👍 / 👎.
| scheduleCooldown(activeMs) | ||
| return | ||
| } | ||
| onActiveWindowStarted() |
There was a problem hiding this comment.
Wait for the previous result before clearing events
If a Perfetto result callback is delayed past the cooldown, scheduleNextCycle() can run while PerfettoProfiler still considers the previous request active, so the next profiler.start no-ops but onActiveWindowStarted() has already cleared pendingRumEvents. That loses the RUM metadata needed by the previous profile and marks a new active window without an actual trace; don't clear/start a new window until the prior result callback has completed or the profiler confirms a new request started.
Useful? React with 👍 / 👎.
1371e7d to
54580e9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54580e9ceb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) | ||
| }.onSuccess { | ||
| sendProfilingRumOperationEvent( | ||
| rumContext = capturedRumContext, |
There was a problem hiding this comment.
Use the written RUM context for operation vitals
When writeScope is delayed long enough for a session/view renewal, newVitalEvent() builds the RUM vital from the later getRumContext() on the storage worker, but this callback publishes the profiler metadata with the earlier capturedRumContext. In that scenario the operation vital is written under the new RUM session/view while the profile buffers the same vital id under the old session/view, breaking correlation; publish from the event that was actually written or build both from the same captured context.
Useful? React with 👍 / 👎.
| // fatal-ANR replay path (DatadogLateCrashReporter, via ApplicationExitInfo on | ||
| // the next launch) can dedupe against an ANR the in-process watchdog/profiling | ||
| // trigger already reported. The storage key keeps the legacy "fatal" name. | ||
| sdkCore.writeLastFatalAnrSent(event.eventTime.timestamp) |
There was a problem hiding this comment.
Avoid deduping distinct fatal ANRs
Persisting every successful non-fatal ANRException into the fatal-ANR marker makes DatadogLateCrashReporter skip any later ApplicationExitInfo ANR whose timestamp is within the new ±10s tolerance. If an app recovers from one ANR and then is killed by a separate ANR within that window (or restarts quickly and hangs again), the fatal crash is treated as already sent and is never reported; store a marker that identifies the replayed ANR source more precisely instead of using this shared timestamp for all non-fatal ANRs.
Useful? React with 👍 / 👎.
| val isRunning = datadogContext.isProfilerRunning() | ||
| val quotaReason = datadogContext.resolveProfilingQuotaReason(sessionId) | ||
| return when { | ||
| isRunning -> ViewEvent.Profiling(status = ViewEvent.ProfilingStatus.RUNNING) |
There was a problem hiding this comment.
Include clock drift on running view events
The shared profiling schema says clock_drift is present when the profiler is active, and the error/long-task/vital paths populate it, but view updates marked RUNNING omit it here. For sessions with a non-zero NTP offset, consumers using the view event's profiling context do not get the offset needed to align RUM timestamps with the profile; pass datadogContext.time.serverTimeOffsetMs as clockDrift, as in the other RUM event paths.
Useful? React with 👍 / 👎.
| sdkCore.getFeature(Feature.PROFILING_FEATURE_NAME)?.withContext { datadogContext -> | ||
| quotaChecker.checkAsync(sessionId, datadogContext) |
There was a problem hiding this comment.
Skip quota checks for sampled-out profiling sessions
This quota request runs on every RUM session renewal before knowing whether profiling will actually run. When continuous profiling is disabled or the profiling sampler excludes the session and no launch trace is active, a DENIED quota response still publishes PROFILING_QUOTA_REASON, so later RUM events are tagged as profiling STOPPED due to quota even though profiling was simply disabled/sampled out; only check quota for an active launch profile or a session selected for continuous profiling.
Useful? React with 👍 / 👎.
What does this PR do?
Merge develop into feature/continuous-profiling
Review checklist (to be filled by reviewers)