Skip to content

🔊 add telemetry for unexpected session id changes#3815

Merged
BenoitZugmeyer merged 2 commits into
mainfrom
benoit/add-telemetry
Sep 10, 2025
Merged

🔊 add telemetry for unexpected session id changes#3815
BenoitZugmeyer merged 2 commits into
mainfrom
benoit/add-telemetry

Conversation

@BenoitZugmeyer

@BenoitZugmeyer BenoitZugmeyer commented Sep 3, 2025

Copy link
Copy Markdown
Member

Motivation

Explore a quickfix for #incident-42289

Changes

This PR adds some telemetry to validate whether it is possible to detect an unexpected cookie change based on cookieStore. This change is gated by the short_session_investigation experimental feature flag

Test instructions

Using my playwright script to stress test browsers and increase the chance of cookie synchronization issues, the telemetry log is showing up when there is an issue.

Checklist

  • Tested locally
  • Tested on staging
  • Added unit tests for this change.
  • Added e2e/integration tests for this change.

@BenoitZugmeyer

This comment was marked as outdated.

@dd-devflow-routing-codex

This comment was marked as outdated.

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 3, 2025

Copy link
Copy Markdown

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage
Patch Coverage: 7.41%
Total Coverage: 92.28% (-0.33%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 15e4867 | Docs | Was this helpful? Give us feedback!

@cit-pr-commenter

cit-pr-commenter Bot commented Sep 3, 2025

Copy link
Copy Markdown

Bundles Sizes Evolution

📦 Bundle Name Base Size Local Size 𝚫 𝚫% Status
Rum 156.51 KiB 157.10 KiB 602 B 0.38%
Rum Recorder 19.53 KiB 19.53 KiB 0 B 0.00%
Rum Profiler 5.18 KiB 5.18 KiB 0 B 0.00%
Logs 55.12 KiB 55.74 KiB 636 B 1.13%
Flagging N/A 931 B 931 B N/A%
Rum Slim 114.29 KiB 114.84 KiB 561 B 0.48%
Worker 23.60 KiB 23.60 KiB 0 B 0.00%
🚀 CPU Performance
Action Name Base Average Cpu Time (ms) Local Average Cpu Time (ms) 𝚫
addglobalcontext 0.012 0.006 -0.006
addaction 0.041 0.021 -0.019
addtiming 0.010 0.004 -0.006
adderror 0.039 0.017 -0.022
startstopsessionreplayrecording 0.002 0.001 -0.001
startview 0.012 0.005 -0.007
logmessage 0.050 0.023 -0.026
🧠 Memory Performance
Action Name Base Consumption Memory (bytes) Local Consumption Memory (bytes) 𝚫 (bytes)
addglobalcontext 25.15 KiB 26.18 KiB 1.02 KiB
addaction 44.90 KiB 48.68 KiB 3.78 KiB
addtiming 24.24 KiB 24.37 KiB 133 B
adderror 51.37 KiB 52.13 KiB 773 B
startstopsessionreplayrecording 23.92 KiB 23.51 KiB -416 B
startview 423.17 KiB 429.66 KiB 6.49 KiB
logmessage 43.91 KiB 43.14 KiB -795 B

🔗 RealWorld

@BenoitZugmeyer

This comment was marked as outdated.

@dd-devflow-routing-codex

This comment was marked as outdated.

dd-mergequeue Bot added a commit that referenced this pull request Sep 3, 2025
Integrated commit sha: d7ddf16

Co-authored-by: BenoitZugmeyer <[email protected]>
@dd-devflow dd-devflow Bot added the staging-36 label Sep 3, 2025
@BenoitZugmeyer
BenoitZugmeyer marked this pull request as ready for review September 9, 2025 15:59
@BenoitZugmeyer
BenoitZugmeyer requested a review from a team as a code owner September 9, 2025 15:59
@BenoitZugmeyer

Copy link
Copy Markdown
Member Author

/to-staging

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Sep 9, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-09-09 16:02:42 UTC ℹ️ Start processing command /to-staging


2025-09-09 16:03:02 UTC ℹ️ Branch Integration: starting soon, merge expected in approximately 11m (p90)

Commit c3337fba16 will soon be integrated into staging-37.


2025-09-09 16:19:24 UTC ℹ️ Branch Integration: this commit was successfully integrated

Commit c3337fba16 has been merged into staging-37 in merge commit bc8c148d83.

Check out the triggered pipeline on Gitlab 🦊

If you need to revert this integration, you can use the following command: /code revert-integration -b staging-37

dd-mergequeue Bot added a commit that referenced this pull request Sep 9, 2025
Integrated commit sha: c3337fb

Co-authored-by: BenoitZugmeyer <[email protected]>
@dd-devflow dd-devflow Bot added the staging-37 label Sep 9, 2025
} else {
const newSessionState = toSessionState(changed.value)
if (newSessionState.id !== initialSessionState.id) {
addTelemetryDebug('Session cookie changed', {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 suggestion: ‏ I think there is still some amount of short session that are due to cookie set with different domain or config resulting to having multiple _dd_s cookies.

What about adding all the _dd_s cookies from cookieStore.getAll('_dd_s') as these will include the domain and other options the cookie were set with and it could be useful to understand why the session was shorter than expected

}

const sessionAge = dateNow() - sessionCreatedTime
if (sessionAge > 14 * ONE_MINUTE) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ question: ‏why not using the session expiration delay?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to prevent a few false positives at the cost of losing a bit of logs. I'm fearing that if we are too close to the actual limit, we could face clock issues. I'm not sure that Date.now() returns the exact same value across tabs, so the cookie could be expired in some tab and still be considered within the 15 minutes limit in another. With a 1 minute margin of error, we get rid of this potential issue.

Do you think that could have a negative impact on the data?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wanted to understand the intent.

IMO, that should be fine to get telemetry data, if we want to later include that in a potential fix, it could be interesting to explore a bit more this hypothesis around different Date.now() values across tabs.

Also, skip session state when there is no id (i.e. the cookie is
expired) because it's noisy and isn't a problematic state.
@BenoitZugmeyer

Copy link
Copy Markdown
Member Author

/to-staging

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Sep 10, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-09-10 09:11:31 UTC ℹ️ Start processing command /to-staging


2025-09-10 09:11:40 UTC ℹ️ Branch Integration: starting soon, merge expected in approximately 16m (p90)

Commit 15e4867676 will soon be integrated into staging-37.


2025-09-10 09:24:34 UTC ℹ️ Branch Integration: this commit was successfully integrated

Commit 15e4867676 has been merged into staging-37 in merge commit 9fafe56e0c.

Check out the triggered pipeline on Gitlab 🦊

If you need to revert this integration, you can use the following command: /code revert-integration -b staging-37

dd-mergequeue Bot added a commit that referenced this pull request Sep 10, 2025
Integrated commit sha: 15e4867

Co-authored-by: BenoitZugmeyer <[email protected]>
@BenoitZugmeyer
BenoitZugmeyer merged commit 45bf5fd into main Sep 10, 2025
21 checks passed
@BenoitZugmeyer
BenoitZugmeyer deleted the benoit/add-telemetry branch September 10, 2025 09:41
BeltranBulbarellaDD pushed a commit that referenced this pull request Sep 12, 2025
* 🔊 add telemetry for unexpected session id changes

* 👌 add cookie values in the telemetry log


Also, skip session state when there is no id (i.e. the cookie is
expired) because it's noisy and isn't a problematic state.
rgaignault pushed a commit that referenced this pull request Sep 15, 2025
* 🔊 add telemetry for unexpected session id changes

* 👌 add cookie values in the telemetry log

Also, skip session state when there is no id (i.e. the cookie is
expired) because it's noisy and isn't a problematic state.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants