Automatically downsample transaction when system is under load (backpressure) by adinauer · Pull Request #3072 · getsentry/sentry-java

adinauer · 2023-12-01T09:53:05Z

📜 Description

Automatically downsample transaction when system is under load by checking:

if any events have been dropped due to queue being full recently (2s)
if any rate limit is in effect (regardless of category)

This check is performed every 10 seconds. With each negative health check we halve tracesSampleRate up to 10 times, meaning the original tracesSampleRate is multiplied by 1, 1/2, 1/4, ... up to 1/1024 (~ 0.001%). Any positive health check resets to the original tracesSampleRate set in SentryOptions.

Also see python PRs for reference.

💡 Motivation and Context

Closes #2829

💚 How did you test it?

Unit tests, manually using Spring Boot sample.

📝 Checklist

I reviewed the submitted code.
I added tests to verify the changes.
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
I updated the docs if needed.
Review from the native team if needed.
No breaking change or entry added to the changelog.
No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

…at/7.0.0

Co-authored-by: Sentry Github Bot <[email protected]> Co-authored-by: Roman Zavarnitsyn <[email protected]>

* reduced timeout of AsyncHttpTransport close to flushTimeoutMillis (default 4s in Android and 15s in Java) to avoid ANRs in Android

* Add deadline timeout for automatic transactions * Update Changelog * Address PR comments * Fix formatting * Add missing test, improve naming * Ensure deadline timeout is only set once

* updated minimum Android SDK version to 19 * removed few workarounds for version < 19

Co-authored-by: Roman Zavarnitsyn <[email protected]> Co-authored-by: Sentry Github Bot <[email protected]>

sentry/src/main/java/io/sentry/TracesSampler.java

sentry/src/main/java/io/sentry/backpressure/BackpressureMonitor.java

sentry/src/main/java/io/sentry/transport/AsyncHttpTransport.java

sentry/src/main/java/io/sentry/transport/RateLimiter.java

github-actions · 2023-12-01T10:41:59Z

Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	393.41 ms	418.63 ms	25.22 ms
Size	1.72 MiB	2.27 MiB	558.11 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`d6d2b2e`	392.55 ms	467.50 ms	74.95 ms
`99a51e2`	405.11 ms	479.65 ms	74.54 ms
`0bd723b`	375.20 ms	452.41 ms	77.20 ms
`c7e2fbc`	377.85 ms	426.35 ms	48.50 ms
`b172d4e`	412.60 ms	492.68 ms	80.08 ms
`4e29063`	364.08 ms	445.51 ms	81.43 ms
`bc4be3b`	360.40 ms	435.04 ms	74.64 ms
`93a76ca`	397.30 ms	455.16 ms	57.87 ms
`8fd337b`	349.16 ms	459.22 ms	110.06 ms
`93a76ca`	377.41 ms	448.22 ms	70.81 ms

App size

Revision	Plain	With Sentry	Diff
`d6d2b2e`	1.72 MiB	2.27 MiB	555.05 KiB
`99a51e2`	1.72 MiB	2.29 MiB	576.34 KiB
`0bd723b`	1.72 MiB	2.29 MiB	578.09 KiB
`c7e2fbc`	1.72 MiB	2.29 MiB	576.40 KiB
`b172d4e`	1.72 MiB	2.29 MiB	578.09 KiB
`4e29063`	1.72 MiB	2.29 MiB	578.38 KiB
`bc4be3b`	1.72 MiB	2.29 MiB	576.53 KiB
`93a76ca`	1.72 MiB	2.29 MiB	576.75 KiB
`8fd337b`	1.72 MiB	2.27 MiB	555.00 KiB
`93a76ca`	1.72 MiB	2.29 MiB	576.75 KiB

Previous results on branch: feat/backpressure

Startup times

Revision	Plain	With Sentry	Diff
`8d081e4`	446.34 ms	535.84 ms	89.50 ms
`9f06b6e`	361.10 ms	430.00 ms	68.90 ms
`95f8fd0`	415.82 ms	470.54 ms	54.72 ms
`3b8dee7`	436.04 ms	526.52 ms	90.48 ms

App size

Revision	Plain	With Sentry	Diff
`8d081e4`	1.72 MiB	2.27 MiB	556.60 KiB
`9f06b6e`	1.72 MiB	2.27 MiB	556.27 KiB
`95f8fd0`	1.72 MiB	2.27 MiB	556.38 KiB
`3b8dee7`	1.72 MiB	2.27 MiB	556.30 KiB

…ly full

sl0thentr0py · 2023-12-01T14:08:32Z

one thing you're missing - a new client report (DiscardReason) called backpressure and use that here

sentry-java/sentry/src/main/java/io/sentry/Hub.java

Lines 668 to 670 in 1ad36f1

    
           options 
        
               .getClientReportRecorder() 
        
               .recordLostEvent(DiscardReason.SAMPLE_RATE, DataCategory.Transaction);

sentry/src/main/java/io/sentry/transport/RateLimiter.java

sentry/src/main/java/io/sentry/transport/QueuedThreadPoolExecutor.java

lbloder

Left comments on some minor things. Looks good otherwise 👍

Do we want to add the configuration also to the ExternalOptions to make this available to other integrations too?
As far as I can see, enabling backpressureHandling is currently only possible via Spring and the SentryProperties or by supplying SentryOptions in code, correct?

sentry/src/main/java/io/sentry/transport/RateLimiter.java

sentry/src/main/java/io/sentry/backpressure/BackpressureMonitor.java

sentry/src/main/java/io/sentry/transport/AsyncHttpTransport.java

…check is run against it

lbloder

LGTM, one minor nitpick :)

sentry/src/main/java/io/sentry/clientreport/DiscardReason.java

markushi and others added 30 commits June 29, 2023 14:14

Move enableNdk to SentryAndroidOptions (#2793)

0566579

Capture failed HTTP requests by default (#2794)

4f55fd9

Merge branch 'main' into feat/7.0.0

d6fc22e

Fix Changelog

36afdd4

Merge branch 'feat/7.0.0' of github.com:getsentry/sentry-java into fe…

590fcb8

…at/7.0.0

7.0.0 -> Unreleased in changelog

1faf168

Do not overwrite UI transaction status if set by the user (#2852)

cae0eea

Reduce flush timeout to 4s on Android to avoid ANRs (#2858)

3838091

Set ip_address = {{auto}} by default (#2860)

19fca04

Merge branch 'main' into feat/7.0.0

6182cc2

Fix test

7717683

Measure AppStart time till First Draw (#2851)

1c0c691

update coroutines to 1.6.1, use CopyableThreadContextElement (#2838)

803f03c

Co-authored-by: Sentry Github Bot <[email protected]> Co-authored-by: Roman Zavarnitsyn <[email protected]>

Reduce timeout of AsyncHttpTransport to avoid ANR (#2879)

8e78ac3

* reduced timeout of AsyncHttpTransport close to flushTimeoutMillis (default 4s in Android and 15s in Java) to avoid ANRs in Android

Add deadline timeout for automatic transactions (#2865)

0b3de21

* Add deadline timeout for automatic transactions * Update Changelog * Address PR comments * Fix formatting * Add missing test, improve naming * Ensure deadline timeout is only set once

Apollo v2 BeforeSpanCallback: Allow returning null (#2890)

ee21d53

Start a new automatic transaction on every click (#2891)

55b103c

Bump min API to 19 (#2883)

b0e93e8

* updated minimum Android SDK version to 19 * removed few workarounds for version < 19

Fix don't overwrite the span status of unfinished spans (#2859)

bdf1379

(Android) Use root transaction instead of last active span (#2855)

cd268a3

Merge branch 'main' into feat/7.0.0

1821d66

Fix changelog

421d724

Make SDK unity-compatible (#2847)

c383ed1

Capture unfinished transactions on crash (#2938)

72e23fc

Observe network state to upload any unsent envelopes (#2910)

d6cd78c

Co-authored-by: Roman Zavarnitsyn <[email protected]> Co-authored-by: Sentry Github Bot <[email protected]>

Merge branch 'main' into feat/7.0.0

227698f

Highlight breaking chnages in changelog

ca595ea

Api dump

ece39a6

release: 7.0.0-beta.1

7554695

Merge branch 'release/7.0.0-beta.1' into feat/7.0.0

96f4697

fix changelog

0d3bb1b

adinauer commented Dec 1, 2023

View reviewed changes

check if queue recently rejected instead of checking if it is current…

1ad36f1

…ly full

sl0thentr0py reviewed Dec 1, 2023

View reviewed changes

sentry/src/main/java/io/sentry/transport/RateLimiter.java Outdated Show resolved Hide resolved

sl0thentr0py reviewed Dec 1, 2023

View reviewed changes

sentry/src/main/java/io/sentry/transport/QueuedThreadPoolExecutor.java Outdated Show resolved Hide resolved

adinauer added 2 commits December 5, 2023 11:21

CR changes; tests; cleanup

94b09a2

Make opt-in; reduce recent reject check to 2s

0e1b15a

adinauer marked this pull request as ready for review December 5, 2023 14:31

adinauer requested review from markushi, romtsn and stefanosiano as code owners December 5, 2023 14:31

bruno-garcia mentioned this pull request Dec 5, 2023

SDK Backpressure and 100% Sampling getsentry/sentry-dotnet#2946

Closed

lbloder reviewed Dec 12, 2023

View reviewed changes

adinauer and others added 8 commits December 12, 2023 08:22

add changelog

c989dee

external option; add tests for options

7b22a09

add backpressure discard reason

51d7297

Consider NoOpHub healthy so it does not cause downsampling in case a …

5b51964

…check is run against it

remove sysos

8c69f98

delay initial health check so it runs against a real hub

4987607

enable backpressure handling for spring samples

ee7cb9f

Merge branch 'main' into feat/backpressure

637bbbe

lbloder approved these changes Dec 13, 2023

View reviewed changes

sentry/src/main/java/io/sentry/clientreport/DiscardReason.java Outdated Show resolved Hide resolved

adinauer and others added 2 commits December 14, 2023 09:30

remove blank line

3194e3f

Merge branch 'main' into feat/backpressure

30caf2f

adinauer merged commit 8ff8fd0 into main Dec 15, 2023

adinauer deleted the feat/backpressure branch December 15, 2023 11:54

bitsandfoxes mentioned this pull request Mar 4, 2024

Backpressure Management getsentry/sentry-dotnet#2410

Closed

adinauer mentioned this pull request Mar 25, 2024

Enable Backpressure Management by Default #3284

Merged

7 tasks

Uh oh!

Comments

Conversation

adinauer commented Dec 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/backpressure

Startup times

App size

Uh oh!

sl0thentr0py commented Dec 1, 2023

Uh oh!

Uh oh!

Uh oh!

lbloder left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lbloder left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

adinauer commented Dec 1, 2023 •

edited

Loading

github-actions bot commented Dec 1, 2023 •

edited

Loading