SDK Crash Detection

Background

As an APM company, the reliability of our SDKs is one of our most essential quality goals. If our SDK breaks the customer, we fail. Our SDK philosophy refers to this as degrade gracefully.

For some SDKs, like mobile SDKs, we primarily rely on users to report SDK crashes because we don't operate them in production. If users don't report them, we are unaware. Instead, we should detect crashes caused by our SDKs when they happen so we can proactively fix them.

The SDK crash detection doesn't seek to detect severe bugs, such as the transport layer breaking or the SDK continuously crashing. CI or other quality mechanisms should find such severe bugs. Furthermore, the solution only targets SDKs maintained by us, Sentry.

In the beginning, this solution only works for the Cocoa SDK crashes. We will roll out this feature to more SDKs in the future.

Solution

The SDK crash detection hooks into post-processing and checks the stacktraces of every event.

sentry/src/sentry/tasks/post_process.py

Lines 1063 to 1086 in 4040cb3

	def sdk_crash_monitoring(job: PostProcessJob):
	from sentry.utils.sdk_crashes.sdk_crash_detection import sdk_crash_detection
	if job["is_reprocessed"]:
	return
	event = job["event"]
	if not features.has("organizations:sdk-crash-detection", event.project.organization):
	return
	if settings.SDK_CRASH_DETECTION_PROJECT_ID is None:
	logger.warning(
	"SDK crash detection is enabled but SDK_CRASH_DETECTION_PROJECT_ID is not set."
	)
	return None
	with metrics.timer("post_process.sdk_crash_monitoring.duration"):
	with sentry_sdk.start_span(op="tasks.post_process_group.sdk_crash_monitoring"):
	sdk_crash_detection.detect_sdk_crash(
	event=event,
	event_project_id=settings.SDK_CRASH_DETECTION_PROJECT_ID,
	sample_rate=settings.SDK_CRASH_DETECTION_SAMPLE_RATE,
	)

If the event is fatal, caused by one of our SDKs, the code strips away most of the data based on an allow list and stores the event to a dedicated Sentry project. The event_stripper only keeps SDK and system library frames. For grouping to work correctly, the event_stripper sets in_app to true for all SDK frames, but the grouping config will change it to in_app false for all Cocoa SDK frames. To not change the grouping logic, we add the following stacktrace rule stack.abs_path:Sentry.framework +app +group to the configured in project with the id configured in the option issues.sdk_crash_detection.cocoa.project_id.

The feature flag organizations:sdk-crash-detection controls whether SDK crash detection runs for an organization. It is managed via flagpole in sentry-options-automator and is currently enabled in the US and s4s2 regions. The per-SDK project IDs and sample rates are still controlled via options (e.g. issues.sdk_crash_detection.cocoa.project_id, issues.sdk_crash_detection.cocoa.sample_rate).