RUM-15278: Add Datadog Test Observability for tests run on MacOS runners#3295
Conversation
9a9332e to
2ec1d52
Compare
| - $ANDROID_HOME/platform-tools/adb emu kill | ||
| - if [[ "$exit_code" -ne 0 ]]; then exit 1; fi | ||
| - exit 0 | ||
| upload-junit-results: |
There was a problem hiding this comment.
Are you sure we cannot use Datadog Agent for instrumented runs? The runner there is still JUnit-compatible and produces results in the JUnit format. Does it mean that Datadog Agent fails to instrumented test run?
There was a problem hiding this comment.
I tried doing it using the Agent, yes, but it didn't work. IIUC it doesn't work because the test runs on Android device, not on the JVM and the tracer can't properly hook to the test.
The best way I know is to upload test results using datadog-ci. It looks like this. Not so bad.
2574321 to
19badf5
Compare
19badf5 to
8107c2f
Compare
This comment has been minimized.
This comment has been minimized.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #3295 +/- ##
===========================================
+ Coverage 71.49% 71.58% +0.09%
===========================================
Files 946 946
Lines 34895 34895
Branches 5906 5906
===========================================
+ Hits 24947 24979 +32
+ Misses 8283 8263 -20
+ Partials 1665 1653 -12 🚀 New features to boost your workflow:
|
|
|
||
| source-secrets-macos: | ||
| - vault login -method=aws -no-print | ||
| - export DD_ANDROID_SECRETS_PATH_PREFIX="kv/aws/arn:aws:iam::486234852809:role/ci-dd-sdk-android/" |
There was a problem hiding this comment.
on MacOS runners we have to get secrets from a different path
| - GRADLE_OPTS="-Xmx3072m" ./gradlew :reliability:core-it:connectedDebugAndroidTest --stacktrace --no-daemon $( (( $ANDROID_API <= 23 )) && echo "-Puse-api21-java-backport -Puse-desugaring" ) || exit_code=$? | ||
| - GRADLE_OPTS="-Xmx3072m -javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=$DD_COMMON_AGENT_CONFIG" DD_TAGS="test.configuration.android_api:$ANDROID_API" ./gradlew :reliability:core-it:connectedDebugAndroidTest --stacktrace --no-daemon $( (( $ANDROID_API <= 23 )) && echo "-Puse-api21-java-backport -Puse-desugaring" ) || exit_code=$? | ||
| - $ANDROID_HOME/platform-tools/adb emu kill | ||
| - datadog-ci junit upload --service dd-sdk-android --tags test.configuration.android_api:$ANDROID_API --xpath-tag test.suite=/testcase/@classname reliability/core-it/build/outputs/androidTest-results/connected/debug/ |
There was a problem hiding this comment.
--tags test.configuration.android_api:$ANDROID_API is used to be able to distinguish between the same tests but run on different android versions.
| - GRADLE_OPTS="-Xmx3072m" ./gradlew :reliability:core-it:connectedDebugAndroidTest --stacktrace --no-daemon $( (( $ANDROID_API <= 23 )) && echo "-Puse-api21-java-backport -Puse-desugaring" ) || exit_code=$? | ||
| - GRADLE_OPTS="-Xmx3072m -javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=$DD_COMMON_AGENT_CONFIG" DD_TAGS="test.configuration.android_api:$ANDROID_API" ./gradlew :reliability:core-it:connectedDebugAndroidTest --stacktrace --no-daemon $( (( $ANDROID_API <= 23 )) && echo "-Puse-api21-java-backport -Puse-desugaring" ) || exit_code=$? | ||
| - $ANDROID_HOME/platform-tools/adb emu kill | ||
| - datadog-ci junit upload --service dd-sdk-android --tags test.configuration.android_api:$ANDROID_API --xpath-tag test.suite=/testcase/@classname reliability/core-it/build/outputs/androidTest-results/connected/debug/ |
There was a problem hiding this comment.
--xpath-tag test.suite=/testcase/@classname is needed, because otherwise all tests end up belonging to the same test suite (for example FeatureSdkCoreTest).
see docs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8107c2f031
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3 participants
Making the results of the tests that are run on MacOS AMI runners available in Datadog CI Visibility.
How it works
We are using 2 things here:
datadog-citool.How results look like
For example here are the results for
test-pyramid:legacy-integration-instrumented-latest-apijob in this PR. You can see 3 entries there:datadog-ciupload with test results.:buildSrcis separate). They might be useful to look at the duration of gradle tasks.Here is a list of test results for
test-pyramid:legacy-integration-instrumented-latest-api.Why do we need
datadog-ciAnother option is using Java Agent like described here. But it doesn't work for Android tests, because the tracer isn't able to instrument the tests run on Android emulator using
AndroidJnitRunner.