ci: data generation workflow#1874
Conversation
bde9926 to
75c5de9
Compare
Codecov Report
@@ Coverage Diff @@
## master #1874 +/- ##
=======================================
Coverage 91.91% 91.91%
=======================================
Files 200 200
Lines 9450 9450
=======================================
Hits 8686 8686
Misses 764 764
Continue to review full report at Codecov.
|
philipphofmann
left a comment
philipphofmann
left a comment
There was a problem hiding this comment.
The purpose of this PR is to have some realistic test data in Sentry? If yes, we could achieve this with a similar approach that sales use, and it doesn't require UI tests running. @thinkocapo, how do you again generate up-to-date test data for your sales demos?
|
@philipphofmann here are the tests we run agains Android, iOS, and React
Native apps/devices
https://github.com/sentry-demos/application-monitoring/tree/master/tests/mobile_native
…On Fri, Jun 10, 2022 at 3:03 PM Philipp Hofmann ***@***.***> wrote:
***@***.**** commented on this pull request.
The purpose of this PR is to have some realistic test data in Sentry? If
yes, we could achieve this with a similar approach that sales use, and it
doesn't require UI tests running. @thinkocapo
<https://github.com/thinkocapo>, how do you again generate up-to-date
test data for your sales demos?
------------------------------
In .github/workflows/data-generator.yml
<#1874 (comment)>
:
> + matrix:
+ include:
+ - iOS: 15.5
+ - iOS: 15.4
+ - iOS: 14.8
+ - iOS: 14.7
+ - iOS: 13.7
+ - iOS: 13.6.1
+ - iOS: 13.5.1
+ - iOS: 12.5.5
+ - iOS: 12.5.4
+ - iOS: 12.5.3
+ - iOS: 12.4.8
+ - iOS: 12.4.1
l: This should do the same
⬇️ Suggested change
- matrix:
- include:
- - iOS: 15.5
- - iOS: 15.4
- - iOS: 14.8
- - iOS: 14.7
- - iOS: 13.7
- - iOS: 13.6.1
- - iOS: 13.5.1
- - iOS: 12.5.5
- - iOS: 12.5.4
- - iOS: 12.5.3
- - iOS: 12.4.8
- - iOS: 12.4.1
+ matrix:
+ iOS: [15.5, 15.4, 14.8, 14.7, 13.7, 13.6.1, 13.5.1, 12.5.5, 12.5.4, 12.5.3, 12.4.8, 12.4.1 ]
------------------------------
In .github/workflows/data-generator.yml
<#1874 (comment)>
:
> @@ -0,0 +1,68 @@
+# Defines a workflow that generates large volumes of real-ish data to work with for various development tasks.
+
+name: Data Generator
Data Generator is a bit too generic IMO. This generates data for
profiling, so maybe Profiling Test Data Generator?
------------------------------
In .github/workflows/saucelabs-UI-tests.yml
<#1874 (comment)>
:
> @@ -38,7 +38,7 @@ jobs:
MATCH_USERNAME: ${{ secrets.MATCH_USERNAME }}
- name: Archiving DerivedData
- uses: ***@***.***
+ uses: ***@***.***
m: Ideally, we could do these updates in another PR so we can merge it
sooner.
—
Reply to this email directly, view it on GitHub
<#1874 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACEB37TWZ4Z4MZB2XSJ3FSLVOM4JVANCNFSM5YMJYPLA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
William Capozzoli
***@***.*** ***@***.***>
willcap.io
|
@thinkocapo, so you run actual test cases to generate the test data for your sales demos? |
|
Hi @philipphofmann I believe the answer to your question is Yes. I'm not sure 'test cases' is the right wording. These are not tests in the traditional sense of testing functions within the SDK. We're running simulations. Our tests are loading up virtual phones in Saucelabs, and then clicking a bunch of buttons and navigating between screens, and the sentry sdk will create error+transaction events during this process and send them to Sentry.io We have metric Alerts set in our org in Sentry.io to make sure the events are coming in. If the events dip below a certain threshold, then this means the 'test' or 'simulation' is down. So this is the only true test we really run, a check that the events are coming in to our org. If the events are malformed (I haven't seen that happene yet), well we're not testing for that, but we would notice it as we start to demo. When we upgrade the SDK version in our APK in Saucelabs, this is typically when we would notice any differences. Or when we update the ViewControllers/Activities and demo app itself. |
80dfc09 to
f1df0a0
Compare
b70a1c6 to
acae938
Compare
011c612 to
3bdce22
Compare
acae938 to
1140672
Compare
Co-authored-by: Philipp Hofmann <[email protected]>
This reverts commit a2ac64c.
This reverts commit 374fb78.
…action run" This reverts commit a8df2cd.
This reverts commit 8f89808.
0617d20 to
55b3124
Compare
philipphofmann
left a comment
philipphofmann
left a comment
There was a problem hiding this comment.
LGTM 🚀 Maybe we can share the same test project?
5 participants
We need a good volume of profile data for development tasks in the backend/frontend. This imports an old test from Specto to drive the Trending Movies sample app, updated to run on real devices in Sauce Labs via a github action.
#skip-changelog