For #182123 we have added android_hardware_smoke_test, and we're running the tests in CI as driver tests. But we're currently not running the instrumented tests in CI. This means that when we want to deliver the instrumented test APK to a hardware manufacturer, there's a chance that something may have broken the instrumented tests and not the driver tests. To resolve this issue, we should remove that chance by actually exercising the instrumented tests in CI, or close the issue if we decide that the risk is acceptable long term.
There are a few ways we could run the instrumented tests in CI:
- Test that the driver tests and the instrumented tests produce the same result images, skipping skia gold. In this option, we would first run the driver tests first with UPDATE_GOLDENS=1 and maybe some new "ignore LUCI" parameter / environment variable. This would produce local goldens. Then we would build the instrumented test with these goldens bundled, and run them.
- Test only the instrumented test, but don't bundle any goldens and disable the behavior which actually compares the goldens.
- Use
goldctl to download the goldens first, place them in the local goldens directory, then build the instrumented test apk with the goldens bundled in.
The downside of Option 1 is that it runs the tests twice sequentially. This feels a bit redundant with the existing driver tests which run against skia gold. However, maybe it's actually not redundant because otherwise the NaiveLocalFileComparator is not exercised in CI. These tests are pretty inexpensive, so CI cost is not very much of a concern. Option 2 skips golden comparison entirely, which risks the test having issues which we would never see. Option 3 sounds pretty ideal at first, but skia goldens update somewhat slowly, so there's a slight risk of them getting out of sync. Normally when using the skia gold comparator, that's okay because it's built for that, but it might be possible that the way this would fail in the instrumented tests would report errors in a way that doesn't work well with the systemic expectations of skia gold. For these reasons, I am leaning toward Option 1.
All that said, the risk of the instrumented tests breaking when the driver tests are still passing feels very low to me. The tests are structured such that the exact same app behavior is triggered by the same message channel messages in both the flutter driver tests and the android instrumented tests. As of writing this, the expected use of the instrumented tests is only once every 2-3 months on demand. So if the time comes that somebody needs the instrumented tests, and we discover an issue has been created in the past two months, it may not be that much of a problem to resolve it at time of discovery. For these reasons, this issue is a low priority.
For #182123 we have added android_hardware_smoke_test, and we're running the tests in CI as driver tests. But we're currently not running the instrumented tests in CI. This means that when we want to deliver the instrumented test APK to a hardware manufacturer, there's a chance that something may have broken the instrumented tests and not the driver tests. To resolve this issue, we should remove that chance by actually exercising the instrumented tests in CI, or close the issue if we decide that the risk is acceptable long term.
There are a few ways we could run the instrumented tests in CI:
goldctlto download the goldens first, place them in the local goldens directory, then build the instrumented test apk with the goldens bundled in.The downside of Option 1 is that it runs the tests twice sequentially. This feels a bit redundant with the existing driver tests which run against skia gold. However, maybe it's actually not redundant because otherwise the NaiveLocalFileComparator is not exercised in CI. These tests are pretty inexpensive, so CI cost is not very much of a concern. Option 2 skips golden comparison entirely, which risks the test having issues which we would never see. Option 3 sounds pretty ideal at first, but skia goldens update somewhat slowly, so there's a slight risk of them getting out of sync. Normally when using the skia gold comparator, that's okay because it's built for that, but it might be possible that the way this would fail in the instrumented tests would report errors in a way that doesn't work well with the systemic expectations of skia gold. For these reasons, I am leaning toward Option 1.
All that said, the risk of the instrumented tests breaking when the driver tests are still passing feels very low to me. The tests are structured such that the exact same app behavior is triggered by the same message channel messages in both the flutter driver tests and the android instrumented tests. As of writing this, the expected use of the instrumented tests is only once every 2-3 months on demand. So if the time comes that somebody needs the instrumented tests, and we discover an issue has been created in the past two months, it may not be that much of a problem to resolve it at time of discovery. For these reasons, this issue is a low priority.