Upgrade android_hardware_smoke_test CI to run instrumented tests#189390
Conversation
6beb088 to
0ba17e6
Compare
…CI in new temporary bringup shards
|
I triggered some LED staging runs, here's an execution time breakdown comparison:
On a cold cache, compiling and executing the instrumented JUnit tests adds around 1.5min to the Vulkan shard and closer to 0.5min to the OpenGL ES shard. I notice that instrumented tests take longer to run after startup (~35s) than driver tests (12s) because they have to restart the activity for each test case. I note that we should not expect much precision from these single-run comparison numbers because CI test runs can contain a lot of variance in their timing. I'll do a more robust comparison averaging several runs before and after this PR merges. |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces instrumented Android hardware smoke tests for Vulkan and OpenGLES backends. It adds corresponding CI targets, updates the test runner script to support building and running instrumented tests via Gradle, manages golden image assets, and documents the architectural decisions in the README. Feedback on the changes highlights a compilation error where path.absolute is incorrectly called with multiple arguments, and suggests refactoring the directory copying logic to use entity.fileSystem instead of a hardcoded LocalFileSystem to support unit testing.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces instrumented Android hardware smoke tests for Vulkan and OpenGLES backends, adding new CI targets, test suite runners, and documentation explaining the execution model. It also implements directory copying and cleanup utilities to handle golden screenshots during test execution. Feedback suggests improving the cleanup utility to ensure empty subdirectories are also removed, keeping the Git worktree completely clean.
There was a problem hiding this comment.
Code Review
This pull request adds instrumented Android hardware smoke tests for Vulkan and OpenGLES backends, updating CI targets, test ownership, and the test runner script to build and run these tests via Gradle. It also includes documentation explaining the chosen execution model. Feedback suggests simplifying the _cleanGoldensDirectory helper function by recursively deleting non-README entities directly instead of handling files and directories separately.
|
#189613 won the race so I had to do a merge on this one, please take another look |
jtmcdole
left a comment
There was a problem hiding this comment.
I can only give animated lgtm gifs once per pr. Its the law.
Roll Flutter from fc1ad955f164 to 8005793c3562 (40 revisions) flutter/flutter@fc1ad95...8005793 2026-07-18 [email protected] Remove `LineContents ` experimental AA line shader that is no longer used (flutter/flutter#189619) 2026-07-18 [email protected] Roll Skia from 9468e96cc40f to ba90f98535de (8 revisions) (flutter/flutter#189680) 2026-07-17 [email protected] Roll Dart SDK from 33e4b71e984b to 666e1e2133b7 (2 revisions) (flutter/flutter#189673) 2026-07-17 [email protected] [iOS] Fix potential use-after-free in a11y bridge channel handler (flutter/flutter#189637) 2026-07-17 [email protected] Migrate dev/tools skill validation to new dart_skills_lint API (flutter/flutter#189626) 2026-07-17 [email protected] Add validation for required fields during xcodebuild (flutter/flutter#187772) 2026-07-17 [email protected] Upgrade android_hardware_smoke_test CI to run instrumented tests (flutter/flutter#189390) 2026-07-17 [email protected] Roll Dart SDK from 0867cb1897b5 to 33e4b71e984b (1 revision) (flutter/flutter#189661) 2026-07-17 [email protected] Roll Skia from 702c3e790232 to 9468e96cc40f (1 revision) (flutter/flutter#189660) 2026-07-17 [email protected] Roll Packages from 9f95026 to 4fdc766 (11 revisions) (flutter/flutter#189659) 2026-07-17 [email protected] Roll Skia from 2e4a3ae035cd to 702c3e790232 (1 revision) (flutter/flutter#189655) 2026-07-17 [email protected] [macOS] Add FlutterPluginRegistrar.valuePublished(byPlugin:) just like iOS (flutter/flutter#189614) 2026-07-17 [email protected] Roll Dart SDK from c69d138c9646 to 0867cb1897b5 (2 revisions) (flutter/flutter#189649) 2026-07-17 [email protected] Roll Skia from 4f5aca109c87 to 2e4a3ae035cd (3 revisions) (flutter/flutter#189646) 2026-07-17 [email protected] [iOS] Fix missing nil checks and improve SemanticsObject bridge API (flutter/flutter#189630) 2026-07-17 [email protected] Roll Skia from 37c5e6b26aee to 4f5aca109c87 (19 revisions) (flutter/flutter#189638) 2026-07-17 [email protected] Roll Fuchsia Linux SDK from lLFbh5kFWbUGgC9Ek... to NL8xtzr8cxr5E8r8E... (flutter/flutter#189632) 2026-07-17 [email protected] Add blendMode parameter to RawImage and RenderImage (flutter/flutter#185938) 2026-07-17 [email protected] [web] Cache WASM network requests in flutter test (flutter/flutter#189623) 2026-07-17 [email protected] android_hardware_smoke_tests: Apply semantic line breaks to readme (flutter/flutter#189613) 2026-07-16 [email protected] ci(github-actions): resolve zizmor github-env findings in composite flutter actions (flutter/flutter#189602) 2026-07-16 [email protected] Remove obsolete packages analysis flag (flutter/flutter#189525) 2026-07-16 [email protected] Roll Dart SDK from e24870ff15bc to c69d138c9646 (2 revisions) (flutter/flutter#189597) 2026-07-16 [email protected] [fuchsia][iwyu] Remove transitive include of fuchsia.input.report. (flutter/flutter#188891) 2026-07-16 [email protected] Remove unnecessary value key hack in `key.dart` (flutter/flutter#189291) 2026-07-16 [email protected] Roll Dart SDK from 81306a2ed317 to e24870ff15bc (1 revision) (flutter/flutter#189569) 2026-07-16 [email protected] [Impeller] Fix atlas growth test (flutter/flutter#189533) 2026-07-16 [email protected] Roll Dart SDK from d402ff7c9c84 to 81306a2ed317 (4 revisions) (flutter/flutter#189558) 2026-07-16 [email protected] Move shared darwin plugin tests into own builder (flutter/flutter#189411) 2026-07-16 [email protected] Roll Skia from ab2410bc857c to 37c5e6b26aee (19 revisions) (flutter/flutter#189549) 2026-07-16 [email protected] [ios] Fix //flutter:unittests build for physical devices (flutter/flutter#189543) 2026-07-16 [email protected] Fix lower DragTarget not being recognized in overlapping targets (flutter/flutter#188979) 2026-07-16 [email protected] Relocate cupertino samples that were under widgets/ (flutter/flutter#188876) 2026-07-15 [email protected] [iOS] Migrate DisplayLinkManager to a shared instance (flutter/flutter#189492) 2026-07-15 [email protected] [Impeller] Call glfwTerminate during global test environment teardown if a playground test called glfwInit (flutter/flutter#189523) 2026-07-15 [email protected] Roll pub packages (flutter/flutter#189515) 2026-07-15 [email protected] Roll Packages from ad2eab1 to 9f95026 (8 revisions) (flutter/flutter#189509) 2026-07-15 [email protected] add `@nonVirtual` to `RenderObject.attached`, fix `WidgetTester.hasRunningAnimations` (flutter/flutter#186832) 2026-07-15 [email protected] Roll vulkan-deps to 0582f446e54a (flutter/flutter#188524) 2026-07-15 [email protected] Roll Dart SDK from 0c408ff6dce9 to d402ff7c9c84 (1 revision) (flutter/flutter#189497) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. ...

This PR adds native Android instrumented tests to the
android_hardware_smoke_testCI suite in parallel to existing driver tests (Part of #187538).Adds two new shards temporarily, both with
bringup: true:android_hardware_smoke_vulkan_instrumented_testsandroid_hardware_smoke_opengles_instrumented_testsThese shards run the driver tests first, similar to existing shards, then subsequently run the instrumented tests. They do this via a new optional
runInstrumentedparam inrun_android_hardware_smoke_tests.dart.We'll remove the parameter and put this behavior in the existing shards after it proves stable.
Why do we need to run the instrumented tests sequentially after the driver tests?
I generally prefer to avoid doing multiple things sequentially in an integration test. It tends to introduce flakiness. So for this suite, why can't we have one shard for driver tests and a separate one for instrumented tests?
The driver tests use a skia gold comparator to compare result images to skia gold at runtime. However, no such comparator is available for instrumented tests.
goldctlisn't designed to run on android. So we need to get the goldens somehow at APK build time.In this PR, the integration tests run in two sequential passes. We first execute a host-driven driver test to save reference screenshots, and then run native JUnit tests to compare against those local files. This is the same way we will generate instrumented test APKs to be delivered to OEMs. I considered optimizing this by downloading the latest approved goldens directly from the Skia Gold server. It turns out it's possible to calculate the public URLs based on the test name and some other parameters. So we could theoretically write a script which cheaply fetches all the goldens for the suite. Then we could configure a shard which runs that script instead of the driver tests, achieving an independent shard for instrumented tests only.
However, this downloader approach is incompatible with presubmit. Any PR that introduces a change to the goldens would fail because the script would only return the latest approved goldens. Even after a developer approved the change on the dashboard, subsequent presubmit runs would still download the old goldens, deadlocking the PR checks. Technically that's not a problem for post-submit runs, but I'd rather keep presubmit and postsubmit identical instead of optimizing only postsubmit with something like this.
Running the driver pass first, as we do in this PR, avoids the problem by dynamically capturing reference screenshots generated by the PR's own code. This keeps the skew at zero and allows tests to pass once approved.
Also, I'm not that comfortable relying on calculating URLs, since there would be no guarantee that the URL scheme will remain stable. The URL scheme hasn't changed since 2020, but compared to relying only on goldctl, it just feels more fragile to me. That being said, I've implemented the download script in a personal agent skill instead, where I can use it at my own risk, and it isn't weight bearing in any way.
Pre-launch Checklist
///).