Skip to content

Upgrade android_hardware_smoke_test CI to run instrumented tests#189390

Merged
auto-submit[bot] merged 9 commits into
flutter:masterfrom
andywolff:devicelab
Jul 17, 2026
Merged

Upgrade android_hardware_smoke_test CI to run instrumented tests#189390
auto-submit[bot] merged 9 commits into
flutter:masterfrom
andywolff:devicelab

Conversation

@andywolff

@andywolff andywolff commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

This PR adds native Android instrumented tests to the android_hardware_smoke_test CI suite in parallel to existing driver tests (Part of #187538).

Adds two new shards temporarily, both with bringup: true:

  • android_hardware_smoke_vulkan_instrumented_tests
  • android_hardware_smoke_opengles_instrumented_tests

These shards run the driver tests first, similar to existing shards, then subsequently run the instrumented tests. They do this via a new optional runInstrumented param in run_android_hardware_smoke_tests.dart.

We'll remove the parameter and put this behavior in the existing shards after it proves stable.

Why do we need to run the instrumented tests sequentially after the driver tests?

I generally prefer to avoid doing multiple things sequentially in an integration test. It tends to introduce flakiness. So for this suite, why can't we have one shard for driver tests and a separate one for instrumented tests?

The driver tests use a skia gold comparator to compare result images to skia gold at runtime. However, no such comparator is available for instrumented tests. goldctl isn't designed to run on android. So we need to get the goldens somehow at APK build time.

In this PR, the integration tests run in two sequential passes. We first execute a host-driven driver test to save reference screenshots, and then run native JUnit tests to compare against those local files. This is the same way we will generate instrumented test APKs to be delivered to OEMs. I considered optimizing this by downloading the latest approved goldens directly from the Skia Gold server. It turns out it's possible to calculate the public URLs based on the test name and some other parameters. So we could theoretically write a script which cheaply fetches all the goldens for the suite. Then we could configure a shard which runs that script instead of the driver tests, achieving an independent shard for instrumented tests only.

However, this downloader approach is incompatible with presubmit. Any PR that introduces a change to the goldens would fail because the script would only return the latest approved goldens. Even after a developer approved the change on the dashboard, subsequent presubmit runs would still download the old goldens, deadlocking the PR checks. Technically that's not a problem for post-submit runs, but I'd rather keep presubmit and postsubmit identical instead of optimizing only postsubmit with something like this.

Running the driver pass first, as we do in this PR, avoids the problem by dynamically capturing reference screenshots generated by the PR's own code. This keeps the skew at zero and allows tests to pass once approved.

Also, I'm not that comfortable relying on calculating URLs, since there would be no guarantee that the URL scheme will remain stable. The URL scheme hasn't changed since 2020, but compared to relying only on goldctl, it just feels more fragile to me. That being said, I've implemented the download script in a personal agent skill instead, where I can use it at my own risk, and it isn't weight bearing in any way.

Pre-launch Checklist

@andywolff
andywolff force-pushed the devicelab branch 2 times, most recently from 6beb088 to 0ba17e6 Compare July 13, 2026 21:26
@Piinks Piinks added engine flutter/engine related. See also e: labels. fyi-android For the attention of Android platform team labels Jul 13, 2026
@andywolff

Copy link
Copy Markdown
Contributor Author

I triggered some LED staging runs, here's an execution time breakdown comparison:

Build Step / Phase Vulkan Baseline (#8676354) Vulkan Staging (This PR) (#8676356) OpenGL ES Baseline (#8676354) OpenGL ES Staging (This PR) (#8676356)
Git Checkout 1m 09s 0m 46s 0m 57s 0m 43s
Environment & Tool Setup 1m 45s 1m 42s 1m 48s 1m 36s
AVD Emulator Startup & Setup 1m 42s 1m 53s 1m 59s 1m 52s
Test Suite: Host-Driven Driver (Total) 3m 10s 3m 09s 3m 18s 2m 49s
-> Actual Dart Test Execution 0m 12s 0m 12s 0m 09s 0m 09s
Test Suite: Instrumented (Total) N/A (Not supported) 1m 27s N/A (Not supported) 1m 22s
-> Actual Android Test Execution N/A (Not supported) ~0m 35s N/A (Not supported) ~0m 35s
Post-Test AVD Cleanup & Teardown 0m 00s 0m 01s 0m 01s 0m 01s
Total Build Duration 7m 46s 9m 10s 8m 03s 8m 33s

On a cold cache, compiling and executing the instrumented JUnit tests adds around 1.5min to the Vulkan shard and closer to 0.5min to the OpenGL ES shard. I notice that instrumented tests take longer to run after startup (~35s) than driver tests (12s) because they have to restart the activity for each test case.

I note that we should not expect much precision from these single-run comparison numbers because CI test runs can contain a lot of variance in their timing. I'll do a more robust comparison averaging several runs before and after this PR merges.

@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces instrumented Android hardware smoke tests for Vulkan and OpenGLES backends. It adds corresponding CI targets, updates the test runner script to support building and running instrumented tests via Gradle, manages golden image assets, and documents the architectural decisions in the README. Feedback on the changes highlights a compilation error where path.absolute is incorrectly called with multiple arguments, and suggests refactoring the directory copying logic to use entity.fileSystem instead of a hardcoded LocalFileSystem to support unit testing.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
@github-actions github-actions Bot removed the engine flutter/engine related. See also e: labels. label Jul 13, 2026
@andywolff andywolff added the CICD Run CI/CD label Jul 14, 2026
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces instrumented Android hardware smoke tests for Vulkan and OpenGLES backends, adding new CI targets, test suite runners, and documentation explaining the execution model. It also implements directory copying and cleanup utilities to handle golden screenshots during test execution. Feedback suggests improving the cleanup utility to ensure empty subdirectories are also removed, keeping the Git worktree completely clean.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
@andywolff
andywolff requested a review from jtmcdole July 16, 2026 03:56
@andywolff
andywolff marked this pull request as ready for review July 16, 2026 03:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds instrumented Android hardware smoke tests for Vulkan and OpenGLES backends, updating CI targets, test ownership, and the test runner script to build and run these tests via Gradle. It also includes documentation explaining the chosen execution model. Feedback suggests simplifying the _cleanGoldensDirectory helper function by recursively deleting non-README entities directly instead of handling files and directories separately.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
jtmcdole
jtmcdole previously approved these changes Jul 16, 2026

@jtmcdole jtmcdole left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm-cat-shake

@andywolff

Copy link
Copy Markdown
Contributor Author

#189613 won the race so I had to do a merge on this one, please take another look

@andywolff
andywolff requested a review from jtmcdole July 17, 2026 06:03

@jtmcdole jtmcdole left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only give animated lgtm gifs once per pr. Its the law.

@andywolff andywolff added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 17, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Jul 17, 2026
Merged via the queue into flutter:master with commit aa6bb3d Jul 17, 2026
20 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 17, 2026
auto-submit Bot pushed a commit to flutter/packages that referenced this pull request Jul 18, 2026
Roll Flutter from fc1ad955f164 to 8005793c3562 (40 revisions)

flutter/flutter@fc1ad95...8005793

2026-07-18 [email protected] Remove `LineContents ` experimental AA line shader that is no longer used (flutter/flutter#189619)
2026-07-18 [email protected] Roll Skia from 9468e96cc40f to ba90f98535de (8 revisions) (flutter/flutter#189680)
2026-07-17 [email protected] Roll Dart SDK from 33e4b71e984b to 666e1e2133b7 (2 revisions) (flutter/flutter#189673)
2026-07-17 [email protected] [iOS] Fix potential use-after-free in a11y bridge channel handler (flutter/flutter#189637)
2026-07-17 [email protected] Migrate dev/tools skill validation to new dart_skills_lint API (flutter/flutter#189626)
2026-07-17 [email protected] Add validation for required fields during xcodebuild (flutter/flutter#187772)
2026-07-17 [email protected] Upgrade android_hardware_smoke_test CI to run instrumented tests (flutter/flutter#189390)
2026-07-17 [email protected] Roll Dart SDK from 0867cb1897b5 to 33e4b71e984b (1 revision) (flutter/flutter#189661)
2026-07-17 [email protected] Roll Skia from 702c3e790232 to 9468e96cc40f (1 revision) (flutter/flutter#189660)
2026-07-17 [email protected] Roll Packages from 9f95026 to 4fdc766 (11 revisions) (flutter/flutter#189659)
2026-07-17 [email protected] Roll Skia from 2e4a3ae035cd to 702c3e790232 (1 revision) (flutter/flutter#189655)
2026-07-17 [email protected] [macOS] Add FlutterPluginRegistrar.valuePublished(byPlugin:) just like iOS (flutter/flutter#189614)
2026-07-17 [email protected] Roll Dart SDK from c69d138c9646 to 0867cb1897b5 (2 revisions) (flutter/flutter#189649)
2026-07-17 [email protected] Roll Skia from 4f5aca109c87 to 2e4a3ae035cd (3 revisions) (flutter/flutter#189646)
2026-07-17 [email protected] [iOS] Fix missing nil checks and improve SemanticsObject bridge API (flutter/flutter#189630)
2026-07-17 [email protected] Roll Skia from 37c5e6b26aee to 4f5aca109c87 (19 revisions) (flutter/flutter#189638)
2026-07-17 [email protected] Roll Fuchsia Linux SDK from lLFbh5kFWbUGgC9Ek... to NL8xtzr8cxr5E8r8E... (flutter/flutter#189632)
2026-07-17 [email protected] Add blendMode parameter to RawImage and RenderImage (flutter/flutter#185938)
2026-07-17 [email protected] [web] Cache WASM network requests in flutter test (flutter/flutter#189623)
2026-07-17 [email protected] android_hardware_smoke_tests: Apply semantic line breaks to readme (flutter/flutter#189613)
2026-07-16 [email protected] ci(github-actions): resolve zizmor github-env findings in composite flutter actions (flutter/flutter#189602)
2026-07-16 [email protected] Remove obsolete packages analysis flag (flutter/flutter#189525)
2026-07-16 [email protected] Roll Dart SDK from e24870ff15bc to c69d138c9646 (2 revisions) (flutter/flutter#189597)
2026-07-16 [email protected] [fuchsia][iwyu] Remove transitive include of fuchsia.input.report. (flutter/flutter#188891)
2026-07-16 [email protected] Remove unnecessary value key hack in `key.dart`  (flutter/flutter#189291)
2026-07-16 [email protected] Roll Dart SDK from 81306a2ed317 to e24870ff15bc (1 revision) (flutter/flutter#189569)
2026-07-16 [email protected] [Impeller] Fix atlas growth test (flutter/flutter#189533)
2026-07-16 [email protected] Roll Dart SDK from d402ff7c9c84 to 81306a2ed317 (4 revisions) (flutter/flutter#189558)
2026-07-16 [email protected] Move shared darwin plugin tests into own builder (flutter/flutter#189411)
2026-07-16 [email protected] Roll Skia from ab2410bc857c to 37c5e6b26aee (19 revisions) (flutter/flutter#189549)
2026-07-16 [email protected] [ios] Fix //flutter:unittests build for physical devices (flutter/flutter#189543)
2026-07-16 [email protected] Fix lower DragTarget not being recognized in overlapping targets (flutter/flutter#188979)
2026-07-16 [email protected] Relocate cupertino samples that were under widgets/ (flutter/flutter#188876)
2026-07-15 [email protected] [iOS] Migrate DisplayLinkManager to a shared instance (flutter/flutter#189492)
2026-07-15 [email protected] [Impeller] Call glfwTerminate during global test environment teardown if a playground test called glfwInit (flutter/flutter#189523)
2026-07-15 [email protected] Roll pub packages (flutter/flutter#189515)
2026-07-15 [email protected] Roll Packages from ad2eab1 to 9f95026 (8 revisions) (flutter/flutter#189509)
2026-07-15 [email protected] add `@nonVirtual` to `RenderObject.attached`, fix `WidgetTester.hasRunningAnimations` (flutter/flutter#186832)
2026-07-15 [email protected] Roll vulkan-deps to 0582f446e54a (flutter/flutter#188524)
2026-07-15 [email protected] Roll Dart SDK from 0c408ff6dce9 to d402ff7c9c84 (1 revision) (flutter/flutter#189497)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected] on the revert to ensure that a human
is aware of the problem.
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD fyi-android For the attention of Android platform team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants