Linux_android_emu android_hardware_smoke_opengles_instrumented_tests is 2.04% flaky #193557

Activity

1. fluttergithubbot

   assigned

   andywolff

   on Sep 30, 2026
2. fluttergithubbot

   added

   P0Critical issues such as a build break or regressionCritical issues such as a build break or regression

   c: flakeTests that sometimes, but not always, incorrectly passTests that sometimes, but not always, incorrectly pass

   team-engineOwned by Engine teamOwned by Engine team

   on Sep 30, 2026
3. andywolff commented on Oct 5, 2026

   

   andywolff

   on Oct 5, 2026

   Contributor

   More actions

   This and #193207 represent a failure case which I'm not sure how to resolve. Sometimes the environment is in a bad state when the test starts. I implemented a detection and retry mechanism for this.

   ║ INFRASTRUCTURE FAILURE: EGL / graphics driver pipeline collapsed on emulator after 2 attempts.

   However, it seems either no retries are needed, or all retries are exhausted. If the test starts in a bad state, it fails that way across all retries. Ideally there would be some way to avoid this from happening, but I don't have any ideas at the moment.
4. andywolff

   marked Linux_android_emu_vulkan_stable android_hardware_smoke_vulkan_tests is 2.02% flaky #193207 as a duplicate of this issue on Oct 5, 2026
5. gaaclarke

   added

   P1High-priority issues at the top of the work listHigh-priority issues at the top of the work list

   and removed

   P0Critical issues such as a build break or regressionCritical issues such as a build break or regression

   on Oct 12, 2026
6. github-project-automation

   added this to Flutter Engine Focuson Oct 12, 2026
7. github-project-automation

   moved this to Todo in Flutter Engine Focuson Oct 12, 2026
8. gaaclarke commented on Oct 12, 2026

   

   gaaclarke

   on Oct 12, 2026

   Member

   More actions

   moving to p1 since they are suppresed
9. gaaclarke

   added

   triaged-engineTriaged by Engine teamTriaged by Engine team

   on Oct 12, 2026
10. github-project-automation

    moved this from Todo to In progress in Flutter Engine Focuson Oct 29, 2026
11. andywolff commented on Nov 3, 2026

    

    andywolff

    on Nov 3, 2026

    Contributor

    More actions

    I took another look at the failures. I didn't see any pattern with regards to a particular machine, and I found two distinct emulator graphics failure modes behind this flakiness.

    The first and more common failure mode (#194928, #191348, #193205) is a SurfaceFlinger display wedge where the 3 platform view tests always produce solid-black screenshots while the non-platform view tests pass. During test startup, CI launches the emulator with hw.gpu.enabled = true and hw.gpu.mode = swangle_indirect, while restoring a default_boot QuickBoot snapshot saved with hw.gpu.enabled=no and hw.gpu.mode=auto. Every CI boot logs WARNING Snapshot.cpp:365 | change of renderer detected and ERROR FrameBuffer.cpp:3642 | Failed to find ColorBuffer:0. Roughly 99% of the time, SurfaceFlinger recovers after waking from the snapshot and binds a new window surface against swangle_indirect. On roughly 1% of boots, that post-restore re-initialization fails (EmulationGl.cpp:702 | Failed to find emulated EGL config 4 and FrameBuffer.cpp:3193 | Failed to create EmulatedEglWindowSurface), leaving the thawed SurfaceFlinger process holding stale pre-snapshot EGLConfig and ColorBuffer handles. The first 6 tests still pass because the newly launched app process initializes fresh GPU state against swangle_indirect and reads pixels in-process via RenderRepaintBoundary.toImage(), whereas the 3 platform view tests take screenshots with adb screencap (or UiAutomation.takeScreenshot()), which reads from SurfaceFlinger's broken display surface.

    The second failure mode (#193557) is a native host-GPU-pipe crash during app launch. Occasionally a newly launched app process fails to open its guest-to-host GPU pipe (E EGL_emulation: Failed to establish connection with the host, W libEGL: eglInitialize(...) failed (EGL_SUCCESS), and SIGSEGV in libEGL.so on OpenGLES, or E MESA: vulkan: Failed to get host connection and ErrorDeviceLost on Vulkan). When that crash happens right after the Dart VM prints its service URI, flutter drive hangs until LUCI kills the entire shard on timeout.

    I tried to reproduce the QuickBoot SurfaceFlinger wedge locally first. I created an API 35 google_apis (userdebug) AVD and cold-booted it headless (emulator -avd ci_repro_api35 -no-window -no-audio -no-boot-anim), which defaults in config.ini to hw.gpu.enabled=no, hw.gpu.mode=auto, and fastboot.forceFastBoot=yes. Running adb emu kill sends a graceful shutdown over the emulator console, triggering QEMU's QuickBoot exit hook to save default_boot automatically (Saving snapshot 'default_boot'). When I tried to restore that snapshot with a different renderer (emulator -avd ci_repro_api35 -no-window -gpu host -no-snapshot-save and -gpu swangle), my local Android Emulator 36.5.11.0 binary (CI pins 35.3.11 and 35.4.9) rejected the cross-renderer snapshot outright with WARNING | Change of GLES renderer detected (or Change of Vulkan renderer detected) followed by Error -1 from the snapshot callback -> Failed to load snapshot 'default_boot' -> The emulator is starting from scratch. Reason: different renderer configured. That means newer 36.x emulator binaries treat cross-renderer QuickBoot loads as unsafe, though it also meant 36.5.11.0 cold-booted cleanly instead of entering the stale-snapshot SurfaceFlinger wedge.

    While I couldn't enter the 35.x snapshot wedge locally on 36.5.11.0, I was able to reproduce the second failure mode (the native host-GPU-pipe startup crash and flutter drive hang) on that booted API 35 emulator by running adb root and chmod 0660 /dev/goldfish_pipe_dprctd /dev/goldfish_address_space so the already-running surfaceflinger kept its open file descriptors while newly launched app processes could not open the host GPU transport:

    • On OpenGLES, the app hit the same logcat failure sequence seen in #193557: E EGL_emulation: Failed to establish connection with the host, W libEGL: eglInitialize(...) failed (EGL_SUCCESS), and SIGSEGV in libEGL.so. That caught an important edge case in our logcat filter: excluding bare 'EGL_SUCCESS' (intended for HWUI's error = EGL_SUCCESS probe) masked libEGL's real eglInitialize(...) failed (EGL_SUCCESS) failure when libEGL_emulation returned EGL_FALSE without setting a TLS error code.
    • On Vulkan, the app hit the Vulkan equivalent (E MESA: vulkan: Failed to get host connection, Could not create Vulkan instance: ErrorDeviceLost, and Could not create surface from invalid Android context), crashing right after the Dart VM printed its service URI and reproducing the exact FlutterDriver VM service connection hang from CI.

    Testing against that live failure mode also showed why our existing retries weren't working. _runRetryLoop ran flutter drive via runCommand, which called foundError on attempt 1 and permanently marked the shard failed even if attempt 2 had ever passed (in CI logs, attempt 2 never passed anyway because the display stayed wedged). App-level retries (ActivityScenario.recreate() and flutter drive --no-build) only restarted the app Activity without touching SurfaceFlinger. We can instead call adb shell stop -> adb shell start to respawn surfaceflinger and zygote, though we have to set sys.boot_completed to 0 beforehand because stop leaves it at 1 and our readiness poll would otherwise launch attempt 2 before userspace finishes booting back up. On native startup crashes, the hung flutter drive process never reached attempt 2 without checking adb shell pidof on stalled VM connections and sending SIGKILL (flutter_tools catches SIGTERM and exits 0).

    The logcat checks had two bugs of their own. FlutterActivityTest logged its attempt marker with Log.i while logcat -d --pid=$pid *:W stripped all INFO lines so post-marker checks never armed, and both the Kotlin and Dart filters matched benign HWUI warnings like EGL_SWAP_BEHAVIOR_PRESERVED and Davey! instead of explicit fatal graphics errors.

    With a background watcher on the emulator restoring /dev/goldfish_pipe_dprctd and /dev/goldfish_address_space permissions (chmod 0666) upon sys.boot_completed=0, attempt 1 detected the Vulkan crash, killed the hung flutter drive, restarted surfaceflinger and zygote64 to fresh PIDs, and passed all 9 golden tests on attempt 2 (+9) with shard exit code 0.

    Although we could only test the host-pipe crash recovery locally, we have strong evidence that adb shell stop and adb shell start will also recover the QuickBoot SurfaceFlinger wedge in CI: in every wedged CI run, host QEMU's swangle_indirect renderer itself is healthy — the freshly spawned Flutter app process creates valid host GPU state and passes all 6 offscreen Impeller tests, and Android HWUI inside that app creates its own EGLSurface without config errors. Only the long-lived surfaceflinger process thawed out of the default_boot RAM snapshot held stale pre-snapshot EGLConfig (config 4) and ColorBuffer handles. Killing and respawning surfaceflinger forces a fresh eglInitialize / eglChooseConfig / EmulatedEglWindowSurface handshake against swangle_indirect just like a newly launched process, which we'll be able to confirm directly in CI logs via the new Restarting Android userspace before retry... message.

    I'll land the retry and userspace-restart fixes first, and as a separate work item, I recommend we rebake the CIPD AVD snapshots with matching -gpu swangle_indirect flags so this mismatch never happens. Maybe we should also upgrade the CIPD emulator binary from 35.x to 36.x or higher to get the behavior that rejects cross-renderer snapshots.
12. auto-submit

    closed this as completedin #195708on Nov 6, 2026
13. github-project-automation

    moved this from In progress to Done in Flutter Engine Focuson Nov 6, 2026
14. mboetger

    added a commit that references this issue on Nov 6, 2026

    Fix android_hardware_smoke_test retry mechanisms (flutter#195708)

    ...

    1f33e88