Skip to content

Linux linux_web_engine_tests is failing somewhat consistently on Firefox #172713

Closed

Description

@matanlurey

Example failure:

00:30 +756 ~45 -1: text_editing_test.dart: GloballyPositionedTextEditingStrategy keeps focus within window/iframe when the focus moves within the flutter view in Chrome but not Safari [E]            
  Expected: JSValue:<[object HTMLElement]>
    Actual: JSValue:<[object HTMLInputElement]>
  
  text_editing_test.dart.browser_test.dart.wasm 1:305801   module0.Error._throwWithCurrentStackTrace

It's failed 12 of 50 times, and 4 time in a row at HEAD in post-submit, turning the tree 🔴.

Activity

  1. added
    P0Critical issues such as a build break or regression
    c: disabled testTracks the restoration of a disabled test
    team-webOwned by Web platform team
    on Jul 24, 2025
  2. matanlurey commented on Jul 24, 2025

    @matanlurey
    ContributorAuthor

    The test implies its something for testing Safari and Chrome but is running on Firefox? Not clear.

  3. jason-simmons commented on Jul 24, 2025

    @jason-simmons
    Contributor

    This test contains a workaround for some unexpected behavior in Firefox:
    https://github.com/flutter/flutter/blob/master/engine/src/flutter/lib/web_ui/test/engine/text_editing_test.dart#L692

    I can reproduce the error using the current tip-of-tree on my Linux setup (running Firefox 128.12.0esr)

    The test works if I replace the expect(domDocument.activeElement, flutterView.dom.rootElement) check with the expect(domDocument.activeElement, textEditing.strategy.domElement) used for Chromium.

    Possibly Firefox has fixed the issue in recent versions? The version of Firefox may not be consistent across all of the LUCI bots, so the test may need to support both behaviors.

    @yjbanov

  4. jason-simmons commented on Jul 24, 2025

    @jason-simmons
    Contributor

    Actually the LUCI recipe is trying to ensure consistency by installing a Firefox CIPD package and using that to run the test suite.

    It's installing Firefox from https://chrome-infra-packages.appspot.com/p/flutter_internal/browsers/firefox/linux-amd64/+/ajAxjAuJqj60NQuTvLhM2v1cYPK9-Ix4pbFOFHMqNoIC

    Interestingly that package is labeled as version:132.0 on CIPD. But when I run the Firefox binary in the package it identifies itself as version 141.0.

    If I set FIREFOX_EXECUTABLE to the binary from CIPD, then the test script launches that version of Firefox as expected. But on my setup I'm still seeing the test pass only when using the Chromium-style expect(domDocument.activeElement, textEditing.strategy.domElement) assertion.

  5. mdebbar commented on Jul 25, 2025

    @mdebbar
    Contributor

    Interestingly that package is labeled as version:132.0 on CIPD. But when I run the Firefox binary in the package it identifies itself as version 141.0.

    I didn't try the linux one, but I tried the macOS: https://chrome-infra-packages.appspot.com/p/flutter_internal/browsers/firefox/mac-arm64/+/msObCvBoFYmpRGiA5RhOfoD9XjwOo8-5C3QLSdUIN6AC

    It was reporting version 132.0 as expected.

    Are you sure you don't have an existing Firefox installation that's v141.0? When I launched the firefox binary from CIPD, it showed me a dialog asking me if I want to switch to the already installed newer version:

    Image
  6. jason-simmons commented on Jul 25, 2025

    @jason-simmons
    Contributor

    Tried this again on my Linux setup, and it looks like Firefox updated itself.

    When I download the CIPD package that is labeled as version:132.0, running firefox --version on the binary in the package prints Mozilla Firefox 132.0.

    But running that Firefox for the first time starts an auto update that replaces the binaries extracted from CIPD. After running and exiting Firefox, running firefox --version at the same path prints Mozilla Firefox 141.0.

    Is it possible that a similar auto update process is happening on LUCI?

    The LUCI recipe that is failing runs multiple Firefox-based test suites, and a suite that runs later in the recipe sometimes fails.
    I'm wondering if the failure may happen if Firefox updated itself while running one of the earlier suites? We could try having the test script run firefox --version to confirm that it's getting the expected version.

  7. mdebbar commented on Jul 25, 2025

    @mdebbar
    Contributor

    That's a plausible theory, I'll try to confirm if that's the case, and I'll look for a way to prevent auto updates in Firefox (Chrome doesn't have this issue because we use Chrome For Testing which has auto updates disabled by default).

  8. jason-simmons commented on Jul 25, 2025

    @jason-simmons
    Contributor

    Confirmed that Firefox auto updates are happening when running Web engine tests in my local environment.

    I extracted the Firefox CIPD package and ran a test suite with:

    FIREFOX_EXECUTABLE=$CIPD_PACKAGE_PATH/firefox dev/felt test --suite=firefox-dart2js-canvaskit-engine
    

    After a few runs the Firefox installation at CIPD_PACKAGE_PATH was replaced by a newer version.

    I was able to prevent this by adding a distribution/policies.json file to the CIPD package containing:

    {
      "policies": {
        "DisableAppUpdate": true
      }
    }
    

    (This only works if there is no system-level policy in a place like /etc/firefox/policies that overrides the distribution-level policy. I don't know if there is any such policy in the system image used on LUCI.)

  9. mdebbar commented on Jul 25, 2025

    @mdebbar
    Contributor

    Yep, I found 2 ways to disable auto-updates. The one you mentioned, and another one that works better for our current setup using prefs.js. Sending a PR soon to try it out.

    After we fix the auto-update issue, we need to delete the updated Firefox executables from those bots. Does the recipe always re-download the CIPD package or does it reuse the downloaded one?

  10. 2 remaining items

  11. jason-simmons commented on Jul 25, 2025

    @jason-simmons
    Contributor

    IIUC the cipd ensure command run by the Firefox dependency step will verify that the files in the destination directory match the contents of the specified package.

  12. mdebbar commented on Jul 25, 2025

    @mdebbar
    Contributor

    I added logging for the firefox version. See: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/3480076/overview

    What happens is we download Firefox from CIPD, then we run multiple suites with the same executable. The first suite runs with the correct version 132.0. But at some point, Firefox updates itself, so a later suite runs with version 141.0.

    ^ This was the confusing part for me because I thought it was not possible for Firefox to update itself in the middle of a test suite. Now it all makes sense, we are running multiple suites with the same executable. We close and re-launch Firefox between suites, so the later suites will get the updated version.

    I'm now trying to confirm that this works: #172775

  13. mdebbar commented on Jul 31, 2025

    @mdebbar
    Contributor

    Update:

    I tried 3 different ways to disable auto-updates:

    1. Setting app.update.* preferences in prefs.js or user.js: Causes tests to hang and eventually time out
    2. Using a policies.json file: Did not have any effect.
    3. Deleting "updater" files: Did not have any effect.

    To unblock releases, I landed a PR that makes the test accept both the old and new Firefox behaviors: #172767

  14. github-actions commented on Aug 14, 2025

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  15. locked as resolved and limited conversation to collaborators on Aug 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionbrowser: firefoxonly manifests in Firefoxc: disabled testTracks the restoration of a disabled testteam-webOwned by Web platform team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions