[Web][Integration Test] Two Chrome instances launched, and the unnecessary Chrome instance breaks the foundation of the integration test [Edited]

With example in https://docs.flutter.dev/cookbook/testing/integration/introduction, after executing the command:

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/app_test.dart \
  -d chrome

One Chrome is opened. Be noticed that it is not labelled with 'chrome is controlled by automated test software'.

Then try to execute the command plus arg --no-headless:

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/app_test.dart \
  -d chrome \
  --no-headless

Two Chrome instances are launched:

One [backend one] is launched as device [this is NOT intend for test cases execution]
The other [frontend one] is launched via webdriver (chromedriver this case) [this is intend for test cases execution, labelled with 'chrome is controlled by automated test software']

[Note: Mostly, the two Chromes are overlaid, please try to drag it away (up or down), then back one would be visible.]

Both Chrome instances have test executed. However, the Chrome is launched not via chromedriver has major negative impact to test:

For js, css loading, increase the networking pressure as need to load two copies
For some write, delete scenarios, the test will result in unexpected results (as two Chromes both will try to execute delete action)

That is, the unnecessary Chrome instance breaks the foundation of the integration test (via flutter drive)

---- Source Code Analysis: Flow the flutter drive flow ----

(1)

flutter/packages/flutter_tools/lib/src/commands/drive.dart

Lines 243 to 244 in 6af40a7

    
           await driverService.start( 
        
             buildInfo,

flutter/packages/flutter_tools/lib/src/commands/drive.dart

Lines 252 to 259 in 6af40a7

    
           platformArgs: <String, Object>{ 
        
             if (traceStartup) 
        
               'trace-startup': traceStartup, 
        
             if (web) 
        
               '--no-launch-chrome': true, 
        
             if (boolArg('multidex')) 
        
               'multidex': true, 
        
           }

flutter/packages/flutter_tools/lib/src/commands/drive.dart

Lines 274 to 275 in 6af40a7

    
           final int testResult = await driverService.startTest( 
        
             testFile,

(2)

flutter/packages/flutter_tools/lib/src/drive/web_driver_service.dart

Line 93 in 6af40a7

final Future<int> runFuture = _residentRunner.run(

(3)

flutter/packages/flutter_tools/lib/src/isolated/resident_web_runner.dart

Lines 307 to 314 in 6af40a7

    
           await device.device.startApp( 
        
             package, 
        
             mainPath: target, 
        
             debuggingOptions: debuggingOptions, 
        
             platformArgs: <String, Object>{ 
        
               'uri': url.toString(), 
        
             }, 
        
           );

(4)

flutter/packages/flutter_tools/lib/src/web/web_device.dart

Lines 120 to 153 in 6af40a7

    
           Future<LaunchResult> startApp( 
        
             covariant WebApplicationPackage package, { 
        
             String? mainPath, 
        
             String? route, 
        
             required DebuggingOptions debuggingOptions, 
        
             Map<String, Object?> platformArgs = const <String, Object?>{}, 
        
             bool prebuiltApplication = false, 
        
             bool ipv6 = false, 
        
             String? userIdentifier, 
        
           }) async { 
        
             // See [ResidentWebRunner.run] in flutter_tools/lib/src/resident_web_runner.dart 
        
             // for the web initialization and server logic. 
        
             String url; 
        
             if (debuggingOptions.webLaunchUrl != null) { 
        
               final RegExp pattern = RegExp(r'^((http)?:\/\/)[^\s]+'); 
        
               if (pattern.hasMatch(debuggingOptions.webLaunchUrl!)) { 
        
                 url = debuggingOptions.webLaunchUrl!; 
        
               } else { 
        
                 throwToolExit('"${debuggingOptions.webLaunchUrl}" is not a vaild HTTP URL.'); 
        
               } 
        
             } else { 
        
               url = platformArgs['uri']! as String; 
        
             } 
        
             final bool launchChrome = platformArgs['no-launch-chrome'] != true; 
        
             if (launchChrome) { 
        
               _chrome = await chromeLauncher.launch( 
        
                 url, 
        
                 cacheDir: _fileSystem.currentDirectory 
        
                     .childDirectory('.dart_tool') 
        
                     .childDirectory('chrome-device'), 
        
                 headless: debuggingOptions.webRunHeadless, 
        
                 debugPort: debuggingOptions.webBrowserDebugPort, 
        
               ); 
        
             }

Per above code, it shows:

Per (1) the --no-launch-chrome is set to true if web, but per (3)(4) it is not passed to startApp, that is, below expression is always true

flutter/packages/flutter_tools/lib/src/web/web_device.dart

Line 143 in 6af40a7

final bool launchChrome = platformArgs['no-launch-chrome'] != true;

However, even the --no-launch-chrome is passed correctly, the code will exit with error, as there exists isAppStarted check

flutter/packages/flutter_tools/lib/src/drive/web_driver_service.dart

Lines 117 to 119 in 6af40a7

    
           if (!isAppStarted) { 
        
             throw ToolExit('Failed to start application'); 
        
           }

@zpsean

@zpsean
I tried to replicate the scenario you mentioned but don't get to see two chrome instances being launched. I see only one as expected:

I am on latest stable (2.10.3) and used the steps as-is mentioned in the link you shared.

@darshankawar

@darshankawar Mostly, the two Chromes are overlaid, please try to drag it away (up or down), then back one would be visible.

I'm running on macOS, with latest stable (2.10.3). And this can be reproduced by my mates.

Thanks. I tried the same but don't see same behavior. Please confirm if the interpretation is not correct to replicate it properly:

100447.mov

Please refer to my screen capture, as my mac may be slow, give me chance to capture them.

flutter_drive_two_chromes.mp4

% ~/Downloads/flutter/bin/flutter --version                                                                                                                                          
Flutter 2.10.3 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 7e9793dee1 (3 weeks ago) • 2022-03-02 11:23:12 -0600
Engine • revision bd539267b4
Tools • Dart 2.16.1 • DevTools 2.9.2

% ~/Downloads/flutter/bin/flutter doctor 
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.10.3, on macOS 12.0.1 21A559 darwin-x64, locale en-CN)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2020.3)
[✓] IntelliJ IDEA Community Edition (version 2021.3.2)
[✓] VS Code (version 1.64.2)
[✓] Connected device (2 available)
[✓] HTTP Host Availability

@darshankawar

@darshankawar
I know the reason we got different results. Sorry, I provided the wrong command in original post, below is the one real used

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/app_test.dart \
  -d chrome \
  --driver-port 4444 \
  --no-headless \
  --profile

Please be noticed that one of the Chrome has labelled 'chrome is controlled by automated test software', the other is not

More, if use below command, the Chrome opened by chromediver is in headless mode

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/app_test.dart \
  -d chrome

Which means, in my understanding, the Chrome in your run is the Chrome which is opened as device, but not the one intend to be used as test (i.e. opened via chromedriver).

#148915 is raised against the user doc https://docs.flutter.dev/cookbook/testing/integration/introduction, to avoid the misleading.

@zpsean

Thanks @zpsean for providing detailed updates on the issue. I think this is an interesting find and do see the same behavior , ie, of two chrome instances being opened when running below command:

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/app_test.dart \
  -d chrome \
  --no-headless

stable, master flutter doctor -v


[✓] Flutter (Channel stable, 2.10.3, on macOS 12.2.1 21D62 darwin-x64, locale
    en-GB)
    • Flutter version 2.10.3 at /Users/dhs/documents/fluttersdk/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 7e9793dee1 (11 hours ago), 2022-03-02 11:23:12 -0600
    • Engine revision bd539267b4
    • Dart version 2.16.1
    • DevTools version 2.9.2

[!] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (5 available)
    • SM G975F (mobile)       • RZ8M802WY0X • android-arm64   • Android 11 (API 30)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 98.0.4758.80

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.


[✓] Flutter (Channel master, 2.13.0-0.0.pre.191, on macOS 12.2.1 21D62
    darwin-x64, locale en-GB)
    • Flutter version 2.13.0-0.0.pre.191 at
      /Users/dhs/documents/fluttersdk/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 69b92f0452 (3 minutes ago), 2022-03-21 21:25:09 -0700
    • Engine revision e6233bd30b
    • Dart version 2.17.0 (build 2.17.0-223.0.dev)
    • DevTools version 2.11.4

[!] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (5 available)
    • SM G975F (mobile)       • RZ8M802WY0X • android-arm64   • Android 11 (API 30)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 98.0.4758.80

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

@jiahaog

/cc @jiahaog

@yjbanov

Apologies, I'm not familiar with this, @yjbanov would you know more?

@darshankawar

@darshankawar @yjbanov, any further comments on this?

This is a duplicate of #67090

@yjbanov

@yjbanov @darshankawar this key point of this issue i want to address is that expecting the Chrome (as device) is not to be opened. In #67090, it is stated as harmless but confusing, but I think, as stated in original post, is harmful, as we can Not reply on the real test results with the Chrome existence (device one).

Both Chrome instances have test executed. However, the Chrome is launched not via chromedriver has major negative impact to test:

For js, css loading, increase the networking pressure as need to load two copies
For some write, delete scenarios, the test will result in unexpected results (as two Chromes both will try to execute delete action)

We can have more discussion how two use option -d and --browser-name, maybe to prevent the Chrome (as device) to be opened have high priority as its impact (negative) to test .

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

[Web][Integration Test] Two Chrome instances launched, and the unnecessary Chrome instance breaks the foundation of the integration test [Edited] #100447

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	platformArgs: <String, Object>{
	if (traceStartup)
	'trace-startup': traceStartup,
	if (web)
	'--no-launch-chrome': true,
	if (boolArg('multidex'))
	'multidex': true,
	}

	final int testResult = await driverService.startTest(
	testFile,

	await device.device.startApp(
	package,
	mainPath: target,
	debuggingOptions: debuggingOptions,
	platformArgs: <String, Object>{
	'uri': url.toString(),
	},
	);

	Future<LaunchResult> startApp(
	covariant WebApplicationPackage package, {
	String? mainPath,
	String? route,
	required DebuggingOptions debuggingOptions,
	Map<String, Object?> platformArgs = const <String, Object?>{},
	bool prebuiltApplication = false,
	bool ipv6 = false,
	String? userIdentifier,
	}) async {
	// See [ResidentWebRunner.run] in flutter_tools/lib/src/resident_web_runner.dart
	// for the web initialization and server logic.
	String url;
	if (debuggingOptions.webLaunchUrl != null) {
	final RegExp pattern = RegExp(r'^((http)?:\/\/)[^\s]+');
	if (pattern.hasMatch(debuggingOptions.webLaunchUrl!)) {
	url = debuggingOptions.webLaunchUrl!;
	} else {
	throwToolExit('"${debuggingOptions.webLaunchUrl}" is not a vaild HTTP URL.');
	}
	} else {
	url = platformArgs['uri']! as String;
	}
	final bool launchChrome = platformArgs['no-launch-chrome'] != true;
	if (launchChrome) {
	_chrome = await chromeLauncher.launch(
	url,
	cacheDir: _fileSystem.currentDirectory
	.childDirectory('.dart_tool')
	.childDirectory('chrome-device'),
	headless: debuggingOptions.webRunHeadless,
	debugPort: debuggingOptions.webBrowserDebugPort,
	);
	}

	if (!isAppStarted) {
	throw ToolExit('Failed to start application');
	}

[Web][Integration Test] Two Chrome instances launched, and the unnecessary Chrome instance breaks the foundation of the integration test [Edited] #100447

Description

Activity

darshankawar commented on Mar 21, 2022

zpsean commented on Mar 21, 2022

darshankawar commented on Mar 21, 2022

zpsean commented on Mar 21, 2022

zpsean commented on Mar 21, 2022

zpsean commented on Mar 22, 2022

darshankawar commented on Mar 22, 2022

darshankawar commented on Mar 22, 2022

jiahaog commented on Mar 22, 2022

zpsean commented on Mar 24, 2022

yjbanov commented on Mar 24, 2022

zpsean commented on Mar 25, 2022

zpsean commented on Mar 25, 2022

github-actions commented on Apr 8, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions