Can't gather performance timeline using flutter drive #90410
Description
I am trying to write a test that uses watchPerformance to write the correct build/integration_response_data.json file for reading in chrome://tracing.
The test passes when using flutter test integration_test/app_test.dart, but not when I run flutter drive to specify a custom integrationDriver from the integration_test_driver package.
I also wonder if it's possible to specify the custom integrationDriver using the flutter test command, not flutter drive, since this seems to be the new way to run integration_test tests.
Steps to Reproduce
- Create the scrolling app from the Handle scrolling cookbook article:
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
void main() {
runApp(MyApp(
items: List<String>.generate(10000, (i) => "Item $i"),
));
}
class MyApp extends StatelessWidget {
final List<String> items;
const MyApp({Key? key, required this.items}) : super(key: key);
@override
Widget build(BuildContext context) {
const title = 'Long List';
return MaterialApp(
title: title,
home: Scaffold(
appBar: AppBar(
title: const Text(title),
),
body: ListView.builder(
// Add a key to the ListView. This makes it possible to
// find the list and scroll through it in the tests.
key: const Key('long_list'),
itemCount: items.length,
itemBuilder: (context, index) {
return ListTile(
title: Text(
items[index],
// Add a key to the Text widget for each item. This makes
// it possible to look for a particular item in the list
// and verify that the text is correct
key: Key('item_${index}_text'),
),
);
},
),
),
);
}
}- Create a file,
integration_test/app_test.dart:
import 'package:flutter/foundation.dart';
import 'package:flutter/widgets.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
// The application under test.
import 'package:integration_test_experiments/main.dart';
void main() {
final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized()
as IntegrationTestWidgetsFlutterBinding;
testWidgets('Scroll to item 50', (WidgetTester tester) async {
// Build our app and trigger a frame.
await tester.pumpWidget(MyApp(
items: List<String>.generate(10000, (i) => "Item $i"),
));
final listFinder = find.byType(Scrollable);
final itemFinder = find.byKey(const ValueKey('item_50_text'));
await binding.watchPerformance(
() async {
// Scroll until the item to be found appears.
await tester.scrollUntilVisible(
itemFinder,
500.0,
scrollable: listFinder,
);
},
reportKey: 'scrolling_summary',
);
// Verify that the item contains the correct text.
expect(itemFinder, findsOneWidget);
});
}- Add a file,
test_driver/perf_driver.dartthat writes the response data in separate timeline files (Document multi-timeline usage #88604):
import 'package:integration_test/integration_test_driver.dart';
Future<void> main() {
return integrationDriver(
responseDataCallback: (data) async {
if (data != null) {
for (var entry in data.entries) {
print('Writing ${entry.key} to the disk.');
await writeResponseData(
entry.value as Map<String, dynamic>,
testOutputFilename: entry.key,
);
}
}
},
);
}Run the integration test using the perf_driver.dart driver configuration:
flutter drive \
--driver=test_driver/perf_driver.dart \
--target=integration_test/app_test.dart \
-d <DEVICE_ID>
Observe exception:
flutter drive \
--driver=test_driver/integration_driver.dart \
--target=integration_test/app_test.dart -d 13181FDD400226
Running "flutter pub get" in integration_test_experiments... 808ms
Running Gradle task 'assembleDebug'... 4.5s
✓ Built build/app/outputs/flutter-apk/app-debug.apk.
Installing build/app/outputs/flutter-apk/app.apk... 3.6s
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:60052/Z7lQhlgRbYw=/
VMServiceFlutterDriver: Isolate found with number: 3018735296192043
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
I/flutter (16092): 00:00 +0: Scroll to item 50
VMServiceFlutterDriver: Connected to Flutter application.
I/flutter (16092): ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
I/flutter (16092): The following SocketException was thrown running a test:
I/flutter (16092): OS Error: Connection refused, errno = 111, address = localhost, port = 41680
I/flutter (16092):
I/flutter (16092): When the exception was thrown, this was the stack:
I/flutter (16092): #0 _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:677:35)
I/flutter (16092): #1 _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1680:26)
I/flutter (16092): #2 RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)
I/flutter (16092): #3 Socket._startConnect (dart:io-patch/socket_patch.dart:1891:22)
I/flutter (16092): #4 Socket.startConnect (dart:io/socket.dart:738:21)
I/flutter (16092): #5 _ConnectionTarget.connect (dart:_http/http_impl.dart:2439:18)
I/flutter (16092): #6 _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2805:12)
I/flutter (16092): #7 _HttpClient._getConnection (dart:_http/http_impl.dart:2810:12)
I/flutter (16092): #8 _HttpClient._openUrl (dart:_http/http_impl.dart:2694:12)
I/flutter (16092): #9 _HttpClient.openUrl (dart:_http/http_impl.dart:2568:7)
I/flutter (16092): #10 _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1027:24)
I/flutter (16092): #11 WebSocket.connect (dart:_http/websocket.dart:379:22)
I/flutter (16092): #12 vmServiceConnectUri (package:vm_service/vm_service_io.dart:32:44)
I/flutter (16092): #13 IntegrationTestWidgetsFlutterBinding.enableTimeline
(package:integration_test/integration_test.dart:244:26)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092): #14 IntegrationTestWidgetsFlutterBinding.traceTimeline
(package:integration_test/integration_test.dart:269:5)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092): #15 IntegrationTestWidgetsFlutterBinding._runAndGetGCInfo
(package:integration_test/integration_test.dart:321:34)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092): #16 IntegrationTestWidgetsFlutterBinding.watchPerformance
(package:integration_test/integration_test.dart:376:43)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092): #17 main.<anonymous closure>
(file:///Users/ryjohn/code/github/johnpryan/integration_test_experiments/integration_test/app_test.dart:23:5)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092): #18 testWidgets.<anonymous closure>.<anonymous closure>
(package:flutter_test/src/widget_tester.dart:176:15)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092): #19 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:817:5)
I/flutter (16092): <asynchronous suspension>
I/flutter (16092):
I/flutter (16092): The test description was:
I/flutter (16092): Scroll to item 50
I/flutter (16092): ════════════════════════════════════════════════════════════════════════════════════════════════════
I/flutter (16092): 00:03 +0: Scroll to item 50 [E]
I/flutter (16092): Test failed. See exception logs above.
I/flutter (16092): The test description was: Scroll to item 50
I/flutter (16092):
I/flutter (16092): Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
I/flutter (16092): For example, 'dart test --chain-stack-traces'.
I/flutter (16092): 00:03 +0 -1: (tearDownAll)
I/flutter (16092): 00:03 +1 -1: Some tests failed.
Failure Details:
Failure in method: Scroll to item 50
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following SocketException was thrown running a test:
OS Error: Connection refused, errno = 111, address = localhost,
port = 41680
When the exception was thrown, this was the stack:
#0 _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:677:35)
#1 _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1680:26)
#2 RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)
#3 Socket._startConnect (dart:io-patch/socket_patch.dart:1891:22)
#4 Socket.startConnect (dart:io/socket.dart:738:21)
#5 _ConnectionTarget.connect (dart:_http/http_impl.dart:2439:18)
#6 _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2805:12)
#7 _HttpClient._getConnection (dart:_http/http_impl.dart:2810:12)
#8 _HttpClient._openUrl (dart:_http/http_impl.dart:2694:12)
#9 _HttpClient.openUrl (dart:_http/http_impl.dart:2568:7)
#10 _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1027:24)
#11 WebSocket.connect (dart:_http/websocket.dart:379:22)
#12 vmServiceConnectUri (package:vm_service/vm_service_io.dart:32:44)
#13 IntegrationTestWidgetsFlutterBinding.enableTimeline (package:integration_test/integration_test.dart:244:26)
<asynchronous suspension>
#14 IntegrationTestWidgetsFlutterBinding.traceTimeline (package:integration_test/integration_test.dart:269:5)
<asynchronous suspension>
#15 IntegrationTestWidgetsFlutterBinding._runAndGetGCInfo (package:integration_test/integration_test.dart:321:34)
<asynchronous suspension>
#16 IntegrationTestWidgetsFlutterBinding.watchPerformance (package:integration_test/integration_test.dart:376:43)
<asynchronous suspension>
#17 main.<anonymous closure> (file:///Users/ryjohn/code/github/johnpryan/integration_test_experiments/integration_test/app_test.dart:23:5)
<asynchronous suspension>
#18 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:176:15)
<asynchronous suspension>
#19 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:817:5)
<asynchronous suspension>
The test description was:
Scroll to item 50
═════════════════════════════════════════════════════════════════
end of failure 1