Microbenchmarks flaky; "Null check operator used on a null value" #150542
Description
When I run the venerable stocks/build_bench.dart microbenchmark, it sometimes prints an error with a stack trace. On my Pixel 8, I tried it in a loop just now and it happened on the 8th try; so the failure rate seems like something in the neighborhood of 10-15%.
I noticed this because I was getting a similar error while developing the new benchmarks I just sent as #150539. I debugged a little and it didn't seem related to anything specific to my code, so I was curious whether an existing benchmark had the same symptom. Turns out it does.
I don't know how many different benchmarks are affected. I picked stocks/build_bench.dart because it seemed the most similar to what I was doing; in particular it's one of three that use WidgetTester.pumpBenchmark.
Curiously it does still print results. The results are even consistent with the results I saw on non-error runs.
Full output:
$ time flutter run --release lib/stocks/build_bench.dart
Launching lib/stocks/build_bench.dart on Pixel 8 in release mode...
Warning: unable to detect project KGP version. Skipping version checking.
This may be because you have applied KGP after the Flutter Gradle Plugin.
Running Gradle task 'assembleRelease'... 1,172ms
✓ Built build/app/outputs/flutter-apk/app-release.apk (35.3MB)
Flutter run key commands.
h List all available interactive commands.
c Clear the screen
q Quit (terminate the application on the device).
I/flutter (10573): (The following exception is now available via WidgetTester.takeException:)
I/flutter (10573): Null check operator used on a null value
I/flutter (10573): #0 WidgetTester.pumpBenchmark (package:flutter_test/src/widget_tester.dart:688)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #1 runBuildBenchmark.<anonymous closure> (package:microbenchmarks/stocks/build_bench.dart:48)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #2 benchmarkWidgets.<anonymous closure> (package:flutter_test/src/widget_tester.dart:451)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #3 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1025)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #4 TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:817)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): (If WidgetTester.takeException is called, the above exception will be ignored. If it is not, then the above exception will be dumped when another exception is caught by the framework or when the test ends, whichever happens first, and then the test will fail due to having not caught or expected the exception.)
I/flutter (10573): Null check operator used on a null value
I/flutter (10573): #0 WidgetTester.pumpBenchmark (package:flutter_test/src/widget_tester.dart:688)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #1 runBuildBenchmark.<anonymous closure> (package:microbenchmarks/stocks/build_bench.dart:48)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #2 benchmarkWidgets.<anonymous closure> (package:flutter_test/src/widget_tester.dart:451)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #3 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1025)
I/flutter (10573): <asynchronous suspension>
I/flutter (10573): #4 TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:817)
I/flutter (10573): <asynchronous suspension>
E/flutter (10573): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Test failed. See exception logs above.
E/flutter (10573):
I/flutter (10573): ================ RESULTS ================
I/flutter (10573): :::JSON::: {"stock_build_iteration":535.2731213872833,"stock_build_iteration_probability_5pct":0.034504185823212435}
I/flutter (10573): ================ FORMATTED ==============
I/flutter (10573): Stock build: 535.3 µs per iteration
I/flutter (10573): Stock build - probability margin of error 0.05: 0.0 percent
Application finished.
time: 17.312s wall (2.835s u, 0.461s s)