Mac framework_tests_impeller is too big and flaky

The Mac framework_tests_impeller LUCI shard is flaky.

Currently in dev/bots/suite_runners/run_framework_tests.dart, runImpeller executes the entire packages/flutter test suite in a single monolithic invocation:

Future<void> runImpeller() async {
  printProgress('${green}Running packages/flutter tests $reset in Impeller$reset');
  await runFlutterTest(
    path.join(flutterRoot, 'packages', 'flutter'),
    options: <String>['--enable-impeller'],
  );
}

This runs a lot of stuff in one shard. 20,000 tests or so. If one test fails, retrying the shard re-runs all of these, which feels wasteful.

I looked at the last 500 try runs of Mac framework_tests_impeller. I noticed that failures spike during peak CI load windows (US midday and evening batch merges), whereas the post-merge builder (flutter/prod/Mac framework_tests_impeller) maintains a 100% pass rate over the last 100 runs.

Hourly Distribution of Try Failures (Pacific Time)

Hour (Pacific)   | Total Builds | Failures | Failure Rate
-------------------------------------------------------
00:00 - 07:00    |          107 |        4 |         3.7%  ####
08:00 - 11:00    |          155 |        2 |         1.3%  ##
12:00 - 15:00    |          121 |        8 |         6.6%  ########  (Peak US midday)
16:00 - 21:00    |           88 |        3 |         3.4%  ###
22:00 - 23:00    |           27 |        4 |        14.8%  ####      (Late batch merges)

From those 500 try runs, failures were found in only four test files. I think these could use some attention, or be split out into their own shard or something.

Here's a report from AI on them:

Failing Test Suite	Failures	Failure Mode
`packages/flutter/test/widgets/draggable_scrollable_sheet_test.dart`	10	Physics fling/drag animation timing
`packages/flutter/test/material/progress_indicator_test.dart`	4	Continuous spinning indicator timeout (`did not complete [E]`)
`packages/flutter/test/widgets/editable_text_cursor_test.dart`	2	Cursor blink animation timing
`packages/flutter/test/widgets/transform_test.dart`	2	3D perspective anti-aliasing mismatch

Root Causes per Failing Suite

Decoupled Host Clocks (progress_indicator_test.dart): Progress indicators run infinite AnimationController.repeat() loops. Under runner CPU throttling, asynchronous microtasks and timer ticks fall behind real wall-clock time, causing tests to exceed timeouts (- did not complete [E]).
Unbounded Fling Decay (draggable_scrollable_sheet_test.dart): Ballistic fling simulations (tester.fling) followed by unbounded tester.pumpAndSettle() generate long micro-step settling tails that hit timeouts or fail precision thresholds (closeTo(.5, precisionErrorTolerance)) under thread starvation.
Cursor Blink Thresholds (editable_text_cursor_test.dart): 500ms blink timer periods oscillate close to pump thresholds under CPU throttling.
Subpixel Anti-Aliasing (transform_test.dart): Evaluates 80 distinct 3D perspective angles comparing composited layers to canvas drawing via strict byte-for-byte image matching (matchesReferenceImage). Metal Impeller's fractional rasterization produces 1-pixel color differences on edges.

Evidence & Recent Build Runs

Attempt 1: Build 8669260472188289793 — Failed in transform_test.dart: 3D transform renders the same with or without needsCompositing.
Attempt 2: Build 8669260470147319649 — Failed in transform_test.dart: 3D transform renders the same with or without needsCompositing.
Attempt 3: Build 8669115127847759697 — transform_test.dart passed; failed on 24 test timeouts in progress_indicator_test.dart.
Attempt 4: Build 8669111987131570129 — transform_test.dart passed; failed on 23 test timeouts in progress_indicator_test.dart.

For the four failing test files, we might be able to make them more likely to pass if we do something like this for each of them:

A. Decouple Animation Loops from Host Clocks (`progress_indicator_test.dart`)

Enforce explicit, discrete time steps via await tester.pump(const Duration(milliseconds: 100)) instead of unconstrained frame loops.
Wrap spinning indicator tests with tester.binding.clock or stop repeating controllers in tearDown so in-flight frame callbacks do not leak across test boundaries in the 20,000-test runner process.

B. Bound Fling Physics and Replace `pumpAndSettle` (`draggable_scrollable_sheet_test.dart`)

Replace unbounded tester.pumpAndSettle() after ballistic flings with bounded settling:

// Instead of unbounded pumpAndSettle:
await tester.pumpAndSettle(const Duration(milliseconds: 100));
// Or pump specific expected animation durations:
await tester.pump(snapAnimationDuration ?? const Duration(milliseconds: 300));

For tests asserting on snap rest states, explicitly advance the controller animation to 1.0 or supply a fixed short snapAnimationDuration rather than relying on real-time friction decay curves.

C. Stabilize Blinking Cursor Tests (`editable_text_cursor_test.dart`)

Set EditableText.debugDeterministicCursor = true across tests that do not specifically assert on blink dynamics.
For blink tests, explicitly verify cursor visibility at the midpoint of the blink cycle (250ms) rather than right at the boundary (500ms).

D. Subpixel Tolerance for 3D Perspective Transforms (`transform_test.dart`)

Allow a 1-pixel delta tolerance in matchesReferenceImage(imageWithCompositing), or reduce the angular sweep density from 80 iterations (step: 0.01) to 16 iterations (step: 0.05), preserving full 3D projection coverage while eliminating 80% of the statistical flake surface area.

Reassigning to team-engine since these tests are owned by the engine team.

Mitigate snapshot race in 3D transform test by reducing loop density (f…

Revert: Mitigate snapshot race in 3D transform test by reducing loop …

Navigation Menu

Repository navigation

Mac framework_tests_impeller is too big and flaky #193689

Hourly Distribution of Try Failures (Pacific Time)

Root Causes per Failing Suite

Evidence & Recent Build Runs

A. Decouple Animation Loops from Host Clocks (`progress_indicator_test.dart`)

B. Bound Fling Physics and Replace `pumpAndSettle` (`draggable_scrollable_sheet_test.dart`)

C. Stabilize Blinking Cursor Tests (`editable_text_cursor_test.dart`)

D. Subpixel Tolerance for 3D Perspective Transforms (`transform_test.dart`)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Mac framework_tests_impeller is too big and flaky #193689

Description

Hourly Distribution of Try Failures (Pacific Time)

Root Causes per Failing Suite

Evidence & Recent Build Runs

Activity

andywolff commented on Oct 1, 2026

A. Decouple Animation Loops from Host Clocks (progress_indicator_test.dart)

B. Bound Fling Physics and Replace pumpAndSettle (draggable_scrollable_sheet_test.dart)

C. Stabilize Blinking Cursor Tests (editable_text_cursor_test.dart)

D. Subpixel Tolerance for 3D Perspective Transforms (transform_test.dart)

eyebrowsoffire commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

A. Decouple Animation Loops from Host Clocks (`progress_indicator_test.dart`)

B. Bound Fling Physics and Replace `pumpAndSettle` (`draggable_scrollable_sheet_test.dart`)

C. Stabilize Blinking Cursor Tests (`editable_text_cursor_test.dart`)

D. Subpixel Tolerance for 3D Perspective Transforms (`transform_test.dart`)