Skip to content

Test randomization for framework_tests_shard failing on skia gold #192992

New issue
New issue
Closed
Closed
Test randomization for framework_tests_shard failing on skia gold#192992
#195716 (+2)
Assignees
andywolff
Labels
P0Critical issues such as a build break or regressione: impellerImpeller rendering backend issues and features requests

Description

@jtmcdole
jtmcdole
opened on Sep 18, 2026

This is a real breakage, but stemming from (I assume) the randomization of test ordering.

Failure build: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20framework_tests_impeller/14358/overview

Randomization:

 RUNNING: cd packages/flutter; ../../bin/flutter test --reporter=expanded --file-reporter=json:/Volumes/Work/s/w/ir/x/t/metrics_1789738816684676.json --test-randomize-ordering-seed=20260918 --fatal-warnings --enable-impeller

Golden image: https://flutter-gold.skia.org/detail?grouping=name%3Dcupertino.cupertinoActionSheet.press-drag%26source_type%3Dflutter&digest=1abad162be65fd60cd499b61798396b4

Activity

  1. jtmcdole
    assigned
    justinmc
    on Sep 18, 2026
  2. jtmcdole
    added
    P0Critical issues such as a build break or regression
    on Sep 18, 2026

  3. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    It's this test:

    flutter/packages/flutter/test/cupertino/action_sheet_test.dart

    Line 852 in 12384f9

    testWidgets('Taps on a button can be slided to other buttons', (WidgetTester tester) async {

    Trying to repro locally. I'm guessing this is an old bug that a specific random seed just happened to trigger now, in which case it would also occur in flutter/packages.

  4. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    I can reproduce this on Mac but not on Linux. The --enable-impeller flag is key, but the seed does not matter. So this fails for me on master on my Mac:

    flutter test --enable-impeller test/cupertino/action_sheet_test.dart

  5. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    https://flutter-gold.skia.org/detail?grouping=name%3Dcupertino.cupertinoActionSheet.press-drag%26source_type%3Dflutter&digest=d85a7c77d7e7fc4bdb7a974d238aad80

    Image

  6. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    This fails back on 3.47.0. Did we recently add the --enable-impeller flag on Mac in CI?

  7. jtmcdole commented on Sep 18, 2026

    @jtmcdole
    jtmcdole
    on Sep 18, 2026
    MemberAuthor

    This has been passing in the past, but on the dashboard it recently started flaking.

  8. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    I checked out a bunch of random commits back to March or so and couldn't find one that passed for me. Though by the time I got to March a bunch of tests were failing (maybe pre-impeller or something).

    Edit I'm suspecting this is due to how goldens work local vs. on CI. I think there are lots of approved goldens for this test on CI?

  9. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    This may be suspicious for rendering flakiness:

    flutter/packages/flutter/lib/src/cupertino/dialog.dart

    Lines 1311 to 1314 in e4cb8ff

    child: ClipRSuperellipse(
    borderRadius: const BorderRadius.all(Radius.circular(12.0)),
    child: BackdropFilter(
    filter: ImageFilter.blur(

  10. justinmc commented on Sep 18, 2026

    @justinmc
    justinmc
    on Sep 18, 2026
    Contributor

    Ok I believe this is an impeller rendering bug. It doesn't seem to be a specific bot's hardware or setup since I see the test both passed and failed on the same bot recently.

    I was able to reproduce the flake locally on both my Mac and Linux machines. I produced 3 distinct digests in 20 runs on Linux and 8 in 20 runs on Mac. If I don't use impeller, it's always 1 deterministic digest on both platforms.

    Test script 
    #!/usr/bin/env bash
# Loop a golden test under Impeller and tally distinct rendered digests.
#   N=20 SEED=20260918 TARGET=test/cupertino/action_sheet_test.dart ./impeller_golden_loop.sh
set -u

N="${N:-20}"
SEED="${SEED:-20260918}"
TARGET="${TARGET:-test/cupertino/action_sheet_test.dart}"

FR="$(git rev-parse --show-toplevel)" || exit 1
cd "$FR/packages/flutter" || exit 1

OUT="$FR/bin/cache/pkg/skia_goldens/packages/flutter/test/cupertino"
RESULTS="/tmp/impeller_runs.txt"
LOGDIR="/tmp/impeller_logs"
mkdir -p "$LOGDIR" "$OUT"
: > "$RESULTS"

if [ ! -f "$TARGET" ]; then
  echo "ERROR: target not found: $PWD/$TARGET"
  exit 1
fi

hash_file() {
  if [ ! -f "$1" ]; then
    echo MISSING
  elif command -v md5 > /dev/null 2>&1; then
    md5 -q "$1"
  else
    md5sum "$1" | cut -d' ' -f1
  fi
}

hash_all() {
  if command -v md5 > /dev/null 2>&1; then
    cat "$OUT"/*.png 2>/dev/null | md5 -q
  else
    cat "$OUT"/*.png 2>/dev/null | md5sum | cut -d' ' -f1
  fi
}

echo "flutter root : $FR"
echo "target       : $TARGET"
echo "seed         : $SEED"
echo "golden output: $OUT"
echo "run logs     : $LOGDIR"
echo

i=1
while [ "$i" -le "$N" ]; do
  rm -f "$OUT"/*.png
  if [ "${IMPELLER:-1}" = "1" ]; then
    "$FR/bin/flutter" test --update-goldens --enable-impeller "--test-randomize-ordering-seed=$SEED" "$TARGET" > "$LOGDIR/run_$i.log" 2>&1
  else
    "$FR/bin/flutter" test --update-goldens "--test-randomize-ordering-seed=$SEED" "$TARGET" > "$LOGDIR/run_$i.log" 2>&1
  fi
  pd="$(hash_file "$OUT/cupertinoActionSheet.press-drag.png")"
  pr="$(hash_file "$OUT/cupertinoActionSheet.pressed.png")"
  al="$(hash_all)"
  cp "$OUT/cupertinoActionSheet.press-drag.png" "$LOGDIR/run${i}_press-drag_$pd.png" 2>/dev/null
  cp "$OUT/cupertinoActionSheet.pressed.png" "$LOGDIR/run${i}_pressed_$pr.png" 2>/dev/null
  printf '%2d  press-drag=%s  pressed=%s  all=%s\n' "$i" "$pd" "$pr" "$al" | tee -a "$RESULTS"
  if [ "$i" -eq 1 ] && [ "$pd" = MISSING ]; then
    echo
    echo "ERROR: first run wrote no goldens. Last 25 lines of $LOGDIR/run_1.log:"
    tail -25 "$LOGDIR/run_1.log"
    exit 1
  fi
  i=$((i + 1))
done

echo
echo "press-drag digests:"
awk '{print $2}' "$RESULTS" | sort | uniq -c | sort -rn
echo
echo "pressed digests:"
awk '{print $3}' "$RESULTS" | sort | uniq -c | sort -rn
echo
echo "all-goldens aggregate:"
awk '{print $4}' "$RESULTS" | sort | uniq -c | sort -rn
    20 runs on Mac 
    flutter root : /Users/jmccandless/Projects/flutter
target       : test/cupertino/action_sheet_test.dart
seed         : 20260918
golden output: /Users/jmccandless/Projects/flutter/bin/cache/pkg/skia_goldens/packages/flutter/test/cupertino
run logs     : /tmp/impeller_logs

 1  press-drag=816371a00355e5261bcbecce7dfe3626  pressed=13955439f2c18d6ebcf88a0f34b47872  all=cf16c42b2f21f69aa30c2d9e75e0b991
 2  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
 3  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
 4  press-drag=738d150aa5224400e73d5d80ab66d07e  pressed=13955439f2c18d6ebcf88a0f34b47872  all=84e6cd3c49f2ab21e0427bc2ecaab3ed
 5  press-drag=4c52f271232e250127515a2f011d10d3  pressed=13955439f2c18d6ebcf88a0f34b47872  all=235e64d26c0a400840c8792e28740843
 6  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
 7  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
 8  press-drag=4c52f271232e250127515a2f011d10d3  pressed=13955439f2c18d6ebcf88a0f34b47872  all=235e64d26c0a400840c8792e28740843
 9  press-drag=816371a00355e5261bcbecce7dfe3626  pressed=13955439f2c18d6ebcf88a0f34b47872  all=cf16c42b2f21f69aa30c2d9e75e0b991
10  press-drag=816371a00355e5261bcbecce7dfe3626  pressed=13955439f2c18d6ebcf88a0f34b47872  all=cf16c42b2f21f69aa30c2d9e75e0b991
11  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
12  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
13  press-drag=74595ae95153b4edeb3dbd5e2a9c79a4  pressed=13955439f2c18d6ebcf88a0f34b47872  all=64f9b1ae5428c6a064c2f7f483c08d00
14  press-drag=816371a00355e5261bcbecce7dfe3626  pressed=13955439f2c18d6ebcf88a0f34b47872  all=cf16c42b2f21f69aa30c2d9e75e0b991
15  press-drag=f3fb0d885696fee556d4249d75497d1d  pressed=13955439f2c18d6ebcf88a0f34b47872  all=34d11ab4eaca895bd1db1456df2361e7
16  press-drag=bdfb535315ebbbe64ffdab191490cb61  pressed=13955439f2c18d6ebcf88a0f34b47872  all=b822ad36b7726509e5cff25d57fa3951
17  press-drag=816371a00355e5261bcbecce7dfe3626  pressed=13955439f2c18d6ebcf88a0f34b47872  all=cf16c42b2f21f69aa30c2d9e75e0b991
18  press-drag=4c52f271232e250127515a2f011d10d3  pressed=13955439f2c18d6ebcf88a0f34b47872  all=235e64d26c0a400840c8792e28740843
19  press-drag=398849dee6c4e355b7817d8a4940ec23  pressed=13955439f2c18d6ebcf88a0f34b47872  all=e8ad4e4ccf9556e1f384c0056def7847
20  press-drag=8f0d4cd6e175b9e9669c5e42687f454d  pressed=13955439f2c18d6ebcf88a0f34b47872  all=9da93e247dfdfbadf69bdb659376e1e8

press-drag digests:
   7 press-drag=bdfb535315ebbbe64ffdab191490cb61
   5 press-drag=816371a00355e5261bcbecce7dfe3626
   3 press-drag=4c52f271232e250127515a2f011d10d3
   1 press-drag=f3fb0d885696fee556d4249d75497d1d
   1 press-drag=8f0d4cd6e175b9e9669c5e42687f454d
   1 press-drag=74595ae95153b4edeb3dbd5e2a9c79a4
   1 press-drag=738d150aa5224400e73d5d80ab66d07e
   1 press-drag=398849dee6c4e355b7817d8a4940ec23

pressed digests:
  20 pressed=13955439f2c18d6ebcf88a0f34b47872

all-goldens aggregate:
   7 all=b822ad36b7726509e5cff25d57fa3951
   5 all=cf16c42b2f21f69aa30c2d9e75e0b991
   3 all=235e64d26c0a400840c8792e28740843
   1 all=e8ad4e4ccf9556e1f384c0056def7847
   1 all=9da93e247dfdfbadf69bdb659376e1e8
   1 all=84e6cd3c49f2ab21e0427bc2ecaab3ed
   1 all=64f9b1ae5428c6a064c2f7f483c08d00
   1 all=34d11ab4eaca895bd1db1456df2361e7
    20 runs on Linux 
    flutter root : /usr/local/google/home/jmccandless/Projects/flutter
target       : test/cupertino/action_sheet_test.dart
seed         : 20260918
golden output: /usr/local/google/home/jmccandless/Projects/flutter/bin/cache/pkg/skia_goldens/packages/flutter/test/cupertino
run logs     : /tmp/impeller_logs

 1  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
 2  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
 3  press-drag=87b9c4434a498040b9abbf6e5f594f8d  pressed=64c22f511688af6b8846664ad70a4d73  all=c8a1a5e832afb7b64a82592bca20df9c
 4  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
^B^N 5  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
 6  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
 7  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
 8  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
 9  press-drag=87b9c4434a498040b9abbf6e5f594f8d  pressed=64c22f511688af6b8846664ad70a4d73  all=c8a1a5e832afb7b64a82592bca20df9c
10  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
11  press-drag=87b9c4434a498040b9abbf6e5f594f8d  pressed=64c22f511688af6b8846664ad70a4d73  all=c8a1a5e832afb7b64a82592bca20df9c
12  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
13  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
14  press-drag=9c7c6116daf70a53439925ec43e8e192  pressed=64c22f511688af6b8846664ad70a4d73  all=b168155a51e24d7c85572cdefe4f2280
15  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
16  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
17  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
18  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
19  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644
20  press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9  pressed=64c22f511688af6b8846664ad70a4d73  all=78d79ddbc4ee0cdf9c8b9f6795a5d644

press-drag digests:
     16 press-drag=f399bcf1f9bd60fd6ee01a5fb7636ff9
      3 press-drag=87b9c4434a498040b9abbf6e5f594f8d
      1 press-drag=9c7c6116daf70a53439925ec43e8e192

pressed digests:
     20 pressed=64c22f511688af6b8846664ad70a4d73

all-goldens aggregate:
     16 all=78d79ddbc4ee0cdf9c8b9f6795a5d644
      3 all=c8a1a5e832afb7b64a82592bca20df9c
      1 all=b168155a51e24d7c85572cdefe4f2280

    Some images from these local runs:

    Linux Mac
    Looks right Image Image
    Looks washed out Image Image

    I'll bisect this, but it's slow going.

  11. justinmc
    added
    e: impellerImpeller rendering backend issues and features requests
    on Sep 18, 2026

  12. justinmc commented on Sep 21, 2026

    @justinmc
    justinmc
    on Sep 21, 2026
    Contributor

    One other weird screenshot that I've occasionally seen show up on Mac:

    Image

  13. justinmc commented on Sep 21, 2026

    @justinmc
    justinmc
    on Sep 21, 2026
    Contributor

    Bisected to: #191096

    CC @andywolff @gaaclarke @b-luk

  14. justinmc
    assigned
    gaaclarke
    and unassigned
    justinmc
    on Sep 21, 2026

  15. 7 remaining items

  16. gaaclarke commented on Sep 23, 2026

    @gaaclarke
    gaaclarke
    on Sep 23, 2026
    Member

    That sounds good to me with one caveat. Some of what you are talking about is meta testing, testing the tests. I would avoid that and stick to unit tests / integration tests with randomization, even if that means a failure is a flake.

    A new RenderTargetCache per test sounds like it may be easier to implement that synchronizing on clearing the cache. That seems like a more holistic solution too since it will cover any cache. I guess we'll have to see what that does to execution times.

  17. andywolff commented on Sep 24, 2026

    @andywolff
    andywolff
    on Sep 24, 2026
    Contributor

    I tried coming up with a minimal reproduction in a real app for the GPU synchronization issue, and couldn't reproduce it on either Android running Vulkan or on Mac. I closed the issue. Instead, I'll work on exposing a way to clear the render target cache in response to memory pressure #193015 and also clear it between tests.

  18. jesswrd
    added a commit that references this issue on Oct 6, 2026

    [Impeller] Add render target cache clearing and test hook (flutter#19…

    575f742
  19. chingjun
    added a commit that references this issue on Oct 7, 2026

    Reset flutter engine internal state in framework test postTest (flutt…

    c97791d
  20. auto-submit
    closed this as completedin #194314on Oct 21, 2026
  21. opensource-fork-sync
    added a commit that references this issue on Oct 21, 2026

    Reland square corners fix (flutter#194314)

    730d1b2
  22. LongCatIsLooong
    added a commit that references this issue on Oct 22, 2026

    Reland square corners fix (flutter#194314)

    433581c
  23. mdebbar
    added a commit that references this issue on Oct 27, 2026

    Revert: Reland square corners fix (flutter#195436)

    9ece365
  24. suryaKommana2662
    added 4 commits that reference this issue on Oct 29, 2026

    [Impeller] Add render target cache clearing and test hook (flutter#19…

    07b47f8

    Reset flutter engine internal state in framework test postTest (flutt…

    a78d0a7

    Reland square corners fix (flutter#194314)

    f9355e9

    Revert: Reland square corners fix (flutter#195436)

    ad8a038

  25. github-actions commented on Nov 4, 2026

    @github-actions
    github-actionsBot
    on Nov 4, 2026 – with GitHub Actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  26. github-actions
    locked as resolved and limited conversation to collaborators on Nov 4, 2026
  27. ChandraChaan
    added a commit that references this issue on Nov 10, 2026

    [Impeller] Reland clipped BackdropFilter square corners fix (flutter#…

    e1fee50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressione: impellerImpeller rendering backend issues and features requests

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions