Golden tests do not write out failure images on a size mismatch #141488
Description
Steps to reproduce
When using matchesGoldenFile / LocalComparisonOutput it will output failure images when the test image does not match the master image. However, this only occurs when the test/master images are identical sizes. If they are mismatched sizes no failure images are written out. This can be frustrating, as we had a test image that was clearly twice as large as it should have been, but it wasn't easy to see the failure case.
- Write some code like so:
await expectLater(
find.text('Save'),
matchesGoldenFile('save.png'),
);- Change the size of the 'Save' widget, or maybe easier change 'save.png' to be a different size.
- Expect a error like this:
The following assertion was thrown while running async test code:
Golden "goldens/save.png": Pixel test failed, image sizes do not match.
Master Image: 200 X 100
Test Image: 400 X 200
- Look in the
failuresdirectory forsave_masterImage.png,save_testImage.png, etc, and be surprised they don't exist.
Expected results
Expected failures/save_masterImage.png, failures/save_testImage.png to exist.
Actual results
They don't exist.
Code sample
Code sample
[Paste your code here]Screenshots or Video
Screenshots / Video demonstration
[Upload media here]
Logs
Logs
[Paste your logs here]Flutter Doctor output
Doctor output
❯ flutter doctor -v
[✓] Flutter (Channel stable, 3.16.8, on macOS 14.2.1 23C71 darwin-arm64, locale en-GB)
• Flutter version 3.16.8 on channel stable at /opt/homebrew/Caskroom/flutter/3.16.0/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 67457e669f (8 days ago), 2024-01-16 16:22:29 -0800
• Engine revision 6e2ea58a5c
• Dart version 3.2.5
• DevTools version 2.28.5
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
• Android SDK at /Users/bramp/Library/Android/sdk
• Platform android-34, build-tools 34.0.0
• Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 15.2)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Build 15C500b
• CocoaPods version 1.14.3
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2022.3)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
[✓] VS Code (version 1.85.2)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.81.20231227
[✓] Connected device (2 available)
• macOS (desktop) • macos • darwin-arm64 • macOS 14.2.1 23C71 darwin-arm64
• Chrome (web) • chrome • web-javascript • Google Chrome 120.0.6099.234
[✓] Network resources
• All expected network resources are available.
• No issues found!