Use case

When golden tests fail, they output the percentage difference between the golden image and the current image. Sometimes a percentage is a useful metric. However, when trying to quickly identify likely flakes, especially for large golden images, it's much more helpful to know the number of mismatched pixels.

I often run into golden failures that have a 0.00% difference. Those failures have represented actual failures that needed attention, as well as flakes that didn't really mean anything of significance.

Proposal

While there is no sure fire way to differentiate between a real failure and a flaky failure, it would be helpful to know the pixel difference along with the percentage difference. For example, a 0.00%, 10px difference is almost certainly a flake, whereas a 0.00%, 150px difference may very well be a legitimate failure.

Additionally, I have my own implementation of a golden matcher that lets me pass a specific number of pixels that are allowed to mismatch. I use this matcher when I have a golden test that I can't seem to get to pass normally, but doesn't seem to be a legitimate failure. I need to know how many pixels to pass in for that allowance. Because Flutter doesn't report the pixel mismatch, this process first requires that I fail the test with my custom matcher to figure out the pixel difference and then apply it. It would be nice to cut one of the steps out of that process.