-
Notifications
You must be signed in to change notification settings - Fork 981
Description
The goal of this issue is to track developing a graphical testing system. Regression tests can't use image hashes because slight differences in systems (e.g., installed fonts or sub-pixel alignment changes) cause hashes to mismatch when changes are minor. An ideal graphical testing system would evaluate how "much" an image changes, and ideally highlight where.
I recall hearing an interview with someone describing how matplotlib runs graphics tests. They generate a bunch of test images (like the ScottPlot cookbook), save them, then after making code changes they generate a new set of images and perform comparisons on the old vs. new images by creating a black and white difference image. The difference image can evaluate how much changed graphically (how large the difference is), and point the developer to exactly where in the image changed.
Cookbook and test images should not be stored in this repository, but a new tool should be developed to create and store them locally (perhaps in a named and dated folder) and facilitate comparisons as described above.