-
-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Reporting rendering regression on PRs #15918
Copy link
Copy link
Open
Labels
A-Build-SystemRelated to build systems or continuous integrationRelated to build systems or continuous integrationA-RenderingDrawing game state to the screenDrawing game state to the screenS-Needs-DesignThis issue requires design work to think about how it would best be accomplishedThis issue requires design work to think about how it would best be accomplished
Metadata
Metadata
Assignees
Labels
A-Build-SystemRelated to build systems or continuous integrationRelated to build systems or continuous integrationA-RenderingDrawing game state to the screenDrawing game state to the screenS-Needs-DesignThis issue requires design work to think about how it would best be accomplishedThis issue requires design work to think about how it would best be accomplished
Good news! Since #13248 (and #15894, #15911 and #15914) CI is now checking screenshots taken during example runs on macOS and Linux for changes.
The list of examples that are executed can be found there: https://github.com/bevyengine/bevy/tree/main/.github/example-run. At the moment, only 2 examples take a screenshot:
breakoutandload_gltfThe macOS run is done on every push to a PR, in the merge queue, and on main. You can find an example run there: https://github.com/bevyengine/bevy/actions/runs/11339649537/job/31534799604
The Linux run is done in the merge queue and on main. You can find an example run there: https://github.com/bevyengine/bevy/actions/runs/11339392720/job/31534437159
As you may have noticed, they both fail: this is because the example
load_gltfis a tiny bit time dependent for the light movement, so not fully deterministic. Failures are currently not reported or blocking for PRs.It is possible to make examples deterministic even when relying on time by applying the patches there, or using the
example-showcasetool.You can see a list of run here: https://pixel-eagle.com/project/B04F67C0-C054-4A6F-92EC-F599FEC2FD1D
I have some improvements planned for the reporting, but now my question to you, dear contributors: which examples should be ran and checked for changes in CI?
This issue will be fixed by changing the list of screenshots compared to a fully deterministic one (either by selecting deterministic examples, or by enabling stable time with the patches), and changes being reported by a comment on PRs (not blocking merging yet)