@mattpap Do you have any suggestions for testing this?

Edit: Also whether column.change is too general for a layout invalidation.

Like this:

Just reiterating that the current process for adding the necessary artifact is quite fraught. This branch was only a ~week old but there were new tests on branch-2.4 that showed up in the downloaded report. Just unzipping the report added the artifacts for those newer tests that showed up as new files to add. A rebase failed while the files existed, and deleting them is difficult due to the special characters in the filenames. Eventually resorted to fit clean -xfd in the tests dir to be able to rebase and re-unzip. I can figure all this out but I think it is a very high bar for ay new contributor. cc @tcmetzger for visibility in to these possibilities.

Given that effectively you're not testing this branch, but a merge, then appearance of files from that merge shouldn't be surprising. In fact it can go much worse that that. A week old PR is actually quite old given there were 19 PRs merged during this timespan. I rebase my PRs pretty much every time I push (assuming there are any changes to pull in). This not only helps me to avoid major conflicts (especially after PRs like #11503), but also makes sure that I'm fully aware what I'm testing. I doubt there is much we can do to improve this, as long as we test merges, that wouldn't be prone to other issues.

(...) deleting them is difficult due to the special characters in the filenames (...)

Which exactly? Troublesome (to a shell) characters are encoded (or dropped) to make it easier to operate on baseline files.

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.