Proposed change: We need to fix test execution as introduced in #3888 so as not to leave behind files that are (git) version controlled.

Motivation: PRs like #3890 are the consequence of regression-test executing leaving behind modified files. Similarly, developers running tests locally before pushing changes will inadvertently push unrelated changes to the repository.