Currently, Travis will check if tests are flaky and if so fail the build. Otherwise, it is considered a pass and surfaced as a comment linking to details. Example: #7472

In order to allow test writers to more easily understand the impact of their changes, we should also clearly surface when a new test is failing, or when an existing test goes from passing to failing.

To do this, Travis also has to run the tests once without the changes applied.