Skip to content

CI: the unit test suite runs twice on every pull request #705

Closed

Description

@goneng

GitXTests is run twice per pull request, in two different steps, and
the second run is incidental rather than intended.

  • Run unit tests runs make unit-test, which is
    -only-testing:GitXTests.
  • Run tests runs make all-tests, which tests every target in the
    scheme - so it runs GitXTests again on its way to GitXUITests.

The workflow already notes this, in a TODO above the second step:

Consider running make ui-test here rather than make all-tests. The
scheme tests every target, so this repeats the GitXTests run of the
"Run unit tests" step above; ui-test narrows it to GitXUITests,
which is all the screenshots need.

What it actually costs

Less than it looks, and worth being honest about: the build in the
second step is needed for the UI tests anyway, so the duplicate is the
test execution alone, about 14 seconds on a recent run. Narrowing the
step to make ui-test would save that, not minutes.

The better reason to fix it

The two runs are not equivalent, and that has already caused confusion.
The Run unit tests step happens before Build baseline and run its UI tests; the Run tests step happens after it. Since launching the
application leaves state in the shared defaults domain, the same suite
can pass in the first step and fail in the second, on one build, in one
job. That is precisely how #703 presented, and the duplicate run is what
made it look like a flake rather than a dependency on what ran before.

Two identical Executed 148 tests summaries in one log, which can
disagree, is also just hard to read.

Narrowing Run tests to make ui-test removes the ambiguity. It would
not have fixed #703 - #702 did that, by making the suite independent of
the preference - but it would have made it much easier to see.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions