Skip to content

Screenshot tests capture the same view three times #676

Closed

Description

@goneng

(This fix touches the same area as #658 -
the previous PR is about which commit the baseline is built from while
this is about what the tests photograph.
We chose not to lend them together so not to make both harder to judge.)


The UI screenshot suite produces four PNGs, but only two distinct images.
Three of the four are byte-identical.

From the artifact of run 35387508172:

b29eb7bb...  00-commit-context-menu.png   172104
e39cb94e...  01-history-view.png          154827
e39cb94e...  02-main-window.png           154827
e39cb94e...  03-staging-view.png          154827

Same SHA-256, same size, same 890x546 dimensions. Only the context menu
shot is its own picture.

Cause

Two of the tests in GitXUITests/GitXScreenshotTests.m are the same test:

- (void)testMainWindowExists {
    XCTAssertTrue([self waitForWindow], ...);
    [self saveWindowScreenshotNamed:@"main-window"];
}

- (void)testHistoryTabScreenshot {
    if (![self waitForWindow]) { return; }
    [self saveWindowScreenshotNamed:@"history-view"];
}

Neither navigates anywhere, and GitX opens on the history view, so
"main-window" is the history view under a second name.

testStagingTabScreenshot is a different failure. It clicks the Stage
button only if (stageButton.exists), and on a miss it falls through and
photographs the same unchanged window. The identical hash above shows the
lookup is missing in CI today.

testCommitContextMenuScreenshot has the same shape: three early returns
that skip the capture entirely. That one at least produces no file.

Why this matters

  • A regression confined to the staging view cannot be caught, because the
    "staging" image has never been the staging view.
  • Any change to the shared window reports as three differences, which
    makes the PR comment look far worse than the change is and trains
    people to skim past it.
  • All of this degrades silently. A missed button and a working button
    produce the same green run.
  • The filenames carry a discovery-order index (extract_screenshots.py
    writes f'{i:02d}-{safe}.png'), so a test that skips its capture also
    renumbers every file after it, and the next comparison sees everything
    as changed.

Suggested fix

Have each test assert it reached the view before the shutter, rather than
falling through: fail on a missing Stage button instead of returning, and
either give testMainWindowExists a distinct view or drop its screenshot
and keep it as a launch assertion. Dropping the index prefix in favour of
the plain attachment name would also stop one skipped test from
renumbering the rest.

Expect the baselines to churn once this lands.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions