Repository navigation
Description
(This fix touches the same area as #658 -
the previous PR is about which commit the baseline is built from while
this is about what the tests photograph.
We chose not to lend them together so not to make both harder to judge.)
The UI screenshot suite produces four PNGs, but only two distinct images.
Three of the four are byte-identical.
From the artifact of run 35387508172:
b29eb7bb... 00-commit-context-menu.png 172104
e39cb94e... 01-history-view.png 154827
e39cb94e... 02-main-window.png 154827
e39cb94e... 03-staging-view.png 154827
Same SHA-256, same size, same 890x546 dimensions. Only the context menu
shot is its own picture.
Cause
Two of the tests in GitXUITests/GitXScreenshotTests.m are the same test:
- (void)testMainWindowExists {
XCTAssertTrue([self waitForWindow], ...);
[self saveWindowScreenshotNamed:@"main-window"];
}
- (void)testHistoryTabScreenshot {
if (![self waitForWindow]) { return; }
[self saveWindowScreenshotNamed:@"history-view"];
}Neither navigates anywhere, and GitX opens on the history view, so
"main-window" is the history view under a second name.
testStagingTabScreenshot is a different failure. It clicks the Stage
button only if (stageButton.exists), and on a miss it falls through and
photographs the same unchanged window. The identical hash above shows the
lookup is missing in CI today.
testCommitContextMenuScreenshot has the same shape: three early returns
that skip the capture entirely. That one at least produces no file.
Why this matters
- A regression confined to the staging view cannot be caught, because the
"staging" image has never been the staging view. - Any change to the shared window reports as three differences, which
makes the PR comment look far worse than the change is and trains
people to skim past it. - All of this degrades silently. A missed button and a working button
produce the same green run. - The filenames carry a discovery-order index (
extract_screenshots.py
writesf'{i:02d}-{safe}.png'), so a test that skips its capture also
renumbers every file after it, and the next comparison sees everything
as changed.
Suggested fix
Have each test assert it reached the view before the shutter, rather than
falling through: fail on a missing Stage button instead of returning, and
either give testMainWindowExists a distinct view or drop its screenshot
and keep it as a launch assertion. Dropping the index prefix in favour of
the plain attachment name would also stop one skipped test from
renumbering the rest.
Expect the baselines to churn once this lands.
Activity
Fix the screenshot tests all photographing the same view (issue #676)
Fix the screenshot tests all photographing the same view (issue #676)
Fix the screenshot tests all photographing the same view (issue #676)