Skip to content

PBGitWindowWiringTests fails when the app was left on the stage view #703

Closed

Description

@goneng

The tests expect the application to be showing the history view. The
screenshot tests, which run earlier in the same CI job, leave it showing
the stage view. Which view the window opens on is a preference that
outlives them both, so the wiring tests inherit the wrong one, and from
that point every rerun fails the same way.

Put briefly: #579 built the trap, #695 sprang it.

So the failure is a property of what ran before these tests, not of the
machine's load and not of the branch under test.

Split out of #643, which was filed for a different failure in the same
test class. See the note there.

The wiring

The window shows one content controller at a time, and
-changeContentController: loads a nib only for the controller it is
about to show, by asking that controller for its view. Which one is
shown is decided in PBGitSidebarController:

if ([PBGitDefaults showStageView])
    [self selectStage];
else
    [self selectCurrentBranch];

PBShowStageView is persisted in the app's defaults domain, and
PBGitSidebarController writes it whenever the Stage row is selected.
So when a GitX has been left on the stage view, these tests build a
window whose history nib never loads. commitController and
commitList stay nil, -historyListSelecting: sets content on nil,
and three assertions fail:

PBGitWindowWiringTests.m: ((history.singleCommitSelected) is true) failed
  -[testTheFocusedHistoryListNamesTheBranchOnItsSelectedCommit]
  -[testACommitCarryingTwoBranchesNamesNeither]
PBGitWindowWiringTests.m: ("(null)") is not equal to ("refs/heads/branch_one")

A diagnostic build reports the state as history: set, commitController: nil, content: 0, selectionIndexes: (null). An array controller answers
an empty index set rather than (null), so the outlet is nil rather
than the selection being empty.

How CI sets it

The screenshot tests drive the real application and switch to
View > Commit. Test methods run alphabetically, so
testStagingTabScreenshot runs last of the four and never switches
back, leaving the app on the stage view and the preference set in the
defaults domain shared by every GitX on the runner. The workflow then
launches that app in Build baseline and run its UI tests, and the
later Run tests step re-runs GitXTests in the same job, where they
inherit it.

Evidence

One CI job, one build, one variable at a time:

condition result
Run unit tests, before the baseline app launches 148/148 pass
the same isolated make unit-test, after it launches exit=2
whole defaults domain wiped first exit=0
only NSWindow Frame GitX removed exit=2
only PBShowStageView turned off exit=0

It reproduces on an idle machine, macOS 15 included, with no CI involved:

defaults write net.phere.GitX PBShowStageView -bool YES
make unit-test ARCH=arm64

Two of the six cases fail and the other four pass. The failing cases
take well under a second, which is what rules out load: a load problem
shows up in the 30 second waits, not in these assertions.

Two details that disguised it

  • Only the arm64 leg runs any tests at all - every test step is gated on
    matrix.abi == 'arm64' - so the intel leg cannot go red and its green
    says nothing. That is the whole "always arm64" pattern.
  • The baseline screenshot cache is only saved by a job that succeeds, so
    a failing pull request rebuilds the baseline on every run and keeps
    failing, while one green run makes later runs skip the baseline
    entirely and pass for the wrong reason.

Fix

#702 makes the suite say which view it needs rather than inherit it:
save PBShowStageView in -setUp, turn it off before the window is
built, and put the old value back in -tearDown.

Still open after that: the screenshot tests continue to leak the
preference into the shared domain, so any other suite that depends on
which view the window opens on stays exposed. Fixing that at source
belongs with the screenshot tests, alongside #700.

Activity

  1. goneng commented on Sep 20, 2026

    @goneng
    CollaboratorAuthor

    How this was found, and what it overturns. Recorded here because three
    things that were said about it along the way were wrong, and the wrong
    versions are more memorable than the right one.

    Corrections

    • Not the machine's load. That is what PBGitWindowWiringTests fails when the machine is under load #643 was filed as, and this
      failure was assumed to be the same thing for days. It is not: the
      failing cases take well under a second, and it reproduces on an idle
      machine. A load problem shows up in the 30 second waits instead.
    • Not arm64, and not the OS. Every test step in the workflow is
      gated on matrix.abi == 'arm64', so the intel leg runs no tests at
      all and its green says nothing. The "always arm64" pattern is that
      gate. It was also blamed on macOS 26 for a while, until it reproduced
      on macOS 15.
    • Not the branch under test. A bisect appeared to pin it on Support choosing the date-format for the history pane's commit date #693's
      date column formatter. It did not: the formatter is present in runs
      that pass once the preference is cleared, and the isolated suite
      passes with it earlier in the very same job. Support choosing the date-format for the history pane's commit date #693 was only implicated
      because it touches a xib, which makes the "can this PR affect the UI"
      gate true and so runs the baseline machinery at all.

    Two earlier comments on #643 proposed mechanisms that are also wrong: a
    late NSUserDefaultsDidChangeNotification, and -historyListSelecting:
    racing a KVO hop on the array controller's selection. Both were deleted
    rather than left to mislead. The second is wrong at the root - the
    selection is never reached, because the array controller is nil. The
    KVO claim was modelled separately across seven array controller
    configurations and the selection lands synchronously in all of them.

    What settled it

    Narrowing one variable at a time inside a single CI job, rather than
    comparing runs. The same isolated make unit-test command passes early
    in the job and fails later in it, which rules out the build, the
    machine and the branch in one step and leaves only what happened in
    between. From there, wiping the defaults domain fixed it, removing the
    saved window frame did not, and turning off PBShowStageView alone did.

    Worth carrying for any future CI probe here: the baseline screenshot
    cache is only saved by a job that succeeds. A failing pull request
    rebuilds the baseline every run and keeps failing, while one green run
    makes later runs skip the baseline and pass for the wrong reason. That
    happened mid-investigation and produced a false green.

    Relationship to #700

    #700 reaches the same two commits - #695 as the trigger - by a different
    route, and treats the cause as step duration. It replaces the menu bar
    traversal with key equivalents, which is worth having on its own, but
    testStagingTabScreenshot still switches to the commit view and never
    switches back, so the preference is still left set. Its own CI run fails
    the same three assertions in the same two cases. The two changes are
    complementary; neither replaces the other.

  2. added a commit that references this issue on Sep 21, 2026
  3. goneng commented on Sep 21, 2026

    @goneng
    CollaboratorAuthor

    Fixed by #702, merged as be838ee: the suite turns PBShowStageView off in -setUp and puts the old value back in -tearDown, so it no longer inherits whichever view ran before it.

    That PR names this issue in its title but without a closing keyword, so the issue stayed open after it merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testTest related

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions