Skip to content

CI: Screenshot baselines drift on every merge to master #628

Closed

Description

@goneng

@hannesa2, the screenshot comparison in BuildPR.yml goes red on PRs that change no UI at all. Latest example is the diff posted on #625, where all four screenshots differ.

What actually differs between the committed baseline and the current run:

  • top commit c161449 tagged 1.4 -> dec174b tagged 1.5.7
  • the whole top of the list (Merge pull request #531 -> #614, February dates -> September dates)
  • the status bar: 3383 commits loaded -> 3523 commits loaded

Why. The fixture repo is pinned only at HEAD:

- name: Checkout fixed repo snapshot for screenshots
if: ${{ env.variableSet != '' && matrix.abi == 'arm64' && github.event_name == 'pull_request' }}
run: |
git clone --no-checkout "$GITHUB_WORKSPACE" /tmp/gitx-screenshot-repo
git -C /tmp/gitx-screenshot-repo checkout 52ff1051d0c8e5cf3ee5

The clone still carries every branch, remote branch and tag; the history view runs with the "All" filter; and the status bar counts every commit reachable from any ref. So the baselines rot on every merge to master, not once per release. Re-baselining on each release tag would not hold either, since the next merge breaks it the same day.

Two ways out.

  1. Freeze the fixture and re-baseline once:
git clone --no-checkout --no-tags "$GITHUB_WORKSPACE" /tmp/gitx-screenshot-repo
cd /tmp/gitx-screenshot-repo
git checkout --detach 52ff1051d0c8e5cf3ee5
git branch -f screenshot-fixture 52ff1051d0c8e5cf3ee5
git for-each-ref --format='%(refname)' refs/heads refs/remotes refs/tags \
  | grep -v refs/heads/screenshot-fixture | xargs -n1 git update-ref -d
  1. Stop committing baselines at all: in the same job, build the app at the latest tag, run the UI tests against the same clone, and compare those PNGs against the ones from the PR build. Drift cancels out, because both sets are taken at the same moment against the same fixture, and the question becomes "did this PR change the UI" instead of "does the UI still match a picture from February". scripts/Screenshots-* then leaves the repo, and so does every re-baseline commit.

Costs of (2), so it is a fair comparison:

  • a second build per screenshot run - the arm64 job is ~8 minutes today, mostly dependencies
  • a new comparison step: screenShotCompare.sh is built on git diff-image, which diffs the working tree against HEAD and so only works with committed baselines. Comparing two directories means calling ImageMagick directly, which the job already installs. The upload-and-comment half of the script is unaffected.
  • a new failure mode: if the app at the latest tag stops building on a newer Xcode, every PR goes red for an unrelated reason. A failed baseline build should skip the comparison rather than fail the PR.

Caching the baseline PNGs keyed on <latest tag>-<fixture commit>-<arch> would build the baseline once per release instead of once per PR, which gets (2) at close to today's CI cost.

I lean to (2) with the cache. Want me to open a PR?

Activity

  1. self-assigned this
    on Sep 13, 2026
  2. hannesa2 commented on Sep 13, 2026

    @hannesa2
    Contributor

    Yes please !

  3. goneng commented on Sep 13, 2026

    @goneng
    CollaboratorAuthor

    (I'm not sure my own suggested solutions are the right way to go... will investigate some more)

  4. assigned and unassigned on Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions