Repository navigation
Description
@hannesa2, the screenshot comparison in BuildPR.yml goes red on PRs that change no UI at all. Latest example is the diff posted on #625, where all four screenshots differ.
What actually differs between the committed baseline and the current run:
- top commit
c161449tagged1.4->dec174btagged1.5.7 - the whole top of the list (
Merge pull request #531->#614, February dates -> September dates) - the status bar:
3383 commits loaded->3523 commits loaded
Why. The fixture repo is pinned only at HEAD:
gitx/.github/workflows/BuildPR.yml
Lines 118 to 122 in b0dc7de
| - name: Checkout fixed repo snapshot for screenshots | |
| if: ${{ env.variableSet != '' && matrix.abi == 'arm64' && github.event_name == 'pull_request' }} | |
| run: | | |
| git clone --no-checkout "$GITHUB_WORKSPACE" /tmp/gitx-screenshot-repo | |
| git -C /tmp/gitx-screenshot-repo checkout 52ff1051d0c8e5cf3ee5 |
The clone still carries every branch, remote branch and tag; the history view runs with the "All" filter; and the status bar counts every commit reachable from any ref. So the baselines rot on every merge to master, not once per release. Re-baselining on each release tag would not hold either, since the next merge breaks it the same day.
Two ways out.
- Freeze the fixture and re-baseline once:
git clone --no-checkout --no-tags "$GITHUB_WORKSPACE" /tmp/gitx-screenshot-repo
cd /tmp/gitx-screenshot-repo
git checkout --detach 52ff1051d0c8e5cf3ee5
git branch -f screenshot-fixture 52ff1051d0c8e5cf3ee5
git for-each-ref --format='%(refname)' refs/heads refs/remotes refs/tags \
| grep -v refs/heads/screenshot-fixture | xargs -n1 git update-ref -d- Stop committing baselines at all: in the same job, build the app at the latest tag, run the UI tests against the same clone, and compare those PNGs against the ones from the PR build. Drift cancels out, because both sets are taken at the same moment against the same fixture, and the question becomes "did this PR change the UI" instead of "does the UI still match a picture from February".
scripts/Screenshots-*then leaves the repo, and so does every re-baseline commit.
Costs of (2), so it is a fair comparison:
- a second build per screenshot run - the arm64 job is ~8 minutes today, mostly dependencies
- a new comparison step:
screenShotCompare.shis built ongit diff-image, which diffs the working tree against HEAD and so only works with committed baselines. Comparing two directories means calling ImageMagick directly, which the job already installs. The upload-and-comment half of the script is unaffected. - a new failure mode: if the app at the latest tag stops building on a newer Xcode, every PR goes red for an unrelated reason. A failed baseline build should skip the comparison rather than fail the PR.
Caching the baseline PNGs keyed on <latest tag>-<fixture commit>-<arch> would build the baseline once per release instead of once per PR, which gets (2) at close to today's CI cost.
I lean to (2) with the cache. Want me to open a PR?
Activity
hannesa2 commentedon Sep 13, 2026
Yes please !
goneng commentedon Sep 13, 2026
(I'm not sure my own suggested solutions are the right way to go... will investigate some more)