Visual regression fails when prompt changes alter the context-token counter #1679

Summary

The Visual Regression workflow has failed consistently since run #2252, even on unrelated pull requests. Only the extension-host-visual job fails; webview-visual and theme-fixtures continue to pass.

The failure is not a container or rendering regression. The chat-dark Electron snapshot includes the live context-token counter. PR #1505 changed the generated system prompt and reduced the deterministic test request from 4.2k / 1.0m to 3.1k / 1.0m. Those changed digits account for the 10-pixel snapshot difference reported on all three Playwright retries.

Evidence

• Last successful run: https://github.com/Zoo-Code-Org/Zoo-Code/actions/runs/35299850259
• First failing run: https://github.com/Zoo-Code-Org/Zoo-Code/actions/runs/35347538950
• Cutoff commit: a0f2e03
• Failing assertion: apps/vscode-e2e/src/visual/electron.visual.ts:241
• Snapshot: apps/vscode-e2e/src/visual/__screenshots__/electron-chat-dark-sidebar.png

The workflow container is pinned to an immutable Playwright 1.62.1 digest, and successful runs immediately before the cutoff used the same image.

Expected behavior

Prompt-only changes that alter token usage should not cause the visual-regression workflow to fail when the rendered layout and styling remain correct.

Suggested fix

Short term: inspect the extension-host-visual-regression artifact and accept the intentional electron-chat-dark-sidebar.png baseline change.

Long term: normalize or mask the context-token value before capturing the sidebar screenshot, while continuing to snapshot the surrounding progress UI. This prevents future system-prompt changes from creating unrelated visual-regression failures.