Skip to content

toMatchScreenshot should not generate screenshots if they pass the comparator #9158

@djrenren

Description

@djrenren

Describe the bug

When vitest is run with --update, toMatchScreenshot unconditionally emits new screenshots (even when the test would otherwise pass). This means that if you have a non-deterministic UI element, every run with --update produces a new screenshot.

This causes problems when used in CI (where it's recommended to run with --update). The presence of updated screenshots does not indicate the need to update the screenshots. This means we have to run CI twice:

  • First, to see if we need to update the screenshot files
  • Second, to get those updates. Even then, we have to be clever not to include unecessary changes.

It seems that --update shouldn't unconditionally update. If you want unconditional updates, it's easy enough to wipe the __screenshots__ directory and run the tests.

Reproduction

  1. Clone the example repo: git clone https://github.com/djrenren/vitestScreenshotTestFailure
  2. Run tests (successfully) cd vitestScreenshotTestFailure && npm test:browser
  3. Observe the screenshots have changed: git status

The example repo renders a single random pixel on a canvas. Enough to differ but not enough to fail the test.

System Info

System:
    OS: macOS 26.1
    CPU: (14) arm64 Apple M3 Max
    Memory: 2.05 GB / 36.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.16.0 - /Users/john/.nvm/versions/node/v22.16.0/bin/node
    npm: 10.9.2 - /Users/john/.nvm/versions/node/v22.16.0/bin/npm
    pnpm: 10.18.3 - /Users/john/.nvm/versions/node/v22.16.0/bin/pnpm
    bun: 1.1.29 - /Users/john/.bun/bin/bun
    Deno: 1.41.3 - /Users/john/.deno/bin/deno
  npmPackages:
    @vitejs/plugin-react: ^5.1.1 => 5.1.1 
    @vitest/browser-playwright: ^4.0.15 => 4.0.15 
    vitest: ^4.0.15 => 4.0.15 
    vitest-browser-react: ^2.0.2 => 2.0.2

Used Package Manager

npm

Validations

Metadata

Metadata

Assignees

Labels

feat: browserIssues and PRs related to the browser runnerp3-minor-bugAn edge case that only affects very specific usage (priority)

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions