Update screenshot collector #1373

tkattkat · 2025-12-05T20:23:43Z

why

when transitioning to v3, we did not use the latest version of screenshot collector
screenshot collector currently fails due to not having page.on and page.off support for the load, and domcontentloaded events.

what changed

added latest version of screenshot collector

test plan

ran evals in cli with additional logging to also verify everything is working as expected

Summary by cubic

Updated the evals CLI screenshot collector to the latest version, adding image-diff filtering and a V3 event bus that emits agent screenshots. This reduces duplicate screenshots and stabilizes capture on v3 pages where navigation events are disabled.

New Features
- Skip similar screenshots using MSE/SSIM thresholds with sharp.
- Event bus integration: agents emit screenshots; collector can ingest them.
- Non-blocking initial/final captures and safer interval capture with error handling.
Dependencies
- Added sharp ^0.34.5 for image processing (evals and core).
- Patch bump via changeset for @browserbasehq/stagehand-evals.

^{Written for commit f4e90f8. Summary will update automatically on new commits.}

changeset-bot · 2025-12-05T20:23:47Z

🦋 Changeset detected

Latest commit: f4e90f8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@browserbasehq/stagehand-evals	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

cubic-dev-ai

No issues found across 4 files

greptile-apps · 2025-12-05T20:29:35Z

Greptile Overview

Greptile Summary

Updates the ScreenshotCollector utility for V3 compatibility and adds intelligent screenshot deduplication.

Switched from generic ScreenshotCapablePage interface to the concrete V3 Page type from @browserbasehq/stagehand
Added sharp library for image processing to enable screenshot deduplication via MSE (Mean Squared Error) and SSIM (Structural Similarity Index) comparisons
Disabled navigation event listeners (page.on("load"), etc.) since V3 pages don't support these yet - documented with TODO comment
Changed default captureOnNavigation from true to false to reflect current V3 limitations
Added new addScreenshot() method for manual screenshot additions with deduplication logic
Improved error handling with .catch() for async operations instead of fire-and-forget calls

Confidence Score: 4/5

This PR is safe to merge with minor considerations around the async final screenshot behavior
The changes are well-structured and address real V3 compatibility issues. The image deduplication logic is sound. One minor issue exists where the final screenshot in stop() may not be captured before returning, but this is unlikely to cause problems in practice since the collector is typically used for incremental captures.
packages/evals/utils/ScreenshotCollector.ts - review async final screenshot handling in stop() method

Important Files Changed

File Analysis

Filename	Score	Overview
packages/evals/utils/ScreenshotCollector.ts	4/5	Major update adding image deduplication via MSE/SSIM, switching to V3 Page type, and adding sharp dependency for image comparison. Minor issue with async final screenshot not being awaited in stop().
packages/evals/package.json	5/5	Added sharp ^0.34.5 dependency for image processing used in screenshot deduplication.
.changeset/beige-taxes-punch.md	5/5	Standard changeset for patch update to stagehand-evals package.
pnpm-lock.yaml	5/5	Lock file updates for sharp 0.34.5 and its native dependencies across platforms.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant ScreenshotCollector
    participant Page
    participant Sharp

    Caller->>ScreenshotCollector: start()
    ScreenshotCollector->>Page: screenshot() [initial]
    Page-->>ScreenshotCollector: Buffer
    ScreenshotCollector->>ScreenshotCollector: store screenshot
    
    loop Every interval (5000ms default)
        ScreenshotCollector->>Page: screenshot()
        Page-->>ScreenshotCollector: Buffer
        ScreenshotCollector->>Sharp: resize & compare (MSE)
        Sharp-->>ScreenshotCollector: MSE value
        alt MSE >= threshold
            ScreenshotCollector->>Sharp: calculate SSIM
            Sharp-->>ScreenshotCollector: SSIM value
            alt SSIM < threshold (different enough)
                ScreenshotCollector->>ScreenshotCollector: store screenshot
            end
        end
    end

    Caller->>ScreenshotCollector: stop()
    ScreenshotCollector->>Page: screenshot() [final, async]
    ScreenshotCollector-->>Caller: Buffer[]

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

packages/evals/utils/ScreenshotCollector.ts

package.json

miguelg719 · 2025-12-15T18:57:43Z

packages/evals/tasks/agent/onlineMind2Web.ts

    });

    const agent = v3.agent({
+      cua: true,


I will address conditional loading on a follow up pr

pirate · 2025-12-15T19:08:20Z

LGTM

@tkattkat

cubic-dev-ai bot reviewed Dec 5, 2025

View reviewed changes

greptile-apps bot reviewed Dec 5, 2025

View reviewed changes

packages/evals/utils/ScreenshotCollector.ts Show resolved Hide resolved

miguelg719 reviewed Dec 13, 2025

View reviewed changes

package.json Show resolved Hide resolved

miguelg719 force-pushed the update-screenshot-collector branch 2 times, most recently from d98af74 to 43a1cab Compare December 13, 2025 18:03

miguelg719 reviewed Dec 15, 2025

View reviewed changes

miguelg719 approved these changes Dec 15, 2025

View reviewed changes

pirate approved these changes Dec 15, 2025

View reviewed changes

miguelg719 force-pushed the update-screenshot-collector branch from 9cd72db to 43a1cab Compare December 15, 2025 19:29

tkattkat and others added 6 commits December 15, 2025 11:42

update screenshot collector

eae0d7a

update collector

be1c9a4

pnpm lock update

507f643

event bus driven screenshot collector

066cfc2

prettier ignores and proper root level

646ad52

Update package.json

f037f6b

miguelg719 force-pushed the update-screenshot-collector branch from 43a1cab to f037f6b Compare December 15, 2025 19:43

resolve conflict

f4e90f8

miguelg719 merged commit cadd192 into main Dec 15, 2025
15 checks passed

This was referenced Dec 15, 2025

Version Packages #1414

Merged

Version Packages CloudEngineHub/stagehand#1

Open

Version Packages kenchikuliu/stagehand#1

Open

Version Packages shaneholloman/stagehand#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update screenshot collector #1373

Update screenshot collector #1373

Uh oh!

tkattkat commented Dec 5, 2025 •

edited by cubic-dev-ai bot

Loading

Uh oh!

changeset-bot bot commented Dec 5, 2025 •

edited

Loading

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

greptile-apps bot commented Dec 5, 2025

Uh oh!

greptile-apps bot left a comment

Uh oh!

Uh oh!

Uh oh!

miguelg719 Dec 15, 2025

Uh oh!

pirate commented Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Update screenshot collector #1373

Update screenshot collector #1373

Uh oh!

Conversation

tkattkat commented Dec 5, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

why

what changed

test plan

Summary by cubic

Uh oh!

changeset-bot bot commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot commented Dec 5, 2025

Greptile Overview

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

miguelg719 Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

pirate commented Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tkattkat commented Dec 5, 2025 •

edited by cubic-dev-ai bot

Loading

changeset-bot bot commented Dec 5, 2025 •

edited

Loading