Skip to content

Conversation

@tkattkat
Copy link
Collaborator

@tkattkat tkattkat commented Dec 16, 2025

why

Currently, we do not await the final screenshot when stop method is called, resulting in missing screenshots within the screenshot collector

what changed

changed stop be async, and added await to stop method calls

test plan


Summary by cubic

Make ScreenshotCollector.stop async and wait for the final screenshot so journeys include the last frame. Updated all agent tasks to await stop to prevent missing screenshots.

  • Bug Fixes

    • stop() now awaits the final capture and returns Promise<Buffer[]>.
    • Added try/catch around the final capture to avoid failing on errors.
    • Updated all task usages to await screenshotCollector.stop().
  • Migration

    • If you use ScreenshotCollector, change calls to await screenshotCollector.stop().

Written for commit fbdb1bb. Summary will update automatically on new commits.

@changeset-bot
Copy link

changeset-bot bot commented Dec 16, 2025

⚠️ No Changeset found

Latest commit: fbdb1bb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 21 files

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Dec 16, 2025

Greptile Overview

Greptile Summary

Fixed a bug where the final screenshot was not being captured before the stop() method returned. The ScreenshotCollector.stop() method previously used a fire-and-forget pattern with .catch() for the final screenshot capture, causing it to return before the screenshot was actually captured.

  • Changed stop() from synchronous to async (Promise<Buffer[]>)
  • Added proper await for final screenshot capture with try/catch error handling
  • Updated all 20 eval task files to await the stop() call

Confidence Score: 5/5

  • This PR is safe to merge - it's a straightforward async/await fix with proper error handling
  • The change is minimal, focused, and correctly fixes the bug. The pattern of making a method async and properly awaiting it is well-established. All call sites have been updated consistently.
  • No files require special attention

Important Files Changed

File Analysis

Filename Score Overview
packages/evals/utils/ScreenshotCollector.ts 5/5 Core fix: Made stop() method async and awaited final screenshot capture. Proper try/catch error handling ensures the final screenshot is captured before returning, fixing the reported bug of missing screenshots.
packages/evals/tasks/agent/alibaba_supplier_search.ts 5/5 Added await to screenshotCollector.stop() call to match the new async signature.
packages/evals/tasks/agent/onlineMind2Web.ts 5/5 Added await to screenshotCollector.stop() call to match the new async signature.
packages/evals/tasks/agent/webvoyager.ts 5/5 Added await to screenshotCollector.stop() call to match the new async signature.

Sequence Diagram

sequenceDiagram
    participant EvalTask
    participant ScreenshotCollector
    participant Page

    EvalTask->>ScreenshotCollector: start()
    loop Every interval
        ScreenshotCollector->>Page: screenshot()
        Page-->>ScreenshotCollector: Buffer
    end
    EvalTask->>ScreenshotCollector: await stop()
    ScreenshotCollector->>ScreenshotCollector: clearInterval()
    ScreenshotCollector->>Page: await screenshot() (final)
    Page-->>ScreenshotCollector: Buffer
    ScreenshotCollector-->>EvalTask: Buffer[]
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

21 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@tkattkat tkattkat merged commit 9d9745c into main Dec 16, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants