fix: CUA coordinating scaling issue #1430

shrey150 · 2025-12-18T00:05:47Z

why

Google, Microsoft, and OpenAI CUA clicks miss targets when using advanced stealth mode with Browserbase
Advanced stealth scales screenshots to 1288×711 but browser reports 2560×1305 viewport
GoogleCUAClient (and others) was scaling coordinates directly from Google's 0-1000 range to viewport dimensions without accounting for the screenshot→viewport size mismatch.

This forced users to disable advanced stealth mode (losing bot protection) to get accurate clicks.

what changed

CUA Clients:

Added actualScreenshotSize field to track screenshot dimensions
Added setScreenshotSize() method to update dimensions
Fixed normalizeCoordinates() with two-step scaling:
1. Convert 0-1000 range → screenshot space
2. Scale screenshot space → viewport space (handles different X/Y scales)

v3CuaAgentHandler.ts:

Added getPNGDimensions() helper to read PNG dimensions from buffer (header-only, no decode)
Updated screenshot provider to detect actual screenshot size and call setScreenshotSize()
Applied same logic to captureAndSendScreenshot()
Graceful fallback if dimension reading fails

test plan

Wrote test scripts locally and ensured that coordinate positioning is now correct

Summary by cubic

Fixes mis-scaled CUA clicks in advanced stealth by mapping model coordinates through the actual screenshot size to the viewport, and prevents corrupted images by encoding screenshots correctly.

Bug Fixes
- Track screenshot dimensions and add setScreenshotSize() in GoogleCUAClient, OpenAICUAClient, and MicrosoftCUAClient.
- Scale coordinates screenshot → viewport (Google: 0–1000 → screenshot → viewport; OpenAI/Microsoft: screenshot → viewport with separate X/Y).
- Read PNG width/height from the buffer header; set screenshot size when available with safe fallback, and pass base64 to captureScreenshot() to avoid corrupted images.
- Fix event name to agent_screenshot_taken_event and emit the screenshot buffer via the bus.

^{Written for commit cabead9. Summary will update automatically on new commits.}

changeset-bot · 2025-12-18T00:05:51Z

🦋 Changeset detected

Latest commit: cabead9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

greptile-apps · 2025-12-18T00:08:51Z

Greptile Summary

Fixes CUA coordinate mis-scaling in advanced stealth mode by tracking actual screenshot dimensions separately from viewport dimensions and implementing two-step coordinate transformation.

Key Changes:

Added actualScreenshotSize field to GoogleCUAClient, OpenAICUAClient, and MicrosoftCUAClient to track actual screenshot dimensions
Added setScreenshotSize() method to all three CUA clients
Implemented getPNGDimensions() helper in v3CuaAgentHandler.ts to read PNG dimensions from buffer headers
Updated coordinate normalization logic:
- Google: 0-1000 range → screenshot space → viewport space
- OpenAI/Microsoft: screenshot space → viewport space (with separate X/Y scaling)
Fixed typo in event name: agent_screensot_taken_event → agent_screenshot_taken_event
Removed unused imageResize() function from utils.ts

Potential Gap:

AnthropicCUAClient also handles coordinates through the computer tool but was not updated with the same scaling fix. If Anthropic CUA is used with advanced stealth mode, it may experience the same coordinate mis-scaling issue.

Confidence Score: 4/5

Safe to merge with minor style improvements possible
Core logic is sound and addresses the coordinate scaling issue correctly. PNG dimension reading is lightweight (header-only). Graceful fallback on dimension reading failure. One style suggestion for code simplification. No consideration given to AnthropicCUAClient which may need the same fix.
Check if AnthropicCUAClient needs the same coordinate scaling fix for consistency

Important Files Changed

Filename	Overview
packages/core/lib/v3/agent/GoogleCUAClient.ts	Added `actualScreenshotSize` tracking and two-step coordinate scaling (0-1000 → screenshot → viewport)
packages/core/lib/v3/agent/OpenAICUAClient.ts	Added `actualScreenshotSize` tracking and coordinate scaling in `convertComputerCallToAction()` from screenshot to viewport space
packages/core/lib/v3/agent/MicrosoftCUAClient.ts	Added `actualScreenshotSize` tracking and coordinate scaling in `convertFunctionCallToAction()` from screenshot to viewport space
packages/core/lib/v3/handlers/v3CuaAgentHandler.ts	Added `getPNGDimensions()` helper and calls to `setScreenshotSize()` in screenshot provider and `captureAndSendScreenshot()`; fixed typo in event name

Sequence Diagram

sequenceDiagram
    participant Handler as v3CuaAgentHandler
    participant Page as Browser Page
    participant Client as CUA Client<br/>(Google/OpenAI/Microsoft)
    participant Model as AI Model
    
    Note over Handler,Model: Screenshot Capture & Size Detection
    Handler->>Page: page.screenshot()
    Page-->>Handler: PNG Buffer (e.g., 1288x711)
    Handler->>Handler: getPNGDimensions(buffer)
    Note right of Handler: Reads PNG header<br/>bytes 16-23 for dimensions
    Handler->>Client: setScreenshotSize(1288, 711)
    Note right of Client: Stores actualScreenshotSize<br/>separate from viewport
    Handler->>Client: setViewport(2560, 1305)
    Note right of Client: Stores currentViewport<br/>(browser reports larger size)
    Handler->>Model: Send base64 screenshot
    
    Note over Handler,Model: Model Returns Coordinates
    Model-->>Client: Coordinates in model space<br/>(Google: 0-1000, Others: screenshot space)
    
    Note over Handler,Model: Two-Step Coordinate Scaling
    Client->>Client: normalizeCoordinates()
    Note right of Client: Step 1: Model → Screenshot<br/>Google: (x/1000) * 1288<br/>Others: x (already in screenshot space)
    Note right of Client: Step 2: Screenshot → Viewport<br/>(screenshotX * 2560/1288)<br/>(screenshotY * 1305/711)
    Client->>Handler: Scaled coordinates (viewport space)
    Handler->>Page: click(x, y)
    Note right of Page: Click at correct position<br/>in 2560x1305 viewport

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts

cubic-dev-ai

1 issue found across 2 files

Prompt for AI agents (all 1 issues)


Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts">

<violation number="1" location="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts:607">
P2: Breaking change: Event name was changed from `agent_screensot_taken_event` to `agent_screenshot_taken_event`, but existing listeners in `onlineMind2Web.ts` still subscribe to the old misspelled name. This will cause those listeners to miss events from this handler.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts

miguelg719 · 2025-12-18T00:16:45Z

is this issue isolated on google cua or does it extend to other cua providers? @shrey150

shrey150 · 2025-12-18T01:11:06Z

is this issue isolated on google cua or does it extend to other cua providers? @shrey150

@miguelg719 this doesn't affect Microsoft CUA, but I'll investigate with both Anthropic and OpenAI CUA

shrey150 · 2025-12-18T01:16:52Z

@greptileai

greptile-apps

_{3 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts

shrey150 · 2025-12-19T00:13:22Z

@greptileai

greptile-apps

_{7 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts

Convert screenshotBuffer to base64 string before passing to captureScreenshot(). Previously, Buffer was passed directly which caused UTF-8 encoding instead of base64, resulting in corrupted screenshot data sent to LLM APIs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

shrey150 · 2025-12-19T00:33:27Z

@greptileai

greptile-apps

_{7 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2025-12-19T00:37:06Z

packages/core/lib/v3/agent/GoogleCUAClient.ts

+    const screenshotX = (x / 1000) * this.actualScreenshotSize.width;
+    const screenshotY = (y / 1000) * this.actualScreenshotSize.height;
+    const scaleX = this.currentViewport.width / this.actualScreenshotSize.width;
+    const scaleY =
+      this.currentViewport.height / this.actualScreenshotSize.height;


style: redundant intermediate variables - can simplify to direct calculation

Suggested change

const screenshotX = (x / 1000) * this.actualScreenshotSize.width;

const screenshotY = (y / 1000) * this.actualScreenshotSize.height;

const scaleX = this.currentViewport.width / this.actualScreenshotSize.width;

const scaleY =

this.currentViewport.height / this.actualScreenshotSize.height;

const scaleX = this.currentViewport.width / this.actualScreenshotSize.width;

const scaleY =

this.currentViewport.height / this.actualScreenshotSize.height;

return {

x: Math.floor((x / 1000) * this.actualScreenshotSize.width * scaleX),

y: Math.floor((y / 1000) * this.actualScreenshotSize.height * scaleY),

};

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/core/lib/v3/agent/GoogleCUAClient.ts Line: 917:921 Comment: **style:** redundant intermediate variables - can simplify to direct calculation ```suggestion const scaleX = this.currentViewport.width / this.actualScreenshotSize.width; const scaleY = this.currentViewport.height / this.actualScreenshotSize.height; return { x: Math.floor((x / 1000) * this.actualScreenshotSize.width * scaleX), y: Math.floor((y / 1000) * this.actualScreenshotSize.height * scaleY), }; ``` <sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub> How can I resolve this? If you propose a fix, please make it concise.

pirate

nice typo catch too

@tkattkat

Bastaff · 2026-01-12T12:42:31Z

We're still experiencing issues with this while using a responsive live view container.
Agent clicks at the wrong coordinates.

Package versions:
"@browserbasehq/sdk": "^2.6.0",
"@browserbasehq/stagehand": "^3.0.7",

Agent config:
modelName: "google/gemini-2.5-computer-use-preview-10-2025",

Stagehand config:
modelName: "google/gemini-2.5-flash",

miguelg719 · 2026-01-12T15:23:41Z

hey @Bastaff can you share your stagehand config, including the browserbaseSessionCreateParams? I'll take a look

Bastaff · 2026-01-12T15:37:09Z

Here you go @miguelg719
Changing the resolution in this code snippet does not change the results.
Changing the resolution of the iframe that shows the live view also doesn't change the results.

async function initStageHand(workflow, config) {
    const logger = createLogger(workflow);

    const stagehandConfig = {
        env: "BROWSERBASE",
        verbose: process.env.NODE_ENV === "development" ? 2 : 1,
        logger: (logLine) => logger.log(logLine),
        model: {
            modelName: "google/gemini-2.5-flash",
            apiKey: process.env.GOOGLE_API_KEY
        },
        domSettleTimeout: 30_000,
        experimental: true,
        useApi: false,
    };
    
    if (config.provider === "BROWSERBASE") {
        stagehandConfig.env = "BROWSERBASE";
        stagehandConfig.apiKey = process.env.BROWSERBASE_API_KEY;
        stagehandConfig.projectId = process.env.BROWSERBASE_PROJECT_ID;
        stagehandConfig.browserbaseSessionID = config.sessionId;
    } 
    
    const stagehand = new Stagehand(stagehandConfig);
    await stagehand.init();
    return stagehand;
}


async function createContext(userId = "anon") {
    try {
        const context = await bb.contexts.create({
            projectId: process.env.BROWSERBASE_PROJECT_ID,
        });
        return context;
    } catch (e) {
        console.error("Error creating Browserbase context:", e);
        throw e;
    }
}

async function createSession(userId, contextId) {
    if (!contextId) {
        const ctx = await createContext(userId);
        contextId = ctx.id;
    }

    const session = await bb.sessions.create({
        projectId: process.env.BROWSERBASE_PROJECT_ID,
        userMetadata: { userId },
        browserSettings: {
            solveCaptchas: true,
            viewport: { width: 1288, height: 711 },
            context: {
                id: contextId,
                persist: true
            },
            advancedStealth: true,
        },
        timeout: 10 * 60,
        keepAlive: true,
        proxies: true,
    });

    await sleep(3000);
    return session;
}

Fix viewport for Gemini CUA

203d308

shrey150 requested a review from miguelg719 December 18, 2025 00:05

greptile-apps bot reviewed Dec 18, 2025

View reviewed changes

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts Show resolved Hide resolved

cubic-dev-ai bot reviewed Dec 18, 2025

View reviewed changes

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts Show resolved Hide resolved

Fix event bus typo

4428128

greptile-apps bot reviewed Dec 18, 2025

View reviewed changes

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts Show resolved Hide resolved

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts Outdated Show resolved Hide resolved

Add fixes for Microsoft and OpenAI

b74a68d

shrey150 changed the title ~~fix: Google CUA coordinating scaling issue~~ fix: CUA coordinating scaling issue Dec 18, 2025

shrey150 requested a review from tkattkat December 18, 2025 19:37

shrey150 added 2 commits December 18, 2025 11:40

fix lint

131c44a

use const

7635271

greptile-apps bot reviewed Dec 19, 2025

View reviewed changes

packages/core/lib/v3/handlers/v3CuaAgentHandler.ts Outdated Show resolved Hide resolved

greptile-apps bot reviewed Dec 19, 2025

View reviewed changes

pirate approved these changes Dec 19, 2025

View reviewed changes

Copilot AI mentioned this pull request Dec 19, 2025

Verify PR #1430: Fix CUA coordinate scaling with screenshot-viewport mismatch #1442

Closed

Add changeset + remove sharp dependency

cabead9

shrey150 merged commit bd2db92 into main Dec 19, 2025
16 checks passed

This was referenced Dec 19, 2025

Version Packages #1414

Merged

Version Packages kenchikuliu/stagehand#1

Open

Version Packages CloudEngineHub/stagehand#1

Open

Version Packages shaneholloman/stagehand#1

Open

fix: CUA coordinating scaling issue #1430

fix: CUA coordinating scaling issue #1430

Uh oh!

Conversation

shrey150 commented Dec 18, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

why

what changed

test plan

Summary by cubic

Uh oh!

changeset-bot bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

greptile-apps bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miguelg719 commented Dec 18, 2025

Uh oh!

shrey150 commented Dec 18, 2025

Uh oh!

shrey150 commented Dec 18, 2025

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shrey150 commented Dec 19, 2025

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shrey150 commented Dec 19, 2025

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

pirate left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Bastaff commented Jan 12, 2026

Uh oh!

miguelg719 commented Jan 12, 2026

Uh oh!

Bastaff commented Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shrey150 commented Dec 18, 2025 •

edited by cubic-dev-ai bot

Loading

changeset-bot bot commented Dec 18, 2025 •

edited

Loading

greptile-apps bot commented Dec 18, 2025 •

edited

Loading