Skip to content

feat: add snapshot.boxes config to include bounding boxes in auto-snapshots #1695

Closed
microsoft/playwright
#42102

Description

The browser_snapshot tool already supports a boxes parameter that includes element bounding boxes as [box=x,y,width,height] in the accessibility snapshot. However, auto-snapshots taken after action tools (like browser_click, browser_type) don't include bounding boxes.

Use case

We're building a tool (workflow-validation-demo-publisher) that creates interactive product demos from automated browser sessions on Red Hat OpenShift AI. After each click action, we need the bounding box of the clicked element to place hotspots accurately in the demo.

Currently, auto-snapshots only contain the accessibility tree without position data, so we have to use a vision model to estimate coordinates -- which is slow (~5s per frame), expensive, and imprecise (~70-80% accuracy vs pixel-perfect).

Proposed change

Add a snapshot.boxes config option (and a corresponding CLI flag) that, when enabled, includes bounding boxes in all auto-snapshots -- not just explicit browser_snapshot calls.

The change is small since the bounding box infrastructure already exists:

  • setIncludeFullSnapshot() (used by browser_snapshot) already handles boxes
  • setIncludeSnapshot() (used by action tools) only sets mode, not boxes
  • The fix is to thread the boxes config through setIncludeSnapshot() and into the auto-snapshot capture path

Config:

{
  "snapshot": {
    "mode": "full",
    "boxes": true
  }
}

CLI: npx @playwright/mcp --snapshot-boxes (or similar, happy to follow existing naming conventions)

Estimated scope

~15-20 lines of code + 1 test. I'm happy to submit the PR if assigned.

Context

This would benefit any downstream consumer that needs element coordinates from MCP tool responses -- not just our demo publisher use case. For example, test automation tools that need to verify element positions, or accessibility auditing tools that need layout information alongside the accessibility tree.

Activity

  1. dcohnlif commented on Aug 3, 2026

    @dcohnlif
    Author

    For reference, the hwatu browser automation tool picked up this signal and implemented the same feature for their own snapshot output: hongnoul/hwatu#18 → hongnoul/hwatu#21 (merged).

    Their implementation adds an opt-in rect: [x, y, w, h] (viewport CSS px, from getBoundingClientRect) per interactable element, with a flag to keep default token cost unchanged. Took less than a day.

    This confirms the demand is real and the scope is small.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions