You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: add snapshot.boxes config to include bounding boxes in auto-snapshots #1695
The browser_snapshot tool already supports a boxes parameter that includes element bounding boxes as [box=x,y,width,height] in the accessibility snapshot. However, auto-snapshots taken after action tools (like browser_click, browser_type) don't include bounding boxes.
Use case
We're building a tool (workflow-validation-demo-publisher) that creates interactive product demos from automated browser sessions on Red Hat OpenShift AI. After each click action, we need the bounding box of the clicked element to place hotspots accurately in the demo.
Currently, auto-snapshots only contain the accessibility tree without position data, so we have to use a vision model to estimate coordinates -- which is slow (~5s per frame), expensive, and imprecise (~70-80% accuracy vs pixel-perfect).
Proposed change
Add a snapshot.boxes config option (and a corresponding CLI flag) that, when enabled, includes bounding boxes in all auto-snapshots -- not just explicit browser_snapshot calls.
The change is small since the bounding box infrastructure already exists:
setIncludeFullSnapshot() (used by browser_snapshot) already handles boxes
setIncludeSnapshot() (used by action tools) only sets mode, not boxes
The fix is to thread the boxes config through setIncludeSnapshot() and into the auto-snapshot capture path
~15-20 lines of code + 1 test. I'm happy to submit the PR if assigned.
Context
This would benefit any downstream consumer that needs element coordinates from MCP tool responses -- not just our demo publisher use case. For example, test automation tools that need to verify element positions, or accessibility auditing tools that need layout information alongside the accessibility tree.
For reference, the hwatu browser automation tool picked up this signal and implemented the same feature for their own snapshot output: hongnoul/hwatu#18 → hongnoul/hwatu#21 (merged).
Their implementation adds an opt-in rect: [x, y, w, h] (viewport CSS px, from getBoundingClientRect) per interactable element, with a flag to keep default token cost unchanged. Took less than a day.
This confirms the demand is real and the scope is small.
Activity
dcohnlif commentedon Aug 3, 2026
For reference, the hwatu browser automation tool picked up this signal and implemented the same feature for their own snapshot output: hongnoul/hwatu#18 → hongnoul/hwatu#21 (merged).
Their implementation adds an opt-in
rect: [x, y, w, h](viewport CSS px, fromgetBoundingClientRect) per interactable element, with a flag to keep default token cost unchanged. Took less than a day.This confirms the demand is real and the scope is small.