Browser Tool Improvements — Field Report

Based on extensive real-world browser automation (email provider signups across 9+ providers) on 2026-03-12.

Issues Encountered & Proposed Fixes

1. 🔴 No CSS Selector Support — Forces Verbose Snapshot→Ref Workflow

Problem: Every single interaction requires a snapshot first to get refs, then using those refs. When you already know the DOM structure (e.g. button.btn-full, input[name=email]), you can't use CSS selectors directly — error: 'selector' is not supported. Use 'ref' from snapshot instead.

Impact: Doubles the number of API calls for every interaction. A simple "fill form and click submit" that should be 3 calls becomes 6+ (snapshot, fill, snapshot again if refs stale, click, etc.)

Proposed Fix: Add optional selector parameter to act actions (click, type, fill, hover) as a fallback when ref isn't provided. Internally, use Playwright's native page.locator(selector) — it already supports CSS, text, role, and combined selectors. Keep ref as the recommended/primary path but allow selector for power users.

File: src/browser/pw-tools-core.interactions.ts — each function (clickViaPlaywright, typeViaPlaywright, etc.) would accept an optional selector and resolve the locator via page.locator(selector) instead of refLocator(page, ref) when selector is provided.

2. 🔴 Refs Go Stale After Any Page Mutation — No Auto-Refresh

Problem: After filling a form field or clicking a button (especially on SPAs), the refs from the previous snapshot become stale. The page DOM has changed, so the old e13/e17/etc. refs may point to nothing or the wrong element. This requires taking a fresh snapshot after nearly every interaction.

Impact: Massive token waste on repeated snapshots. In a 5-field form, you might need 3-4 snapshots.

Proposed Fix:

• Option A: Auto-snapshot after mutations — after each act call, if the page DOM changed significantly, return updated refs in the response.
• Option B: Stable refs — use aria-ref IDs (refs="aria") which survive minor DOM mutations since they're based on accessibility tree identity rather than DOM position.
• Option C: Return a refsStale: true hint in the act response when the page has navigated or mutated significantly, so the agent knows to re-snapshot.

File: src/browser/pw-tools-core.interactions.ts — post-interaction hook, src/browser/pw-session.ts — ref storage/staleness tracking.

3. 🟡 evaluate() Can't Access DevTools APIs (getEventListeners, etc.)

Problem: When trying to debug JS-framework buttons (no href, no onclick), evaluate() with getEventListeners() fails because it's a DevTools Console API, not available in page context.

Impact: Can't programmatically discover how framework-managed elements are wired up. Had to make 4+ sequential evaluate calls to incrementally inspect an element.

Proposed Fix: Add an inspectElement action (or enhance evaluate with a devtools: true option) that uses CDP's DOMDebugger.getEventListeners directly instead of page.evaluate. This would return the element's event listeners, computed styles, and DOM properties in one call.

File: New helper in src/browser/pw-tools-core.interactions.ts or src/browser/cdp.helpers.ts using the existing withPageScopedCdpClient pattern from pw-tools-core.snapshot.ts.

4. 🟡 No Batch/Compound Actions — Every Step Is a Separate Round-Trip

Problem: Filling a 5-field form and clicking submit requires 6+ separate tool calls (type×5 + click). Each is a network round-trip with snapshot overhead.

Proposed Fix: The fill action with fields array already exists but isn't well-documented. More importantly, add a batch or sequence action kind that accepts an array of actions to execute in order:

{
  "kind": "sequence",
  "steps": [
    {"kind": "fill", "ref": "e13", "text": "[email protected]"},
    {"kind": "fill", "ref": "e17", "text": "password123"},
    {"kind": "click", "ref": "e25"}
  ]
}

File: src/browser/pw-tools-core.interactions.ts — new sequenceViaPlaywright function; src/agents/tools/browser-tool.actions.ts — action routing.

5. 🟡 Form Values Clearing on Certain Interactions

Problem: On Mailfence, after filling multiple form fields and clicking a JS-framework button, the form reset and cleared all values. This required re-filling everything.

Root Cause Hypothesis: The fill() Playwright method dispatches input/change events. Some frameworks (React, Vue) only update internal state on specific event sequences. If the framework doesn't recognize the synthetic fill, it may reset on next render.

Proposed Fix:

• Add a slowly: true option (already exists!) that uses click() + type() with delay instead of fill(). This simulates real keystrokes.
• Better: detect when fill didn't stick by re-reading the field value after fill and warn if it's empty.
• Document this as a known pattern: "For React/Vue forms, use slowly: true"

File: src/browser/pw-tools-core.interactions.ts — typeViaPlaywright already handles slowly, but the post-fill verification is new.

6. 🟢 No CAPTCHA Detection/Signaling

Problem: Multiple signup attempts failed silently at CAPTCHAs. The agent kept trying to interact with the page not realizing a CAPTCHA was blocking.

Proposed Fix: During snapshot, detect common CAPTCHA iframes (hCaptcha, reCAPTCHA, Cloudflare Turnstile) and include a captchaDetected: true flag + type in the snapshot response. This lets the agent immediately bail or request human intervention instead of wasting calls.

File: src/browser/pw-tools-core.snapshot.ts — post-snapshot analysis for known CAPTCHA selectors/iframes.

7. 🟢 No Built-in "Wait for Element" with Ref

Problem: After clicking "Submit", need to wait for a success message or next page. Currently using kind: "wait" with timeMs (blind wait) or textGone/text (text-based). No way to wait for a specific element ref to appear.

Proposed Fix: Add waitForRef option to the wait action — waits until a specific accessibility role/name pattern appears in the page.

Priority Order for Implementation

1. CSS selector fallback (🔴 highest impact — halves API calls)
2. Ref staleness detection (🔴 prevents silent failures)
3. Batch/sequence actions (🟡 major efficiency gain)
4. Element inspector via CDP (🟡 debugging aid)
5. Post-fill verification (🟡 prevents form reset surprises)
6. CAPTCHA detection (🟢 nice-to-have)
7. Wait for element (🟢 nice-to-have)