Skip to content

feat(browser): batch actions, CSS selector support, and click delayMs#44934

Closed
diwakarrankawat wants to merge 3 commits into
openclaw:mainfrom
diwakarrankawat:feat/browser-batch-actions-and-selectors
Closed

feat(browser): batch actions, CSS selector support, and click delayMs#44934
diwakarrankawat wants to merge 3 commits into
openclaw:mainfrom
diwakarrankawat:feat/browser-batch-actions-and-selectors

Conversation

@diwakarrankawat

Copy link
Copy Markdown
Contributor

Summary

Three additive improvements to the browser act tool that reduce token cost and latency for browser automation:

1. 🎯 CSS Selector Support (selector parameter)

All element-targeting actions (click, type, hover, drag, scrollIntoView, select) now accept an optional selector parameter alongside ref.

// Before: always need a snapshot first to get refs
{"kind": "click", "ref": "e23"}

// After: can use CSS selectors directly when you know the DOM
{"kind": "click", "selector": "button.submit"}
{"kind": "type", "selector": "input[name=email]", "text": "[email protected]"}

When selector is provided, Playwright's page.locator() is used directly — no snapshot needed. ref still works as before (fully backward compatible).

Impact: Eliminates snapshot→ref roundtrips for known DOM structures. A 5-field form that needed 3-4 snapshots now needs zero.

2. ⏱️ Click Delay (delayMs on click)

The click action now accepts delayMs. When set, the element is hovered first, waits the specified duration, then clicks.

// Before: 3 separate tool calls
{"kind": "hover", "ref": "e23"}
{"kind": "wait", "timeMs": 500}
{"kind": "click", "ref": "e23"}

// After: 1 tool call
{"kind": "click", "selector": "button.submit", "delayMs": 500}

Impact: 3 roundtrips → 1. Saves ~4-6 seconds and tokens per interaction.

3. 📦 Batch Actions (kind: "batch")

New action kind that executes multiple actions sequentially in a single tool call:

{
  "kind": "batch",
  "actions": [
    {"kind": "click", "selector": "input[name=email]"},
    {"kind": "type", "selector": "input[name=email]", "text": "[email protected]"},
    {"kind": "click", "selector": "input[name=password]"},
    {"kind": "type", "selector": "input[name=password]", "text": "secret"},
    {"kind": "click", "selector": "button[type=submit]", "delayMs": 300}
  ],
  "stopOnError": true
}

Returns results array: { results: [{ ok: true }, { ok: true }, ...] }

Impact: A 5-step form fill goes from 5 tool calls (~10-15 seconds + tokens each) to 1 call (~2 seconds total). Estimated 80-90% reduction in token cost for multi-step browser automation.

Motivation

From real-world automation field testing (see #44431) and file upload flakiness reports (#38844), three patterns emerged:

  1. Agents waste tokens taking snapshots just to get refs for elements they already know
  2. Human-like hover→click requires 3 separate tool calls
  3. Multi-step flows burn tokens on AI inference between each step when no decision-making is needed

These changes are purely additive — no existing behavior is modified. All existing ref-based workflows continue working identically.

Files Changed

  • src/browser/client-actions-core.ts — Action type definitions
  • src/browser/pw-tools-core.interactions.ts — Execution functions
  • src/browser/pw-tools-core.shared.ts — Helper: requireRefOrSelector()
  • src/browser/pw-ai.ts — Export new batchViaPlaywright
  • src/browser/routes/agent.act.ts — Route handler for batch + selector
  • src/browser/routes/agent.act.shared.ts — Shared action dispatch

Testing

  • Build passes: pnpm build
  • Backward compatible: all existing ref-based actions unchanged
  • Manual testing with real browser automation on production sites

🤖 AI-Assisted

This PR was developed with AI assistance (Claude Code + OpenClaw). The changes were motivated by hands-on browser automation pain points encountered while building a CDP mouse simulator tool for human-like browser interaction.

Addresses #44431, #38844

@greptile-apps

greptile-apps Bot commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds three additive improvements to the browser act tool: CSS selector support for element targeting, a delayMs option for click (hover-then-click), and a new batch action kind for executing multiple actions in a single tool call. The selector/ref refactoring across clickViaPlaywright, hoverViaPlaywright, dragViaPlaywright, typeViaPlaywright, selectOptionViaPlaywright, and scrollIntoViewViaPlaywright is clean and backward-compatible. However, the batch implementation has two significant security issues:

  • evaluateEnabled bypass: executeSingleAction invokes evaluateViaPlaywright and waitForViaPlaywright (with fn) without checking the evaluateEnabled configuration flag. Direct calls to evaluate and wait --fn are guarded by a 403 response in the route handler, but that guard is entirely skipped when those actions appear inside a batch payload. An operator who has disabled JS evaluation via evaluateEnabled=false can have that restriction circumvented.
  • Unvalidated raw input in batch: body.actions is forwarded as a raw JSON array with no field-level sanitization, bypassing the toStringOrEmpty / toBoolean / toNumber normalization that every direct action handler applies.

Additional notes:

  • SELECTOR_ALLOWED_KINDS is a Set created on every HTTP request; it should be a module-level constant.
  • There is no recursion depth limit for nested batch actions, which could exhaust the call stack with a crafted payload.

Confidence Score: 2/5

  • Not safe to merge as-is — the batch handler bypasses the evaluateEnabled security gate, allowing arbitrary JS execution in the browser even when the operator has explicitly disabled it.
  • The selector/ref refactoring and delayMs additions are low-risk and well-implemented. However, the batch feature introduces a concrete privilege-escalation path: any client can embed an evaluate or wait --fn action inside a batch to execute arbitrary JavaScript in the controlled browser, regardless of the evaluateEnabled config setting. This is a meaningful security regression for deployments that rely on that flag. The unvalidated raw-input forwarding compounds the risk. These issues need to be fixed before merging.
  • src/browser/pw-tools-core.interactions.ts (executeSingleAction) and src/browser/routes/agent.act.ts (batch case) require the most attention.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/browser/pw-tools-core.interactions.ts
Line: 690-706

Comment:
**`evaluate` and `wait --fn` bypass `evaluateEnabled` security gate**

`executeSingleAction` handles both `evaluate` and `wait` (with `fn`) actions directly without checking the `evaluateEnabled` config flag. In the HTTP route handler (`agent.act.ts`), there are explicit 403 guards:

```typescript
// agent.act.ts – direct evaluate route
if (!evaluateEnabled) {
  return jsonError(res, 403, "act:evaluate is disabled by config...");
}

// agent.act.ts – direct wait route
if (fn && !evaluateEnabled) {
  return jsonError(res, 403, "wait --fn is disabled by config...");
}
```

These guards are entirely skipped when either action kind appears inside a `batch` payload, because `executeSingleAction` calls `evaluateViaPlaywright` and `waitForViaPlaywright` unconditionally. An operator who has set `evaluateEnabled=false` to restrict arbitrary JS execution in the browser will have that restriction bypassed by wrapping the call in a batch.

The fix is to pass `evaluateEnabled` into `executeSingleAction` (or `batchViaPlaywright`) and replicate the same guards:

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/browser/routes/agent.act.ts
Line: 369-382

Comment:
**Raw, unvalidated user input passed directly into batch executor**

`body.actions` is taken as a raw JavaScript array without any field-level sanitization and forwarded straight to `batchViaPlaywright`. Every other action kind in this route handler runs its inputs through helpers like `toStringOrEmpty`, `toBoolean`, `toNumber`, `parseClickButton`, etc. before dispatching.

Because `BrowserActRequest` is a TypeScript type (erased at runtime), the cast inside `executeSingleAction` provides no runtime safety. A few concrete consequences:

1. A `{ kind: "evaluate", fn: "…" }` object bypasses the `evaluateEnabled` check (see the other comment), but also passes `fn` to `evaluateViaPlaywright` without the `toStringOrEmpty` normalisation that the direct route applies.
2. A `{ kind: "click", ref: {}, selector: null }` object arrives in `requireRefOrSelector` as non-string values; the current guard (`typeof ref === "string" ? … : ""`) handles this safely, but future action kinds may not.
3. Action-specific defaults (e.g., `doubleClick` coerced via `toBoolean`) are never applied, so the typed functions receive raw JSON values.

The batch handler should at minimum validate that each element is a plain object with a known `kind` string before passing it to `executeSingleAction`.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/browser/routes/agent.act.ts
Line: 35-46

Comment:
**`SELECTOR_ALLOWED_KINDS` set re-created on every request**

A new `Set` object is allocated for every incoming HTTP request. Since the set's contents are static, this should be a module-level constant:

```suggestion
    const SELECTOR_ALLOWED_KINDS = new Set([
      "batch",
      "click",
      "drag",
      "hover",
      "scrollIntoView",
      "select",
      "type",
      "wait",
    ]);
    if (Object.hasOwn(body, "selector") && !SELECTOR_ALLOWED_KINDS.has(kind)) {
```

Move the `const SELECTOR_ALLOWED_KINDS = new Set([...])` declaration to module scope (outside `registerBrowserAgentActRoutes`) so it is created once and shared across all requests.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/browser/pw-tools-core.interactions.ts
Line: 698-706

Comment:
**No recursion depth limit for nested batches**

`executeSingleAction` delegates nested `batch` actions back to `batchViaPlaywright`, which calls `executeSingleAction` again. There is no depth limit, so a malicious or buggy payload with deeply nested batches will exhaust the JS call stack (V8 typically limits at ~10 000 frames). Consider tracking and enforcing a maximum nesting depth (e.g., 5) and returning an error result rather than crashing:

```typescript
async function executeSingleAction(
  action: BrowserActRequest,
  cdpUrl: string,
  targetId?: string,
  depth = 0,
): Promise<void> {
  // ...
  case "batch":
    if (depth >= MAX_BATCH_DEPTH) {
      throw new Error("Maximum batch nesting depth exceeded");
    }
    await batchViaPlaywright({ ..., depth: depth + 1 });
    break;
```

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: 1a413a7

Comment thread src/browser/pw-tools-core.interactions.ts
Comment thread src/browser/routes/agent.act.ts Outdated
Comment thread src/browser/routes/agent.act.ts Outdated
Comment thread src/browser/pw-tools-core.interactions.ts
…ayMs

Adds three improvements to the browser act tool:

1. CSS selector support: All element-targeting actions (click, type,
   hover, drag, scrollIntoView, select) now accept an optional
   'selector' parameter alongside 'ref'. When selector is provided,
   Playwright's page.locator() is used directly, skipping the need
   for a snapshot to obtain refs. This reduces roundtrips for agents
   that already know the DOM structure.

2. Click delay (delayMs): The click action now accepts an optional
   'delayMs' parameter. When set, the element is hovered first, then
   after the specified delay, clicked. This enables human-like
   hover-before-click in a single tool call instead of three
   (hover + wait + click).

3. Batch actions: New 'batch' action kind that accepts an array of
   actions to execute sequentially in a single tool call. Supports
   'stopOnError' (default true) to control whether execution halts
   on first failure. Results are returned as an array. This eliminates
   the AI inference roundtrip between each action, dramatically
   reducing latency and token cost for multi-step flows.

Addresses: openclaw#44431, openclaw#38844
…input validation, recursion limit

Fixes all 4 issues raised by Greptile review:

1. Security: batch actions now respect evaluateEnabled flag.
   executeSingleAction and batchViaPlaywright accept evaluateEnabled
   param. evaluate and wait-with-fn inside batches are rejected
   when evaluateEnabled=false, matching the direct route guards.

2. Security: batch input validation. Each action in body.actions
   is validated as a plain object with a known kind string before
   dispatch. Applies same normalization as direct action handlers.

3. Perf: SELECTOR_ALLOWED_KINDS moved to module scope as a
   ReadonlySet<string> constant (was re-created on every request).

4. Security: max batch nesting depth of 5. Nested batch actions
   track depth and throw if MAX_BATCH_DEPTH exceeded, preventing
   call stack exhaustion from crafted payloads.
@diwakarrankawat
diwakarrankawat force-pushed the feat/browser-batch-actions-and-selectors branch from 1f27edb to f352d57 Compare March 13, 2026 10:11
@obviyus

obviyus commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2c6a15f8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +370 to +374
const result = await pw.batchViaPlaywright({
cdpUrl,
targetId: tab.targetId,
actions,
stopOnError,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Forward evaluateEnabled when running batch actions

registerBrowserAgentActRoutes reads evaluateEnabled, but the batchViaPlaywright call never passes it through, so executeSingleAction sees evaluateEnabled as falsy for every batch. In practice this makes {"kind":"evaluate"} and {"kind":"wait","fn":...} fail inside batches even when browser.evaluateEnabled=true, which breaks the advertised batch behavior for evaluation-based flows.

Useful? React with 👍 / 👎.

Comment on lines +826 to +827
default:
throw new Error(`Unsupported batch action kind: ${(action as { kind: string }).kind}`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle all declared action kinds in batch dispatcher

executeSingleAction does not implement resize or close, even though these are valid BrowserActRequest kinds handled by the normal /act path. Any batch containing either action will always hit the default branch and return Unsupported batch action kind, so batch execution is inconsistent with the request schema and single-action behavior.

Useful? React with 👍 / 👎.

@diwakarrankawat

diwakarrankawat commented Mar 14, 2026

Copy link
Copy Markdown
Contributor Author

Closing this as @vincentkoc picked it up in #45457 with additional tests and hardening. Glad to see the features ship! Will contribute more focused PRs going forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants