Skip to content

Commit 4cdd9b8

Browse files
authored
feat(browser): extract action — answer page questions via a bounded sub-model call (#113861)
* feat(browser): add page extract action * docs(browser): document extract action * improve(browser): make extract discoverable in tool description * fix(browser): keep extract dep types module-local
1 parent a9f1ed8 commit 4cdd9b8

32 files changed

Lines changed: 1030 additions & 11 deletions

docs/cli/browser.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,19 @@ openclaw browser close t1
135135
136136
Raw target ids are volatile diagnostic handles, not durable agent memory: when Chromium replaces the underlying raw target during a navigation or form submit, OpenClaw keeps the stable `tabId`/label attached to the replacement tab when it can prove the match. Prefer `suggestedTargetId`.
137137
138-
## Snapshot / screenshot / actions
138+
## Extract / snapshot / screenshot / actions
139+
140+
Answer a question from the current page without printing the page content:
141+
142+
```bash
143+
openclaw browser extract "What is the main conclusion?"
144+
openclaw browser extract "Which deadline is listed?" --target-id docs --timeout-ms 90000
145+
```
146+
147+
`extract` uses the selected agent model, returns only the wrapped answer, and
148+
reports `NOT_FOUND` when the answer is absent. Its overall timeout defaults to
149+
60 seconds and is clamped to 5–120 seconds. It requires a Playwright-backed
150+
profile; use `snapshot` when you need refs or when extraction is unavailable.
139151
140152
Snapshot:
141153
@@ -275,7 +287,7 @@ Current existing-session limits:
275287
- File uploads require `--ref` / `--input-ref`, do not support CSS `--element`, and support one file at a time.
276288
- Dialog hooks do not support `--timeout`.
277289
- Screenshots support page captures and `--ref`, but not CSS `--element`.
278-
- `responsebody`, download interception, PDF export, and batch actions still require a managed browser or raw CDP profile.
290+
- `extract`, `responsebody`, download interception, PDF export, and batch actions still require a managed browser or raw CDP profile.
279291
280292
## Remote browser control (node host proxy)
281293

docs/docs_map.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1340,7 +1340,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
13401340
- H2: If the command is missing
13411341
- H2: Profiles
13421342
- H2: Tabs
1343-
- H2: Snapshot / screenshot / actions
1343+
- H2: Extract / snapshot / screenshot / actions
13441344
- H2: State and storage
13451345
- H2: Debugging
13461346
- H2: Existing Chrome via MCP

docs/tools/browser-control.md

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ agent tools, but nothing listens on the loopback control port.
2323
- Status/start/stop: `GET /`, `GET /doctor`, `POST /start`, `POST /stop`, `POST /reset-profile`
2424
- Profiles: `GET /profiles`, `POST /profiles/create`, `DELETE /profiles/:name`
2525
- Tabs: `GET /tabs`, `POST /tabs/open`, `POST /tabs/focus`, `DELETE /tabs/:targetId`, `POST /tabs/action`
26-
- Snapshot/screenshot: `GET /snapshot`, `POST /screenshot`
26+
- Snapshot/screenshot/extract: `GET /snapshot`, `POST /screenshot`, `POST /extract`
2727
- Actions: `POST /navigate`, `POST /act`
2828
- Hooks: `POST /hooks/file-chooser`, `POST /hooks/dialog`
2929
- Downloads: `POST /download`, `POST /wait/download`
@@ -84,8 +84,8 @@ Other runtime failures may still return `{ "error": "<message>" }` without a
8484

8585
### Playwright requirement
8686

87-
Some features (navigate/act/AI snapshot/role snapshot, element screenshots,
88-
PDF) require Playwright. If Playwright isn't installed, those endpoints return
87+
Some features (navigate/act/AI snapshot/role snapshot, extract, element
88+
screenshots, PDF) require Playwright. If Playwright isn't installed, those endpoints return
8989
a clear 501 error.
9090

9191
What still works without Playwright:
@@ -107,6 +107,7 @@ What still needs Playwright:
107107
- AI snapshots that depend on Playwright's native AI snapshot format
108108
- CSS-selector element screenshots (`--element`)
109109
- full browser PDF export
110+
- page-question extraction
110111

111112
Element screenshots also reject `--full-page`; the route returns `fullPage is
112113
not supported for element screenshots`.
@@ -198,6 +199,7 @@ openclaw browser snapshot --urls
198199
openclaw browser snapshot --selector "#main" --interactive
199200
openclaw browser snapshot --frame "iframe#main" --interactive
200201
openclaw browser snapshot --out snapshot.txt
202+
openclaw browser extract "What is the page's main conclusion?"
201203
openclaw browser console --level error
202204
openclaw browser errors --clear
203205
openclaw browser requests --filter api --clear
@@ -265,6 +267,12 @@ openclaw browser set device "iPhone 14"
265267
266268
Notes:
267269
270+
- Use `browser extract "<question>"` or agent-tool `action="extract"` when you
271+
need an answer from the current page but do not need interaction refs. It
272+
sanitizes readable page content, caps it at 80,000 characters, runs one
273+
model call, and returns only the wrapped answer. The overall timeout defaults
274+
to 60 seconds and is clamped to 5–120 seconds. If extraction fails, fall back
275+
to `snapshot`; existing-session profiles do not support extraction.
268276
- The agent-facing `browser` tool exposes `action=download` (required `ref` and
269277
`path`) and `action=waitfordownload` (optional `path`). Both return the saved
270278
download URL, suggested filename, and guarded local path. Explicit download

docs/tools/browser.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ OpenClaw can run a **dedicated Chrome/Brave/Edge/Chromium profile** that the age
1818
- A separate browser profile named **openclaw** (orange accent by default).
1919
- Deterministic tab control (list/open/focus/close).
2020
- Agent actions (click/type/drag/select), snapshots, screenshots, PDFs.
21+
- Question answering over readable page text without returning a full snapshot.
2122
- Playwright-backed profiles save direct attachment navigations under the managed downloads directory and return `{ url, suggestedFilename, path }` metadata after final-URL policy validation.
2223
- Playwright-backed agent actions return a `downloads` array with the same managed metadata when the action immediately starts one or more downloads.
2324
- A bundled `browser-automation` skill that teaches agents the snapshot,
@@ -101,6 +102,12 @@ Plugin-bundled skills are listed in the agent's available skills when the
101102
plugin is enabled. The full skill instructions load on demand, so routine
102103
turns do not pay the full token cost.
103104

105+
For “read this page and answer X,” use browser `action="extract"` with a
106+
`query`. It sends sanitized, bounded readable text through one model call and
107+
returns only the answer; keep `snapshot` for choosing actions and obtaining
108+
refs. Extraction requires a Playwright-backed profile and falls back to a
109+
snapshot workflow when it cannot complete.
110+
104111
## Missing browser command or tool
105112

106113
If `openclaw browser` is unknown after an upgrade, `browser.request` is missing, or the agent reports the browser tool as unavailable, the usual cause is a `plugins.allow` list that omits `browser` and no root `browser` config block exists. Add it:

extensions/browser/index.test.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,7 @@ describe("browser plugin", () => {
239239

240240
const tool = factory({
241241
sessionKey: "agent:main:webchat:direct:123",
242+
agentId: "main",
242243
agentDir: "/tmp/agent",
243244
workspaceDir: "/tmp/workspace",
244245
activeModel: { provider: "openai", modelId: "gpt-5.5" },
@@ -251,6 +252,7 @@ describe("browser plugin", () => {
251252
await tool.execute("call-1", { action: "status" });
252253
expect(runtimeApiMocks.createBrowserTool).toHaveBeenCalledWith({
253254
agentSessionKey: "agent:main:webchat:direct:123",
255+
agentId: "main",
254256
agentDir: "/tmp/agent",
255257
workspaceDir: "/tmp/workspace",
256258
activeModel: { provider: "openai", model: "gpt-5.5" },

extensions/browser/plugin-registration.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ function createLazyBrowserTool(opts?: {
6363
sandboxBridgeUrl?: string;
6464
allowHostControl?: boolean;
6565
agentSessionKey?: string;
66+
agentId?: string;
6667
agentDir?: string;
6768
workspaceDir?: string;
6869
activeModel?: {
@@ -105,6 +106,7 @@ function createBrowserToolOptions(ctx: OpenClawPluginToolContext): {
105106
sandboxBridgeUrl?: string;
106107
allowHostControl?: boolean;
107108
agentSessionKey?: string;
109+
agentId?: string;
108110
agentDir?: string;
109111
workspaceDir?: string;
110112
activeModel?: {
@@ -126,6 +128,7 @@ function createBrowserToolOptions(ctx: OpenClawPluginToolContext): {
126128
? { allowHostControl: ctx.browser.allowHostControl }
127129
: {}),
128130
...(ctx.sessionKey ? { agentSessionKey: ctx.sessionKey } : {}),
131+
...(ctx.agentId ? { agentId: ctx.agentId } : {}),
129132
...(ctx.agentDir ? { agentDir: ctx.agentDir } : {}),
130133
...(ctx.workspaceDir ? { workspaceDir: ctx.workspaceDir } : {}),
131134
...(ctx.activeModel?.provider || ctx.activeModel?.modelId

extensions/browser/skills/browser-automation/SKILL.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ Use this skill when you need the `browser` tool for anything beyond a single pag
2121
- `suggestedTargetId` is the label when one exists, otherwise the stable `tabId` handle like `t1`.
2222
- Avoid relying on raw DevTools `targetId` except for immediate diagnostics; it can change under Chromium target replacement.
2323
3. Read before you click:
24+
- For “read the page and answer X,” use `action="extract"` with `query` so only the answer returns.
25+
- Use `action="snapshot"` instead when you need action refs or page structure.
26+
- If extract returns `NOT_FOUND` or asks for snapshot fallback, inspect the page with a snapshot.
2427
- Use `action="snapshot"` on the intended `targetId`.
2528
- Use the same `targetId` for follow-up actions so refs stay on the same tab.
2629
- For durable Playwright refs, request `refs="aria"` when supported. If you receive `axN` refs from `snapshotFormat="aria"`, use them only after that same snapshot call; stale or unbound `axN` refs fail fast and need a fresh snapshot.

0 commit comments

Comments
 (0)