feat(browser): extract action β answer page questions via a bounded sub-model call#113861
Conversation
openclaw-barnacle
Bot
added
docs
|
Live verification transcript (isolated dev gateway, managed headless Chromium, $ openclaw browser open http://127.0.0.1:8931/manual.html
$ openclaw browser extract "What is the service kit part number for pressure regulator servicing, and what torque value applies?"
[analyzed by openai/gpt-5.6-luna]
Service kit part number: SK-7741-B; torque value: 2.4 Nm.
$ openclaw browser extract "What is the price of the Alpine Anorak?" # not on this page
NOT_FOUND
$ openclaw browser extract "What is the maximum certified operating altitude?"
6,800 meters.Sub-model completions were ~450-500 ms each; answers arrived wrapped in the external-untrusted-content envelope with model attribution (envelope elided above for brevity β it is the same wrapper the vision-describe path uses). |
clawsweeper
Bot
added
proof: sufficient
|
Codex review: needs maintainer review before merge. Reviewed July 25, 2026, 4:59 PM ET / 20:59 UTC. ClawSweeper reviewWhat this changesThis PR adds a Browser plugin Merge readinessKeep this PR open for explicit maintainer product-direction review. The proposed Browser plugin feature is additive, uses existing plugin-SDK helpers rather than adding a new SDK API, and the PR includes credible terminal-based live proof; however, it introduces a new model-backed browser and CLI action, so acceptance should be intentional rather than inferred from passing implementation evidence. Priority: P3 Review scores
Verification
How this fits togetherThe Browser plugin gives agents and operators control over managed Chromium pages through tool calls, CLI commands, and the browser control server. This change adds a read-only path from the selected pageβs HTML through existing content-extraction and simple-completion helpers to a short, externally wrapped answer. flowchart LR
A[Agent or CLI question] --> B[Browser extract action]
B --> C[Playwright-managed page]
C --> D[Sanitized readable page text]
D --> E[Bounded selected-model completion]
E --> F[External-content wrapped answer]
B --> G[Snapshot fallback on failure]
Decision needed
Why: The implementation is additive and tested, but it creates a new public tool and CLI behavior with an intentional model-call and page-content-processing product contract. Before merge
Agent review detailsSecurityNone. PR surfaceSource +545, Tests +444, Docs +30. Total +1019 across 32 files. View PR surface stats
Review metricsNone. Stored data modelPersistent data-model change detected: Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Adopt Do we have a high-confidence way to reproduce the issue? Not applicable as a bug reproduction: this PR adds a new capability. The PR does provide a concrete after-fix runtime transcript covering successful extraction and the NOT_FOUND path on a managed browser profile. Is this the best way to solve the issue? Unclear until product direction is confirmed. The implementation follows the existing Browser plugin and plugin-SDK seams and keeps snapshots available, but maintainers must decide whether this new public model-backed reading mode belongs in the supported Browser contract. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 9faaa789017b. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
|
|
Rank-up move dispositions:
|
|
Merged via squash.
|
1 participant
What Problem This Solves
Answering a question about a page currently means pulling snapshots into model context β often several rounds of snapshot + scroll on long pages β which is the slowest and most token-expensive step pattern in browser sessions. The page content transits the main model's context every time, even when the caller only needs one fact.
Why This Change Was Made
Reading is separable from acting. A new
extractaction answers a question about the current page with one bounded sub-model call and returns only the answer:page.content()on the resolved tab (existing-session profiles return the standard Playwright-required 501).sanitizeHtmlβhtmlToMarkdownβnormalizeWhitespaceviaopenclaw/plugin-sdk/web-content-extractorβ the same helpers the web-fetch pipeline uses; no new SDK surface.prepareSimpleCompletionModelForAgent+completeWithPreparedSimpleCompletionModel(openclaw/plugin-sdk/simple-completion-runtime), instructed to answer strictly from page content, replyNOT_FOUNDwhen absent, and treat page text as data (prompt-injection hardening). Input is capped (80k chars, context-window-aware scaling, surrogate-safe truncation) and the whole call runs under a plugin-side deadline (default 60s, clamped 5β120s) withAbortSignalcomposition β an improvement over the vision-describe path, which relies solely on host-side media timeouts.[analyzed by <model>]attribution and{url, chars, truncated, model}details; on completion failure the caller is told to fall back tosnapshot(no partial dumps).Wiring follows the existing screenshot-describe precedent exactly: deps injected via
browserToolDeps, SDK helpers re-exported through the plugin'ssdk-setup-tools.tsbarrel. Registered across all surfaces: schema enum +queryparam + output schema, dispatch, control-server route, node-proxy allowlist, CLI (openclaw browser extract "<question>"), tool description, docs, and the bundled skill.User Impact
"Read the page and answer X" becomes one tool call, and page content stops flowing through the main model's context β the answer comes back as a short wrapped string with model attribution. The wins are context hygiene and main-model token cost in longer sessions, and correctness on pages too large to read comfortably through snapshots; snapshots remain the tool for choosing actions and refs.
Honest scoping from measurement: on a one-shot benchmark (single question per fresh agent, local fixture pages up to 400 KB,
gpt-5.6-luna), wall-clock is dominated by session cold-start and shows no significant delta vs. main, and a strong model answers small/medium pages fine from inline page content βextract's value materializes in long-lived sessions and skill-guided agents, not toy one-shots. The bundled skill now teaches it and the tool description prefers it for reading tasks.Evidence
chrome.internal.test.tsbaseline failure reproducible on plainorigin/main.git diff --checkclean.openai/gpt-5.6-luna, local 120-section fixture manual): buried-fact query returned exactlyService kit part number: SK-7741-B; torque value: 2.4 Nm; a second buried fact returned6,800 meters; a question about content not on the page returnedNOT_FOUND. Sub-model completions ~450β500 ms each; answers arrived wrapped in the external-content envelope with model attribution. Transcript in the first PR comment.