fix(duckduckgo): guard out-of-range numeric HTML entities by WeeLi-009 · Pull Request #96583 · openclaw/openclaw

WeeLi-009 · 2026-06-24T22:44:17Z

Summary: DuckDuckGo's HTML entity decoder ran String.fromCodePoint on parsed numeric entities without validation, so out-of-range values threw RangeError (breaking the whole results-page parse) and surrogate-range values produced lone surrogates. This validates the code point (in range, non-surrogate) and otherwise keeps the original entity text.

What Problem This Solves

DuckDuckGo HTML result parsing decoded numeric HTML entities with String.fromCodePoint(...) without validating the parsed code point first.

That meant hostile or malformed result text could break real parsing behavior:

&#99999999; is outside Unicode range and throws RangeError.
&#55296; / &#xD800; is in the UTF-16 surrogate range and decodes into a lone surrogate (\uD800), producing ill-formed output.
Valid supplementary entities such as 😀 should still decode to the complete emoji surrogate pair.

Fix

The DuckDuckGo entity decoder now validates numeric entity code points before decoding:

decimal and hex numeric entities are parsed into a code point first
only integer code points in 0..0x10ffff are decoded
surrogate-range code points 0xd800..0xdfff are not decoded
invalid numeric entities are preserved as escaped entity text instead of throwing or producing lone UTF-16 surrogate units

This is a range-validation fix in extensions/duckduckgo/src/ddg-client.ts. It does not use a truncation helper because this path is not slicing or truncating text; it is decoding HTML entities. The important invariant is the same UTF-16 safety rule: never emit a lone surrogate into parsed search-result text.

Evidence

Command run from an isolated worktree checked out at fix/duckduckgo-entity-range:

node --import tsx demo.mts

Terminal output:

CASE surrogate decimal entity
INPUT Bad &#55296; end
BEFORE output="Bad \ud800 end"
BEFORE codeUnits=0042 0061 0064 0020 d800 0020 0065 006e 0064
BEFORE lone-surrogate=true
AFTER output="Bad &#55296; end"
AFTER codeUnits=0042 0061 0064 0020 0026 0023 0035 0035 0032 0039 0036 003b 0020 0065 006e 0064
AFTER lone-surrogate=false

CASE surrogate hex entity
INPUT Bad &#xD800; end
BEFORE output="Bad \ud800 end"
BEFORE codeUnits=0042 0061 0064 0020 d800 0020 0065 006e 0064
BEFORE lone-surrogate=true
AFTER output="Bad &#xD800; end"
AFTER codeUnits=0042 0061 0064 0020 0026 0023 0078 0044 0038 0030 0030 003b 0020 0065 006e 0064
AFTER lone-surrogate=false

CASE out-of-range entity
INPUT Bad &#99999999; end
BEFORE error=RangeError
AFTER output="Bad &#99999999; end"
AFTER codeUnits=0042 0061 0064 0020 0026 0023 0039 0039 0039 0039 0039 0039 0039 0039 003b 0020 0065 006e 0064
AFTER lone-surrogate=false

CASE valid supplementary emoji entity
INPUT Smile &#128512; end
BEFORE output="Smile 😀 end"
BEFORE codeUnits=0053 006d 0069 006c 0065 0020 d83d de00 0020 0065 006e 0064
BEFORE lone-surrogate=false
AFTER output="Smile 😀 end"
AFTER codeUnits=0053 006d 0069 006c 0065 0020 d83d de00 0020 0065 006e 0064
AFTER lone-surrogate=false

The demo calls the real fixed testing.decodeHtmlEntities entry from extensions/duckduckgo/src/ddg-client.ts. After the fix, surrogate-range and out-of-range entities stay escaped, valid emoji entities still decode, and every AFTER case reports lone-surrogate=false.

Tests

Focused regression coverage is in extensions/duckduckgo/src/ddg-search-provider.test.ts:

invalid decimal numeric entity stays intact instead of throwing
invalid hex numeric entity stays intact
surrogate-range decimal and hex entities stay intact
valid supplementary-plane emoji entity still decodes

Validation run:

node scripts/run-vitest.mjs extensions/duckduckgo/src/ddg-search-provider.test.ts

Result:

Test Files  1 passed (1)
Tests  10 passed (10)

clawsweeper · 2026-06-24T22:45:27Z

Codex review: needs maintainer review before merge. Reviewed June 26, 2026, 2:53 AM ET / 06:53 UTC.

Summary
The branch adds a DuckDuckGo numeric HTML entity code-point guard and regression tests for out-of-range, surrogate-range, and valid supplementary-plane entities.

PR surface: Source +6, Tests +14. Total +20 across 2 files.

Reproducibility: yes. source-reproducible: current main routes DuckDuckGo title, URL, and snippet fields through unchecked numeric entity decoding that can throw or emit lone surrogates.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Next step before merge

[P2] The PR already contains the narrow fix and regression test; remaining action is maintainer review and normal merge gates, not an automated repair branch.

Security
Cleared: The diff only changes local DuckDuckGo string decoding and tests; it adds no dependency, workflow, package, secret, permission, install, or code-execution surface.

Review details

Best possible solution:

Land this plugin-local decoder guard with regression coverage after maintainer review; keep Telegram and web-fetch decoder work in their own surface-specific PRs.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main routes DuckDuckGo title, URL, and snippet fields through unchecked numeric entity decoding that can throw or emit lone surrogates.

Is this the best way to solve the issue?

Yes. The plugin-local code-point guard is the narrowest maintainable fix; a broader shared decoder refactor is unnecessary for this PR, and adjacent Telegram/web-fetch surfaces are tracked separately.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e9f9a68d68c9.

Label changes

Label justifications:

P2: Malformed DuckDuckGo HTML can break or corrupt web-search results, but the blast radius is limited to one bundled plugin parser.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal before/after output from an isolated worktree calling the real DuckDuckGo decoder test export, plus a focused test command and passing result.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal before/after output from an isolated worktree calling the real DuckDuckGo decoder test export, plus a focused test command and passing result.

Evidence reviewed

PR surface:

Source +6, Tests +14. Total +20 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	8	2	+6
Tests	1	14	0	+14
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	22	2	+20

Acceptance criteria:

[P1] node scripts/run-vitest.mjs extensions/duckduckgo/src/ddg-search-provider.test.ts.

What I checked:

Root policy read: Root AGENTS.md was read fully; its review guidance requires beyond-diff source, tests, history, and scoped policy checks for PR verdicts. (AGENTS.md:1, e9f9a68d68c9)
Scoped extension policy: extensions/AGENTS.md keeps bundled plugin runtime behavior inside the owning plugin boundary; this PR stays within DuckDuckGo plugin files. (extensions/AGENTS.md:1, e9f9a68d68c9)
Current main failure path: Current main sends parsed decimal and hex numeric entity values directly to String.fromCodePoint without range or surrogate checks. (extensions/duckduckgo/src/ddg-client.ts:74, e9f9a68d68c9)
Parser impact: DuckDuckGo result title, URL, and snippet values all flow through decodeHtmlEntities before being returned from the provider. (extensions/duckduckgo/src/ddg-client.ts:137, e9f9a68d68c9)
Runtime entry point: The provider loads ddg-client and calls runDuckDuckGoSearch from tool execution, so the decoder sits on the model-visible DuckDuckGo web_search path. (extensions/duckduckgo/src/ddg-search-provider.ts:45, e9f9a68d68c9)
Latest release behavior: v2026.6.10 still has unchecked String.fromCodePoint calls for DuckDuckGo numeric entities and no invalid-entity regression coverage in the adjacent test range. (extensions/duckduckgo/src/ddg-client.ts:52, aa69b12d0086)

Likely related people:

vincentkoc: Commit c6ca11e introduced the bundled DuckDuckGo plugin and its parser, tests, and provider files through the merged web-search plugin PR. (role: feature introducer; confidence: high; commits: c6ca11e5a5d0; files: extensions/duckduckgo/src/ddg-client.ts, extensions/duckduckgo/src/ddg-search-provider.test.ts, extensions/duckduckgo/src/ddg-search-provider.ts)
Narahari Raghava: git blame on current main attributes the active DuckDuckGo decoder and parse lines to cf512f6, which carried these files into the current tree. (role: current-source carrier; confidence: medium; commits: cf512f639b31; files: extensions/duckduckgo/src/ddg-client.ts, extensions/duckduckgo/src/ddg-search-provider.test.ts, extensions/duckduckgo/src/ddg-search-provider.ts)
steipete: The merged web_search schema PR exercised the DuckDuckGo web_search provider path and runtime contract adjacent to this parser. (role: adjacent web-search contributor; confidence: medium; commits: e360e105a303; files: src/agents/tools/web-search.ts, src/agents/tools/web-search.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

decodeHtmlEntities decoded numeric entities with String.fromCodePoint(parseInt(...)) without a range check, so an out-of-range entity such as &#99999999; or &#x110000; threw RangeError and made the whole results page fail to parse. Validate the code point is within 0..0x10FFFF and keep the original entity text otherwise. Add a regression covering decimal and hex out-of-range entities plus a valid astral entity.

…6583) decodeHtmlEntities decoded numeric entities with String.fromCodePoint(parseInt(...)) without a range check, so an out-of-range entity such as &#99999999; or &#x110000; threw RangeError and made the whole results page fail to parse. Validate the code point is within 0..0x10FFFF and keep the original entity text otherwise. Add a regression covering decimal and hex out-of-range entities plus a valid astral entity.

…6583) decodeHtmlEntities decoded numeric entities with String.fromCodePoint(parseInt(...)) without a range check, so an out-of-range entity such as &#99999999; or &#x110000; threw RangeError and made the whole results page fail to parse. Validate the code point is within 0..0x10FFFF and keep the original entity text otherwise. Add a regression covering decimal and hex out-of-range entities plus a valid astral entity. (cherry picked from commit 2720ac0)

…6583) decodeHtmlEntities decoded numeric entities with String.fromCodePoint(parseInt(...)) without a range check, so an out-of-range entity such as &#99999999; or &#x110000; threw RangeError and made the whole results page fail to parse. Validate the code point is within 0..0x10FFFF and keep the original entity text otherwise. Add a regression covering decimal and hex out-of-range entities plus a valid astral entity.

openclaw-barnacle Bot added extensions: duckduckgo size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 24, 2026

WeeLi-009 force-pushed the fix/duckduckgo-entity-range branch from 5da140b to 7f8dc7b Compare June 25, 2026 00:22

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 25, 2026

github-actions Bot mentioned this pull request Jun 26, 2026

🦞 OpenClaw 生态日报 2026-06-26 96loveslife/big_model_radar#60

Open

WeeLi-009 force-pushed the fix/duckduckgo-entity-range branch from 7f8dc7b to 7c44ffd Compare June 26, 2026 06:24

vincentkoc merged commit 2720ac0 into openclaw:main Jun 27, 2026
86 checks passed

github-actions Bot mentioned this pull request Jun 27, 2026

📡 Upstream Digest — 2026-06-27 16:59 UTC curtismercier/openclaw-mods#1160

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(duckduckgo): guard out-of-range numeric HTML entities#96583

fix(duckduckgo): guard out-of-range numeric HTML entities#96583
vincentkoc merged 1 commit into
openclaw:mainfrom
WeeLi-009:fix/duckduckgo-entity-range

WeeLi-009 commented Jun 24, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

WeeLi-009 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Fix

Evidence

Tests

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

WeeLi-009 commented Jun 24, 2026 •

edited

Loading

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading