Skip to content

fix(logging): keep redacted token hints UTF-16 safe#103341

Merged
steipete merged 1 commit into
openclaw:mainfrom
zhangguiping-xydt:fix/problem-logging-redact-utf16-safe
Jul 10, 2026
Merged

fix(logging): keep redacted token hints UTF-16 safe#103341
steipete merged 1 commit into
openclaw:mainfrom
zhangguiping-xydt:fix/problem-logging-redact-utf16-safe

Conversation

@zhangguiping-xydt

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes an issue where operators logging non-ASCII secrets could receive malformed redacted token hints when a surrogate pair crossed the retained first 6 or last 4 UTF-16 code units. The redaction step could introduce an unpaired surrogate into JSON-formatted log output even though the original secret was well-formed.

Why This Change Was Made

This is a root-cause fix in the shared token hint constructor. It uses the repository's canonical surrogate-safe UTF-16 slicing helper for both retained edges, so every existing caller gets the same boundary guarantee.

ASCII hints keep the existing 6/4 code-unit budget, short tokens remain fully masked, and complete surrogate pairs that fit inside the budget remain visible. This does not change redaction patterns, logging configuration, public APIs, schemas, protocols, or persistence formats. A consumer sweep confirmed registered secrets, regex matches, URL/form values, authorization codes, and structured sensitive fields all converge on the same helper.

User Impact

Unicode secrets now produce well-formed redacted hints, so JSON log consumers no longer receive surrogate halves introduced by redaction. Existing ASCII token hints are unchanged; when an emoji would straddle a hint edge, the output retains one fewer code unit instead of emitting half of the character.

Evidence

Before the fix, the focused regression failed with:

Expected: "abcde…wxyz"
Received: "abcde�…wxyz"

Local E2E live proof on the patched checkout:

  • Started a real OpenClaw gateway and observed HTTP 200 from /healthz.
  • Sent three synthetic Unicode token assignments through the production subsystem logger in JSON console mode.
  • Parsed all three emitted JSON records and verified every output string was UTF-16 well-formed.
  • Confirmed none of the raw synthetic secrets appeared in the output.
{"level":"info","subsystem":"gateway/auth","message":"TOKEN=abcde…wxyz"}
{"level":"info","subsystem":"gateway/auth","message":"TOKEN=abcdef…abc"}
{"level":"info","subsystem":"gateway/auth","message":"TOKEN=abcd😀…😀ab"}

Focused regression:

Test Files  1 passed (1)
Tests       117 passed (117)

Additional checks:

  • pnpm exec oxfmt --check --threads=1 src/logging/redact.ts src/logging/redact.test.ts
  • node scripts/run-oxlint.mjs src/logging/redact.ts src/logging/redact.test.ts
  • pnpm tsgo:core
  • pnpm tsgo:test:src
  • git diff --check

The pre-submit scan found no related open PR, and the refreshed main branch still used ordinary String.prototype.slice for both hint edges. The change is limited to the shared hint construction and its regression coverage.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 10, 2026
@clawsweeper

clawsweeper Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 10, 2026, 6:17 AM ET / 10:17 UTC.

Summary
The PR changes the shared token-hint masker to use surrogate-safe UTF-16 slicing for both retained edges and adds regression coverage for split and intact emoji boundaries.

PR surface: Source +1, Tests +23. Total +24 across 2 files.

Reproducibility: yes. Current main can split a valid surrogate pair at either fixed code-unit hint boundary, and the proposed cases directly exercise both failures.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Next step before merge

  • [P2] The active PR is correct and needs ordinary branch refresh and maintainer landing, not a separate ClawSweeper repair job.

Security
Cleared: The patch preserves masking, can only retain the same or fewer hint code units, and adds no dependency, permission, execution, secret-access, or supply-chain surface.

Review details

Best possible solution:

Refresh the branch against current main, retain the shared-helper implementation and focused cases, then land it after exact-head checks remain green.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main can split a valid surrogate pair at either fixed code-unit hint boundary, and the proposed cases directly exercise both failures.

Is this the best way to solve the issue?

Yes. Fixing the single shared maskToken constructor with the existing normalization helper is narrower and less drift-prone than repairing individual logging consumers.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f1b75bf5fa34.

Label changes

Label changes:

  • add P2: The PR fixes malformed Unicode in redacted operational logs through a narrow shared-code change with limited blast radius.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides after-fix live gateway output through the production subsystem logger, parsed JSON for both split edges and an intact-pair case, plus confirmation that no raw synthetic secret appeared.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body provides after-fix live gateway output through the production subsystem logger, parsed JSON for both split edges and an intact-pair case, plus confirmation that no raw synthetic secret appeared.

Label justifications:

  • P2: The PR fixes malformed Unicode in redacted operational logs through a narrow shared-code change with limited blast radius.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body provides after-fix live gateway output through the production subsystem logger, parsed JSON for both split edges and an intact-pair case, plus confirmation that no raw synthetic secret appeared.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides after-fix live gateway output through the production subsystem logger, parsed JSON for both split edges and an intact-pair case, plus confirmation that no raw synthetic secret appeared.
Evidence reviewed

PR surface:

Source +1, Tests +23. Total +24 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 3 2 +1
Tests 1 23 0 +23
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 26 2 +24

What I checked:

  • Current-main defect: The shared token-hint constructor on current main retains the first six and last four UTF-16 code units with raw String.prototype.slice, which can leave an unpaired surrogate at either edge. (src/logging/redact.ts:357, f1b75bf5fa34)
  • Canonical helper contract: The existing normalization helper supports negative indices and removes dangling low or high surrogate halves at the resulting slice boundaries. (packages/normalization-core/src/utf16-slice.ts:16, f1b75bf5fa34)
  • Patch implementation: The PR routes both retained token-hint edges through sliceUtf16Safe without changing the short-token threshold, hint budgets, or masking format. (src/logging/redact.ts:356, 7968f6caa125)
  • Regression coverage: The added cases cover a surrogate pair split at the leading edge, a split at the trailing edge, and complete pairs that fit within both budgets; the assertion rejects unpaired surrogates while accepting valid emoji pairs. (src/logging/redact.test.ts:312, 7968f6caa125)
  • Validation and real behavior: The current head has passing lint, type, dependency, build, security, test, and real-behavior-proof checks; the PR body also records parsed after-fix JSON from a real gateway production subsystem logger with no raw synthetic secret present. (7968f6caa125)
  • Redaction history: Recent merged history shows steipete extending the shared redaction path for registered secrets and amknight implementing the broad tool-output redaction surface that uses the same masker. (src/logging/redact.ts:350, 4bf70be01a21)

Likely related people:

  • steipete: Recent merged work extended the shared redaction path and maintained the UTF-16 slicing helper used by this patch. (role: recent area contributor; confidence: high; commits: 4bf70be01a21, b22c36f1125c, 062f88e3e3af; files: src/logging/redact.ts, packages/normalization-core/src/utf16-slice.ts)
  • amknight: The broad merged tool-output secret-redaction implementation substantially shaped the current shared redaction module and its masker call paths. (role: redaction feature contributor; confidence: medium; commits: 3faf669801d0; files: src/logging/redact.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete

Copy link
Copy Markdown
Contributor

Land-ready proof for exact head 7968f6caa125f7de52d98a1f469d6c1f22911c7c:

  • Hosted CI: 65 relevant checks passed; no failed or pending checks.
  • Sanitized direct AWS Crabbox run run_6eef08122fdc on lease cbx_4a4eff4eae05: the trusted bootstrap verified the exact head and absence of IAM credentials, isolated HOME, installed the pinned toolchain, and passed pnpm test src/logging/redact.test.ts — 117/117 tests.
  • Fresh branch autoreview: clean, overall patch-correct confidence 0.99.
  • git diff --check: passed.
  • Native review artifacts validate with zero findings, and OPENCLAW_TESTBOX=1 scripts/pr prepare-run 103341 passed without changing the PR head.

The best-fix review traced all hint-producing callers to the private maskToken() owner. The change preserves the six/four UTF-16-unit disclosure budgets and only backs off an edge that would otherwise emit a surrogate half.

Known proof gaps: none.

@steipete
steipete merged commit 328eaa3 into openclaw:main Jul 10, 2026
136 of 144 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

steipete added a commit that referenced this pull request Jul 10, 2026
Completes the sibling memory-host masker missed by #103341.

Co-authored-by: 张贵萍0668001030 <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 11, 2026
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 11, 2026
)

Completes the sibling memory-host masker missed by openclaw#103341.

Co-authored-by: 张贵萍0668001030 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants