Skip to content

fix(agents): quarantine normalized runtime tools#88994

Closed
vincentkoc wants to merge 1 commit into
mainfrom
fuzz-provider-tool-context-20260601
Closed

fix(agents): quarantine normalized runtime tools#88994
vincentkoc wants to merge 1 commit into
mainfrom
fuzz-provider-tool-context-20260601

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jun 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Quarantines provider-normalized runtime tools that still contain schemas unsupported by the assistant/runtime projection.
  • Preserves plugin/channel/MCP ownership metadata on readable normalized clones before reporting diagnostics, so warnings still identify the owning plugin surface.
  • Hardens metadata preservation against hostile provider-normalized results with unreadable names or throwing array slots.
  • Renames the diagnostic callback from pre-normalization-only to general tool-schema diagnostics because callers now receive both pre-provider and post-provider quarantine events.

Linked context

Closes #

Related #88936
Related #88950
Related #88959
Related #88977

Was this requested by a maintainer or owner?

Maintainer fuzzing/hardening sweep for provider/tool-schema runtime crash surfaces.

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: provider tool-schema normalization could return runtime-incompatible or hostile normalized tools that reached assistant/Codex dynamic-tool paths before final quarantine.
  • Real environment tested: local macOS worktree with Node wrapper tests, plus AWS Crabbox Linux changed gate.
  • Exact steps or command run after this patch: node scripts/run-vitest.mjs src/agents/runtime-plan/tools.test.ts src/agents/runtime-plan/tools.diagnostics.test.ts --reporter=dot; node scripts/run-vitest.mjs src/agents/tools-effective-inventory.test.ts src/commands/doctor/shared/active-tool-schema-warnings.test.ts src/agents/embedded-agent-runner/run/attempt.spawn-workspace.context-engine.test.ts extensions/codex/src/app-server/dynamic-tool-build.test.ts src/flows/doctor-core-checks.runtime.test.ts src/flows/doctor-core-checks.runtime-errors.test.ts --reporter=dot; ./node_modules/.bin/oxfmt --check --threads=1 ...; OPENCLAW_OXLINT_SKIP_LOCK=1 node scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.json ...; git diff --check origin/main...HEAD; node scripts/crabbox-wrapper.mjs run --provider aws --idle-timeout 90m --ttl 240m --timing-json --shell -- "env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 corepack pnpm check:changed".
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output): local focused tests passed 14 runtime-plan tests and 143 caller/doctor/Codex-path tests; AWS Crabbox changed gate passed with provider aws, run run_ba62f5cbca01, lease cbx_54d56c47e7e8, exit 0, lease stopped.
  • Observed result after fix: normalized tools with $dynamicRef are quarantined after provider normalization; unreadable normalized tool names and throwing normalized array slots are quarantined without crashing metadata preservation; healthy normalized tools still keep plugin/MCP metadata.
  • What was not tested: live provider/plugin runtime with private plugin data; Blacksmith Testbox proof was blocked by missing local Blacksmith auth.
  • Proof limitations or environment constraints: direct Blacksmith check failed with not authenticated - run 'blacksmith auth login' first, so broad proof used AWS Crabbox instead.
  • Before evidence (optional but encouraged): current main only filtered before provider normalization, allowing provider-normalized incompatible/hostile tools to continue into the returned runtime tool list.

Tests and validation

Which commands did you run?

  • node scripts/run-vitest.mjs src/agents/runtime-plan/tools.test.ts src/agents/runtime-plan/tools.diagnostics.test.ts --reporter=dot
  • node scripts/run-vitest.mjs src/agents/tools-effective-inventory.test.ts src/commands/doctor/shared/active-tool-schema-warnings.test.ts src/agents/embedded-agent-runner/run/attempt.spawn-workspace.context-engine.test.ts extensions/codex/src/app-server/dynamic-tool-build.test.ts src/flows/doctor-core-checks.runtime.test.ts src/flows/doctor-core-checks.runtime-errors.test.ts --reporter=dot
  • ./node_modules/.bin/oxfmt --check --threads=1 extensions/codex/src/app-server/dynamic-tool-build.ts src/agents/embedded-agent-runner/run/attempt.ts src/agents/runtime-plan/tools.test.ts src/agents/runtime-plan/tools.ts src/agents/tools-effective-inventory-build.ts src/agents/tools-effective-mcp-inventory.ts src/commands/doctor/shared/active-tool-schema-warnings.ts src/flows/doctor-core-checks.runtime.ts
  • OPENCLAW_OXLINT_SKIP_LOCK=1 node scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.json extensions/codex/src/app-server/dynamic-tool-build.ts src/agents/embedded-agent-runner/run/attempt.ts src/agents/runtime-plan/tools.test.ts src/agents/runtime-plan/tools.ts src/agents/tools-effective-inventory-build.ts src/agents/tools-effective-mcp-inventory.ts src/commands/doctor/shared/active-tool-schema-warnings.ts src/flows/doctor-core-checks.runtime.ts
  • git diff --check origin/main...HEAD
  • .agents/skills/autoreview/scripts/autoreview --mode commit --commit HEAD
  • node scripts/crabbox-wrapper.mjs run --provider aws --idle-timeout 90m --ttl 240m --timing-json --shell -- "env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 corepack pnpm check:changed"

What regression coverage was added or updated?

  • Added runtime-plan coverage for provider-normalized $dynamicRef quarantine after provider normalization.
  • Added hostile normalized-tool regressions for unreadable normalized names and throwing normalized array slots.
  • Updated caller diagnostics to consume general tool-schema diagnostics rather than only pre-normalization diagnostics.

What failed before this fix, if known?

  • A provider-normalized tool that stayed runtime-incompatible could still be returned by normalizeAgentRuntimeTools.
  • Hostile normalized tool results could be touched by metadata preservation before safe runtime projection.

If no test was added, why not?

N/A.

Risk checklist

Did user-visible behavior change? (Yes/No)

Yes. Unsupported provider-normalized runtime tools are now quarantined instead of passed to assistant runtime paths.

Did config, environment, or migration behavior change? (Yes/No)

No.

Did security, auth, secrets, network, or tool execution behavior change? (Yes/No)

Yes. This hardens tool execution registration boundaries by preventing unsupported/hostile normalized tool definitions from reaching runtime paths.

What is the highest-risk area?

Tool availability for providers/plugins that depend on schema normalization.

How is that risk mitigated?

The pre-provider filter still lets provider-normalizable schemas reach normalization hooks, and the new filter runs only after hooks have had their chance to produce runtime-compatible schemas. Healthy normalized tools are preserved and retain ownership metadata.

Current review state

What is the next action?

Maintainer review.

What is still waiting on author, maintainer, CI, or external proof?

Blacksmith Testbox proof is blocked by missing local auth; AWS Crabbox changed gate passed instead.

Which bot or reviewer comments were addressed?

Local autoreview first caught a test typecheck issue in the diagnostic source accumulator; fixed and rerun. Final autoreview is clean: no accepted/actionable findings, overall: patch is correct (0.83).

@vincentkoc vincentkoc self-assigned this Jun 1, 2026
@openclaw-barnacle openclaw-barnacle Bot added commands Command implementations agents Agent runtime and tooling extensions: codex size: M maintainer Maintainer-authored PR labels Jun 1, 2026
@clawsweeper

clawsweeper Bot commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 1, 2026, 4:04 AM ET / 08:04 UTC.

Summary
The PR adds a post-provider runtime-tool compatibility filter, preserves tool ownership metadata during normalization, and updates diagnostics from pre-normalization-only to general tool-schema quarantine reporting.

PR surface: Source +48, Tests +152. Total +200 across 8 files.

Reproducibility: yes. source-reproducible rather than locally rerun in this read-only review. Current main filters before provider normalization but does not re-filter the normalized return from normalizeAgentRuntimeTools, and the PR adds a regression case for a provider-normalized $dynamicRef schema.

Review metrics: 1 noteworthy metric.

  • Runtime compatibility gate: 1 post-provider filter added. This is the merge-relevant behavior change because unsupported normalized tools are now removed at the shared runtime normalization boundary.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] Merging intentionally quarantines provider-normalized tools that still contain runtime-unsupported schemas; existing plugin/provider setups relying on those tools will see them unavailable rather than passed onward.

Maintainer options:

  1. Accept quarantine as hardening (recommended)
    Maintainers can accept that unsupported provider-normalized schemas become unavailable because the PR keeps provider normalization first and reports owner-aware diagnostics.
  2. Require live plugin upgrade proof
    If maintainers suspect active private plugins depend on these normalized schemas, ask for one live provider/plugin run that shows healthy tools remain registered and only unsupported schemas are quarantined.
  3. Pause if availability is contractual
    If any affected provider/plugin treats these schemas as a supported contract, pause this PR and move the compatibility decision into a narrower migration or opt-in path.

Next step before merge

  • [P2] The PR has a protected maintainer label and an intentional tool-availability compatibility risk, so the next action is maintainer review rather than cleanup closure or automated repair.

Security
Cleared: No concrete security or supply-chain regression was found; the diff does not touch dependencies, workflows, secrets, or downloads and instead tightens tool registration boundaries.

Review details

Best possible solution:

Land the centralized post-normalization quarantine only after maintainers accept the intentional tool-availability behavior and final CI/proof remain green.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible rather than locally rerun in this read-only review. Current main filters before provider normalization but does not re-filter the normalized return from normalizeAgentRuntimeTools, and the PR adds a regression case for a provider-normalized $dynamicRef schema.

Is this the best way to solve the issue?

Yes, this is the best fix shape among the inspected options. Centralizing the post-provider quarantine in normalizeAgentRuntimeTools covers embedded, doctor, inventory, and Codex callers without duplicating caller-specific guards, while caller-side Codex projection remains a later defense.

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 6627b4fbdd31.

Label changes

Label changes:

  • add P2: This is a bounded agent/provider hardening fix with real runtime impact but no evidence of an urgent live outage in the PR context.
  • add merge-risk: 🚨 compatibility: The patch can make existing provider/plugin tools unavailable after normalization when their schemas still violate the runtime projection contract.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides terminal-style after-fix proof: focused Node wrapper tests, formatting/lint checks, clean autoreview, and AWS Crabbox changed gate exit 0 for the patch.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body provides terminal-style after-fix proof: focused Node wrapper tests, formatting/lint checks, clean autoreview, and AWS Crabbox changed gate exit 0 for the patch.

Label justifications:

  • P2: This is a bounded agent/provider hardening fix with real runtime impact but no evidence of an urgent live outage in the PR context.
  • merge-risk: 🚨 compatibility: The patch can make existing provider/plugin tools unavailable after normalization when their schemas still violate the runtime projection contract.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body provides terminal-style after-fix proof: focused Node wrapper tests, formatting/lint checks, clean autoreview, and AWS Crabbox changed gate exit 0 for the patch.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides terminal-style after-fix proof: focused Node wrapper tests, formatting/lint checks, clean autoreview, and AWS Crabbox changed gate exit 0 for the patch.
Evidence reviewed

PR surface:

Source +48, Tests +152. Total +200 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 7 89 41 +48
Tests 1 156 4 +152
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 8 245 45 +200

What I checked:

  • PR implementation: The PR commit filters provider-normalized tools with filterRuntimeCompatibleTools after metadata preservation and reports those diagnostics through the renamed callback before returning only compatible tools. (src/agents/runtime-plan/tools.ts:158, 8fd9d208bb74)
  • Current main behavior: Current main prefilters provider-normalizable tools before provider normalization but returns the normalized list without a second runtime compatibility projection at this helper boundary. (src/agents/runtime-plan/tools.ts:89, 12d5043913)
  • Caller boundary: The Codex dynamic-tool bridge projects tool.parameters to inputSchema and quarantines projection violations before creating Codex dynamic-tool specs, so the changed helper tightens the earlier shared normalization boundary rather than adding the only caller-side guard. (extensions/codex/src/app-server/dynamic-tools.ts:309, 6627b4fbdd31)
  • Codex dependency contract: Upstream Codex dynamic tools carry raw input_schema JSON and app-server validates each dynamic tool with parse_tool_input_schema, returning an error for unsupported schemas, so preventing unsupported OpenClaw schemas before registration matches the dependency boundary. (../codex/codex-rs/app-server/src/request_processors/thread_processor.rs:316, cf0911076f23)
  • Release/API provenance: The latest release tag did not yet include the diagnostic callback added on current main, so this PR's callback rename is main-only cleanup rather than removal of a shipped callback contract. (src/agents/runtime-plan/tools.ts:11, e93216080aa1)
  • Validation evidence from PR body: The PR body reports focused runtime-plan and caller/doctor/Codex-path tests, oxfmt, oxlint, diff check, clean local autoreview, and AWS Crabbox changed gate run_ba62f5cbca01 exit 0. (8fd9d208bb74)

Likely related people:

  • vincentkoc: Current-main blame and history for the runtime tool normalization/projection helper point to the same recent area work, and this PR continues that provider/tool-schema hardening line. (role: recent area contributor; confidence: high; commits: 12d5043913, 8fd9d208bb74; files: src/agents/runtime-plan/tools.ts, src/agents/tool-schema-projection.ts, extensions/codex/src/app-server/dynamic-tool-build.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jun 1, 2026
This was referenced Jun 1, 2026
Closed
Closed
@clawsweeper clawsweeper Bot mentioned this pull request Jun 1, 2026
Closed
@vincentkoc vincentkoc mentioned this pull request Jun 8, 2026
Closed
@vincentkoc

vincentkoc commented Jun 8, 2026

Copy link
Copy Markdown
Member Author

Closing this rather than landing a compatibility-risk runtime change from the cleanup queue.

This is the most plausible real hardening branch in the batch, because it quarantines provider-normalized tools after normalization. But it is still draft, marked compatibility risk, touches shared runtime/doctor/Codex surfaces, and has no linked release-blocking incident. That combination is not "critical enough to land now"; it needs a focused owner pass if we want it back.

Reopen path: a current-main provider normalization crash or assistant runtime failure where an incompatible normalized tool reaches execution, with repro and owner-reviewed compatibility story.

@vincentkoc vincentkoc closed this Jun 8, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 14, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

agents Agent runtime and tooling commands Command implementations extensions: codex maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vincentkoc