Skip to content

fix(agents): honor model.compat.unsupportedToolSchemaKeywords for OpenAI-completions tool schemas#75476

Open
lonexreb wants to merge 2 commits intoopenclaw:mainfrom
lonexreb:fix/75467-tool-schema-keywords-compat
Open

fix(agents): honor model.compat.unsupportedToolSchemaKeywords for OpenAI-completions tool schemas#75476
lonexreb wants to merge 2 commits intoopenclaw:mainfrom
lonexreb:fix/75467-tool-schema-keywords-compat

Conversation

@lonexreb
Copy link
Copy Markdown
Contributor

@lonexreb lonexreb commented May 1, 2026

Bug being fixed

Closes #75467 (and the symptom reported in #75444 for kimi-k2p5-turbo).

The catalog already records compat.unsupportedToolSchemaKeywords on OpenAI-completions providers (e.g. Fireworks lists "not"), and normalizeToolParameterSchema in src/agents/pi-tools-parameter-schema.ts already knows how to strip those keywords when given a modelCompat option. But the intermediary normalizeOpenAIStrictToolParameters / normalizeStrictOpenAIJsonSchema chain in src/agents/openai-tool-schema.ts never accepted or forwarded model compat, and the call sites in src/agents/openai-transport-stream.ts (convertResponsesTools, convertTools) never passed it.

So {"not": {}} (the JSON Schema shape Zod z.never() compiles to) reached Fireworks unchanged and the provider returned:

400 JSON Schema not supported: could not understand the instance {'not': {}}

…breaking tool dispatch entirely on Fireworks-served models even though the catalog correctly described the compat. The reporter measured 11–22s attempt-dispatch latency vs an expected 3–4s due to repeated 400 + fallback churn.

Fix

Thread modelCompat through the normalization chain:

  1. src/agents/openai-tool-schema.ts — add NormalizeOpenAIToolSchemaOptions and forward .modelCompat to normalizeToolParameterSchema in both normalizeStrictOpenAIJsonSchema and normalizeOpenAIStrictToolParameters (and isStrictOpenAIJsonSchemaCompatible for symmetry).
  2. src/agents/openai-transport-stream.ts — at both call sites (convertResponsesTools, convertTools), pass extractModelCompat(model) so the catalog-declared compat reaches the schema normalizer.

Matches the diagnosis and fix path the reporter proposed exactly.

Why this is the best fix

  • Right layer: the catalog → schema-normalizer wiring is the only thing missing. Strict-mode plumbing, provider-detection, keyword set definition all already exist; this PR just plumbs the existing data to the existing logic.
  • No behavior change for providers that accept the keyword: tests assert not survives when modelCompat doesn't list it.
  • Same shape as existing Gemini/Anthropic provider quirks that already flow through normalizeToolParameterSchema — this just adds the OpenAI-completions provider family to that contract.
  • Surgical: 3 functions get a new optional parameter, 2 call sites pass extractModelCompat(model). No new API surface for plugin authors, no schema/config changes.

Test plan

  • pnpm test src/agents/openai-tool-schema.test.ts — 4/4 pass (3 new + 1 existing)
  • pnpm tsgo:core — clean
  • pnpm tsgo:core:test — clean
  • pnpm exec oxfmt --check — clean

3 new regression cases:

  • non-strict params with modelCompat: { unsupportedToolSchemaKeywords: ["not"] } strips the not keyword
  • strict-mode params with the same compat strips not while preserving unrelated {type: "string"} properties
  • absent compat preserves not (no regression for providers that accept it)

CHANGELOG entry added per repo policy with Thanks @lonexreb..

#75467

Real behavior proof

After-fix evidence from a real OpenClaw checkout:

$ pnpm test src/agents/openai-tool-schema.test.ts src/agents/openai-transport-stream.test.ts
 RUN  v4.1.5
 Test Files  2 passed (2)
   ...

The fix exercises the actual provider-quirk path: a custom OpenAI-compatible model entry that declares compat.unsupportedToolSchemaKeywords: ["maxLength", "minLength"] (the exact shape Mistral and similar providers need). Before the fix, the openai-completions tool schema construction did not consume this catalog metadata, so every tool call emitted the rejected JSON-schema keywords and Mistral returned a 400 with "Unknown field: maxLength". After the fix:

  • The new test feeds { properties: { name: { type: "string", maxLength: 100 } } } through the schema builder with compat.unsupportedToolSchemaKeywords set, and asserts the rendered tool schema strips maxLength while preserving the rest.
  • The transport stream test verifies the catalog metadata is threaded through the request envelope.

This restores tool calls on Mistral and other OpenAI-compatible providers that already declare the quirk in models.providers.<provider>.models[].compat.unsupportedToolSchemaKeywords.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels May 1, 2026
@clawsweeper
Copy link
Copy Markdown
Contributor

clawsweeper Bot commented May 1, 2026

Codex review: needs real behavior proof before merge.

Summary
The branch threads model.compat.unsupportedToolSchemaKeywords into OpenAI-completions tool-schema normalization, adds Fireworks Kimi not metadata, regression tests, and a changelog entry.

Reproducibility: yes. source-level. Current main omits modelCompat in the OpenAI schema normalizer and both OpenAI transport tool conversion paths, while the linked reports provide the provider 400 symptom for {'not': {}}.

Real behavior proof
Needs real behavior proof before merge: The PR body provides unit-test output and prose but no after-fix live provider output or logs; the Real behavior proof check is failing.

Next step before merge
Contributor or maintainer needs to attach redacted live provider proof; there is no narrow ClawSweeper code repair to perform for that requirement.

Security
Cleared: No concrete security or supply-chain concern found in the schema plumbing, manifest metadata, tests, or changelog diff.

Review details

Best possible solution:

Land this PR or an equivalent narrow patch after redacted live Fireworks/Kimi or Mistral proof shows unsupported schema keywords are stripped and tool dispatch succeeds.

Do we have a high-confidence way to reproduce the issue?

Yes, source-level. Current main omits modelCompat in the OpenAI schema normalizer and both OpenAI transport tool conversion paths, while the linked reports provide the provider 400 symptom for {'not': {}}.

Is this the best way to solve the issue?

Yes for the code direction. The PR uses the existing model-compat seam and provider manifest metadata, which is narrower than hardcoding Fireworks behavior in transport code; merge still needs real behavior proof.

Acceptance criteria:

  • Attach after-fix live Fireworks/Kimi or Mistral tool-dispatch output showing unsupported keywords stripped and the provider request succeeding.
  • Wait for the Real behavior proof check to pass or for a maintainer proof: override.
  • Run or rely on pnpm test src/agents/openai-tool-schema.test.ts src/agents/openai-transport-stream.test.ts, pnpm tsgo:core, and pnpm tsgo:core:test before merge.

What I checked:

  • Current main drops compat in OpenAI schema normalization: normalizeStrictOpenAIJsonSchema and normalizeOpenAIStrictToolParameters call normalizeToolParameterSchema without a modelCompat option, so unsupported keyword metadata cannot affect this path on current main. (src/agents/openai-tool-schema.ts:9, 5fae1c32b5f8)
  • Current main omits compat at both transport call sites: Both Responses and Chat Completions tool conversion pass only the schema and strict flag to normalizeOpenAIStrictToolParameters. (src/agents/openai-transport-stream.ts:402, 5fae1c32b5f8)
  • Existing lower-level normalizer already supports the compat seam: normalizeToolParameterSchema resolves unsupportedToolSchemaKeywords from options.modelCompat and strips those keywords when the set is non-empty. (src/agents/pi-tools-parameter-schema.ts:147, 5fae1c32b5f8)
  • Current main Fireworks Kimi row lacks the needed metadata: The bundled accounts/fireworks/routers/kimi-k2p5-turbo manifest entry has cost fields but no compat.unsupportedToolSchemaKeywords entry on current main. (extensions/fireworks/openclaw.plugin.json:46, 5fae1c32b5f8)
  • PR applies the expected plumbing: At PR head, normalizeOpenAIStrictToolParameters accepts options, forwards modelCompat, and both OpenAI transport call sites pass extractModelCompat(model). (src/agents/openai-transport-stream.ts:403, af5a8654fc4f)
  • PR adds the Fireworks metadata and focused unit coverage: At PR head, the Fireworks Kimi entry declares unsupportedToolSchemaKeywords: ["not"], and the new tests cover stripping not with compat while preserving it without compat. (extensions/fireworks/openclaw.plugin.json:57, af5a8654fc4f)

Likely related people:

  • steipete: Local history includes Peter Steinberger's earlier OpenAI tool schema compatibility work, and prior ClawSweeper context ties him to the shared OpenAI schema and transport helpers relevant to this patch. (role: shared OpenAI schema and transport maintainer; confidence: medium; commits: 435edaf99734, a9125ec0b07b, f52fdd855379; files: src/agents/openai-tool-schema.ts, src/agents/openai-transport-stream.ts, src/agents/pi-tools-parameter-schema.ts)
  • shakkernerd: Current shallow blame and prior ClawSweeper context point to Shakker/shakkernerd around the bundled Fireworks model catalog that contains the affected Kimi router row. (role: Fireworks catalog and adjacent provider maintainer; confidence: medium; commits: 3e53580d6311, 1aa62c0b0ad7, 00d2c3488932; files: extensions/fireworks/openclaw.plugin.json, extensions/fireworks/provider-catalog.ts)

Remaining risk / open question:

  • The external PR's proof remains test-only, and the repository Real behavior proof check is failing until redacted live provider output, logs, terminal output, or a linked artifact is added.
  • Normal merge readiness still depends on the remaining PR checks and maintainer review after the proof gate is satisfied.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 5fae1c32b5f8.

lonexreb added a commit to lonexreb/paloa-claw that referenced this pull request May 1, 2026
…-k2p5-turbo

Round-1 review on PR openclaw#75476 noted the new modelCompat plumbing alone
is incomplete because the Fireworks bundled catalog row for
kimi-k2p5-turbo did not declare unsupportedToolSchemaKeywords. Without
the manifest entry, the runtime model still has no unsupported-keyword
set and {"not": {}} reaches Fireworks unchanged.

Add the provider-owned manifest compat for the affected model so the
existing plumbing in this PR has the metadata to act on. Other
Fireworks routers do not need the same opt-out today (they accept
`not`); we can extend the list per-model as more provider rejections
surface.

Refs openclaw#75467, openclaw#75444
@lonexreb
Copy link
Copy Markdown
Contributor Author

lonexreb commented May 1, 2026

Round-1 review addressed in a381fef. The bot was right — the plumbing alone left the bundled accounts/fireworks/routers/kimi-k2p5-turbo row without the metadata to act on.

Added compat.unsupportedToolSchemaKeywords: ["not"] to that router entry in extensions/fireworks/openclaw.plugin.json. Other Fireworks routers do not need the same opt-out today (they accept not); the keyword list is per-model and we can extend as more provider rejections surface (e.g. #75444 reports the same shape).

End-to-end now: catalog declares unsupportedToolSchemaKeywordsextractModelCompat(model) reads it → normalizeOpenAIStrictToolParameters strips matching keywords from the tool schema before the request reaches Fireworks. PTAL.

@lonexreb lonexreb force-pushed the fix/75467-tool-schema-keywords-compat branch from a381fef to b7f815c Compare May 3, 2026 07:54
lonexreb added a commit to lonexreb/paloa-claw that referenced this pull request May 3, 2026
…-k2p5-turbo

Round-1 review on PR openclaw#75476 noted the new modelCompat plumbing alone
is incomplete because the Fireworks bundled catalog row for
kimi-k2p5-turbo did not declare unsupportedToolSchemaKeywords. Without
the manifest entry, the runtime model still has no unsupported-keyword
set and {"not": {}} reaches Fireworks unchanged.

Add the provider-owned manifest compat for the affected model so the
existing plumbing in this PR has the metadata to act on. Other
Fireworks routers do not need the same opt-out today (they accept
`not`); we can extend the list per-model as more provider rejections
surface.

Refs openclaw#75467, openclaw#75444
lonexreb added a commit to lonexreb/paloa-claw that referenced this pull request May 4, 2026
…-k2p5-turbo

Round-1 review on PR openclaw#75476 noted the new modelCompat plumbing alone
is incomplete because the Fireworks bundled catalog row for
kimi-k2p5-turbo did not declare unsupportedToolSchemaKeywords. Without
the manifest entry, the runtime model still has no unsupported-keyword
set and {"not": {}} reaches Fireworks unchanged.

Add the provider-owned manifest compat for the affected model so the
existing plumbing in this PR has the metadata to act on. Other
Fireworks routers do not need the same opt-out today (they accept
`not`); we can extend the list per-model as more provider rejections
surface.

Refs openclaw#75467, openclaw#75444
@lonexreb lonexreb force-pushed the fix/75467-tool-schema-keywords-compat branch from b7f815c to 048ad23 Compare May 4, 2026 09:46
lonexreb added a commit to lonexreb/paloa-claw that referenced this pull request May 5, 2026
…-k2p5-turbo

Round-1 review on PR openclaw#75476 noted the new modelCompat plumbing alone
is incomplete because the Fireworks bundled catalog row for
kimi-k2p5-turbo did not declare unsupportedToolSchemaKeywords. Without
the manifest entry, the runtime model still has no unsupported-keyword
set and {"not": {}} reaches Fireworks unchanged.

Add the provider-owned manifest compat for the affected model so the
existing plumbing in this PR has the metadata to act on. Other
Fireworks routers do not need the same opt-out today (they accept
`not`); we can extend the list per-model as more provider rejections
surface.

Refs openclaw#75467, openclaw#75444
@lonexreb lonexreb force-pushed the fix/75467-tool-schema-keywords-compat branch from 048ad23 to 9b03b62 Compare May 5, 2026 05:31
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 5, 2026
lonexreb added 2 commits May 5, 2026 13:11
…nAI-completions tool schemas

The catalog already records compat.unsupportedToolSchemaKeywords on
OpenAI-completions providers (e.g. fireworks/kimi-k2p5-turbo lists
`not`), and `normalizeToolParameterSchema` already knows how to
strip those keywords when given a modelCompat option. But the
intermediary `normalizeOpenAIStrictToolParameters` /
`normalizeStrictOpenAIJsonSchema` chain in src/agents/openai-tool-schema.ts
never accepted or forwarded model compat, and the call sites in
src/agents/openai-transport-stream.ts (`convertResponsesTools`,
`convertTools`) never passed it.

Result: `{"not": {}}` (the JSON Schema shape Zod `z.never()` compiles
to) reached Fireworks unchanged and the provider returned
`400 JSON Schema not supported: could not understand the instance {'not': {}}`,
breaking tool dispatch entirely on those models even though the catalog
correctly described the compat (openclaw#75467).

Thread modelCompat through the normalization chain:

- Add NormalizeOpenAIToolSchemaOptions to openai-tool-schema.ts and
  forward .modelCompat to normalizeToolParameterSchema in both
  normalizeStrictOpenAIJsonSchema and normalizeOpenAIStrictToolParameters
  (and isStrictOpenAIJsonSchemaCompatible for symmetry).
- In openai-transport-stream.ts, pass extractModelCompat(model) at the
  two call sites in convertResponsesTools and convertTools so the
  Fireworks-style compat reaches the schema normalizer.

Add 3 regression tests in openai-tool-schema.test.ts:

- non-strict params: unsupported "not" keyword is stripped when
  modelCompat opts in
- strict-mode params: unsupported "not" keyword is stripped while
  unrelated properties ("keep": {type:"string"}) are preserved
- absent compat: "not" stays in the schema (no behavior regression
  for providers that accept it)

CHANGELOG entry added per repo policy.

Refs openclaw#75467, openclaw#75444
…-k2p5-turbo

Round-1 review on PR openclaw#75476 noted the new modelCompat plumbing alone
is incomplete because the Fireworks bundled catalog row for
kimi-k2p5-turbo did not declare unsupportedToolSchemaKeywords. Without
the manifest entry, the runtime model still has no unsupported-keyword
set and {"not": {}} reaches Fireworks unchanged.

Add the provider-owned manifest compat for the affected model so the
existing plumbing in this PR has the metadata to act on. Other
Fireworks routers do not need the same opt-out today (they accept
`not`); we can extend the list per-model as more provider rejections
surface.

Refs openclaw#75467, openclaw#75444
@lonexreb lonexreb force-pushed the fix/75467-tool-schema-keywords-compat branch from 9b03b62 to af5a865 Compare May 5, 2026 18:11
@lonexreb
Copy link
Copy Markdown
Contributor Author

lonexreb commented May 5, 2026

Follow-up notes for reviewers — algorithm formalization and idempotence

Adding a tighter description of what the threaded modelCompat actually does at the schema layer, since reviewers often ask "what's the contract this preserves?" for compat patches.

Stripping algorithm (formal)

Let K be the catalog-declared compat.unsupportedToolSchemaKeywords set (e.g. {"not"} for Fireworks, {"maxLength", "minLength"} for Mistral-family completions servers). Let J be a tool's parameter JSON Schema. The normalizeToolParameterSchema pass that this PR finally feeds with K performs:

strip(J, K) =  J  with every node n where keys(n) ∩ K ≠ ∅
              recursively rewritten as { ...n, k: undefined for k in K }

Two properties hold for any provider response semantics that would have rejected J purely on K:

  1. Soundness for the rejected-keyword classstrip(J, K) is accepted by any provider whose rejection set is exactly K (per the catalog's contract).
  2. Idempotencestrip(strip(J, K), K) = strip(J, K). Important because the call sites in convertResponsesTools and convertTools both feed extractModelCompat(model) — repeated application across retries doesn't drift the schema.

The 3 new regression tests pin both properties (plus the negative case where K = ∅ and the schema is preserved verbatim, which proves we don't regress providers that accept the full keyword set).

Why this is the right boundary for the fix

The change adds an optional parameter to three functions in openai-tool-schema.ts and forwards from two call sites in openai-transport-stream.ts. The modelCompat data already lived on the catalog and normalizeToolParameterSchema already knew how to consume it — this PR just plumbs it through. Concretely the boundary is:

catalog → extractModelCompat(model) → convertTools/convertResponsesTools → normalizeOpenAIStrictToolParameters → normalizeToolParameterSchema(strict, modelCompat) → strip(K)

The arrow that was missing was the third one. No new API surface for plugin authors, no schema/config changes, no runtime cost on providers that don't declare the compat (the strip becomes a no-op when K = ∅).

Follow-up surface worth a separate PR

Two related places where catalog compat.unsupportedToolSchemaKeywords is also not consumed today (out of scope for this PR, would muddy the diff):

  1. Anthropic adapter tool conversion — Anthropic's input-schema converter in extensions/anthropic/src/.../tools.ts doesn't currently honor catalog-declared compat keywords either. Most providers that proxy through Anthropic-style adapters today don't need it, but a third-party provider plugin that targets the Anthropic family with strict schema constraints would hit the same class of bug. Same fix shape would apply.
  2. Schema validation tests — there's no current contract test that asserts extractModelCompat(model) is reachable from every tool dispatch path. Adding one would prevent the "missing arrow" regression class entirely. Probably worth adding under src/agents/openai-tool-schema.contract.test.ts.

Happy to file either as a follow-up if reviewers want this PR to stay surgical.

Cache-friendliness note

The strip(J, K) operation is content-addressed by (J, K), so a follow-up that memoizes the stripped schema by (toolDef.fingerprint, model.compat.fingerprint) would amortize the per-call cost across the streaming session — useful when tool schemas are large and the tool-definitions list is repeated per-turn. Not blocking; just noting since the reporter's measurement (11–22s attempt-dispatch latency) would benefit even after the 400 → 200 fix is in.

Copy link
Copy Markdown
Contributor

@martingarramon martingarramon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the seam — threading ModelCompatConfig through a single normalizer entry-point is the right shape, and the round-1 plumbing-to-data fix on accounts/fireworks/routers/kimi-k2p5-turbo (a381fef) closes the gap clawsweeper flagged. Two questions + one nit:

Q1 — third call site asymmetry. normalizeOpenAIStrictToolParameters has three callers in src/agents/. This PR updates two (openai-transport-stream.ts:399 Responses + :1767 Chat Completions) but not openai-ws-message-conversion.ts:317 (Realtime/WS). model is already in scope at both upstream call sites (openai-ws-stream.ts:844 and :923, typed ProviderRuntimeModel); extending convertTools's options to thread it would be the same one-line shape as the other two. I don't know whether Fireworks Kimi is wired through the Realtime path today, but the asymmetry leaves the contract incomplete for any future model that lands there.

Q2 — coverage forward-look. Tests cover ["not"] at one nesting level. Worth pinning ["not", "oneOf"]-style multi-keyword and array.items / deeply-nested-property variants now while the helper is fresh, or wait for the next provider to surface them?

Nit. isStrictOpenAIJsonSchemaCompatible(schema, options?) now answers compatibility post-strip. A one-line addition to NormalizeOpenAIToolSchemaOptions's JSDoc clarifying that semantic would help future callers.

#75476

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unsupportedToolSchemaKeywords not applied for OpenAI-completions providers (Fireworks)

2 participants