feat(openai): add mandatory tool-use categories for GPT-5 models [v3 1/6]#66371
feat(openai): add mandatory tool-use categories for GPT-5 models [v3 1/6]#66371100yenadmin wants to merge 7 commits into
Conversation
GPT 5.4 on OpenClaw answers factual, computational, and system-state questions from training data instead of calling tools. This produces hallucinated or stale answers and breaks the agentic flow. Hermes Agent solves this with an explicit mandatory tool-use list. This PR ports that pattern: 8 categories where GPT-5 must always use a tool (arithmetic, hashing, timestamps, system state, file contents, git, current facts, network checks). Placed in stablePrefix for now; will move to sectionOverrides once tool_enforcement becomes a first-class section ID. Part of GPT 5.4 Enhancement v3 sprint. Closes #66346. Tracking issue: #66345.
Greptile SummaryAdds Confidence Score: 5/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: extensions/openai/prompt-overlay.ts
Line: 132-139
Comment:
**Stale comment doesn't reflect `OPENAI_GPT5_TOOL_ENFORCEMENT` placement**
The block comment explains why `OPENAI_GPT5_TOOL_CALL_STYLE` lives in `stablePrefix`, but `OPENAI_GPT5_TOOL_ENFORCEMENT` is now also in `stablePrefix` for a different reason (temporary staging until PR 3 introduces `sectionOverrides.tool_enforcement`). A reader landing here has no indication that the third entry has a different rationale and will eventually move.
```suggestion
// tool_call_style is NOT overridden via sectionOverrides because the
// default section includes dynamic channel-specific approval guidance
// from buildExecApprovalPromptGuidance() that varies per runtime
// channel. Overriding it with a static string would lose that dynamic
// content. Instead, the tool-first reinforcement lives in stablePrefix
// so it's always present alongside the default tool_call_style section.
// tool_enforcement is also in stablePrefix temporarily; PR 3 (#66348)
// will promote it to a first-class sectionOverrides.tool_enforcement key.
return {
stablePrefix: [OPENAI_GPT5_OUTPUT_CONTRACT, OPENAI_GPT5_TOOL_CALL_STYLE, OPENAI_GPT5_TOOL_ENFORCEMENT].join("\n\n"),
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (2): Last reviewed commit: "fix(openai): handle minimal tool profile..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds a GPT‑5-specific “mandatory tool use” block to the OpenAI provider’s system prompt overlay to reduce GPT‑5 models answering time/math/system-state questions from training data instead of using tools (Hermes parity gap #1).
Changes:
- Introduces
OPENAI_GPT5_TOOL_ENFORCEMENTwith mandatory tool-use categories. - Appends the new enforcement block into
stablePrefixwithinresolveOpenAISystemPromptContribution.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88ad3f2ea0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address review feedback: - Replace generic "terminal" with canonical "exec" - Replace "read or search tools" with "read, search_files" - Replace "web search" with "web_search if available" - Replace "code execution" with "code_execution" - Add session_status as alternative for time/date - Replace em dashes with periods (per GPT-5 output contract)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16b8e7152b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
All 7 test assertions now include OPENAI_GPT5_TOOL_ENFORCEMENT in the stablePrefix array, matching the runtime behavior.
Address two Copilot review comments on PR #66371: 1. tools.profile="minimal" only has session_status registered, so instructing GPT-5 to "ALWAYS use exec/web_search" can produce invalid tool calls. Updated OPENAI_GPT5_TOOL_ENFORCEMENT to: - Say "use an available tool" (conditional) - Prefer session_status for time/system state when available - Explicitly instruct the model to state the limitation and ask to enable the needed tools/profile if the preferred tool is unavailable, rather than guessing. 2. stablePrefix assertion is repeated in 7 places in the test file, making it brittle to future overlay tweaks. Factored into a EXPECTED_GPT5_STABLE_PREFIX helper constant at the top of the test file.
|
@copilot @greptile-apps Re-review please — addressed both comments in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 07f4ebb012
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ofiles Codex P1 review: the previous text unconditionally said "NEVER answer from memory" and "state the limitation and ask to enable tools" even for trivial prompts like "2+2". In minimal/messaging profiles where exec/code_execution are not registered, this created a functional regression — GPT-5 would refuse to answer instead of just computing. Fix: carve out an explicit exception for trivial deterministic questions when no tool is available: - Trivial arithmetic, well-known facts → answer directly, prefix with "from memory" so the user knows it is not grounded - Stale-prone categories (time, system state, files, git, live facts) → still say we cannot verify without the tool When a tool IS available, the rule remains strong: use it.
|
@copilot Re-review please — |
Parity audit: previous iterations drifted significantly from the Hermes reference text based on low-confidence bot review suggestions. The entire point of the GPT 5.4 Enhancement v3 sprint is to replicate Hermes's success on GPT-5, so the prompt must mirror Hermes as closely as possible to preserve the cross-comparison. Reverts the following drift: - "NEVER answer from memory" -> softened conditional phrasing (REVERT) - Added "When No Tool Is Available" carve-out (REVERT) - Added "Network checks" category (REVERT) - Removed "Your memory and user profile describe the USER..." (REVERT) - Replaced arrows -> with colons (REVERT) Keeps these minimal changes vs Hermes, justified by real OpenClaw runtime differences: - terminal -> exec (no `terminal` tool in OpenClaw) - execute_code -> code_execution (canonical OpenClaw ID) - read_file -> read (canonical OpenClaw ID) - search_files -> exec (no first-class file-search tool) These substitutions are documented in a comment above the constant so future reviewers understand what is load-bearing (the semantics) vs what is a mechanical tool-ID mapping. The concerns raised by bot reviewers about minimal/messaging profiles (2+2 becoming a forced refusal) are noted but are the EXACT same behavior Hermes exhibits with its unconditional rule. If Hermes ships with that rule and works, OpenClaw must ship with it too to preserve the parity benchmark.
| NEVER answer these from memory or mental computation — ALWAYS use a tool: | ||
| - Arithmetic, math, calculations → use exec or code_execution | ||
| - Hashes, encodings, checksums → use exec (e.g. sha256sum, base64) | ||
| - Current time, date, timezone → use exec (e.g. date) | ||
| - System state: OS, CPU, memory, disk, ports, processes → use exec | ||
| - File contents, sizes, line counts → use read or exec | ||
| - Git history, branches, diffs → use exec | ||
| - Current facts (weather, news, versions) → use web_search | ||
|
|
There was a problem hiding this comment.
The PR description says GPT-5 gets 8 mandatory tool-use categories (including “Network checks”), but the new OPENAI_GPT5_TOOL_ENFORCEMENT list currently has 7 bullets and is missing the “Network checks (port open, DNS, connectivity)” category. Either add the missing category to the prompt block or update the PR description/issue references so they match exactly.
| - File contents, sizes, line counts → use read or exec | ||
| - Git history, branches, diffs → use exec | ||
| - Current facts (weather, news, versions) → use web_search | ||
|
|
There was a problem hiding this comment.
This block says “NEVER … ALWAYS use a tool” and then references exec/code_execution/read/web_search as mandatory tools. In OpenClaw, tools.profile="minimal" only includes session_status by default (see src/agents/tool-catalog.ts), so GPT‑5 can be instructed to call tools that are not available in the current runtime, leading to repeated invalid tool calls instead of a graceful limitation message. Please add an explicit fallback in the enforcement text for when the preferred tool is unavailable (e.g., don’t guess; state that the tool isn’t enabled and ask to enable an appropriate tools.profile/allowlist).
| If the needed tool is unavailable or not enabled in this runtime, do not guess and do not pretend to have used it. State that the required tool is not enabled, and ask the user to enable an appropriate tools.profile or tool allowlist for the needed capability. |
There was a problem hiding this comment.
@steipete @pashpashpash working on this and trying to figure out how to handle this. Code review agents aren't qualified for prompt engineering however, it is NOT wrong about tools.profile=minimal here (GPT 5.4 will potentially loop or simply try to guess and fail > preferred method is to state tool not enabled and ask to enable on tool list.
However, GPT 5.4 is "lazy" here and if prompt is changed for this, it will affect negatively the performance of all other users. It is recommended to allow this to simply error out or handle as separate PR that changes prompting for "minimal" tools profiles. My gut check is very few users on OC actually use the profile and it is better that 99% of users have a better experience in this small update versus 1% that need a "workaround".
| // Ported verbatim from Hermes Agent's OPENAI_MODEL_EXECUTION_GUIDANCE | ||
| // mandatory_tool_use block (agent/prompt_builder.py lines 207-218) with | ||
| // only tool ID substitutions for OpenClaw-canonical names: | ||
| // terminal -> exec (no `terminal` tool in OpenClaw) | ||
| // execute_code -> code_execution (canonical OpenClaw ID) | ||
| // read_file -> read (canonical OpenClaw ID) | ||
| // search_files -> exec (no first-class file-search tool; | ||
| // use shell grep via exec) |
There was a problem hiding this comment.
The header comment claims there is “no first-class file-search tool” and maps search_files -> exec, but OpenClaw exposes first-class grep/find tools (e.g. src/agents/system-prompt.ts tool list). Please update this comment (and consider pointing the enforcement guidance at grep/find instead of shelling out via exec) so the overlay documentation matches actual tool availability.
|
Superseded by consolidated final-sprint PRs. This PR's changes are incorporated into:
The v3 PRs were split across 6 separate PRs which made review harder and touched the same files redundantly. The final-sprint consolidation combines them into 3 clean, independently-mergeable PRs rebased on current |
GPT 5.4 Enhancement v3 — PR 1/6
Tracking: #66345 | Issue: #66346
Priority: P0 — CRITICAL | Gap closure: ~35%
Problem
GPT 5.4 on OpenClaw answers factual, computational, and system-state questions from training data instead of calling tools. This is the #1 root cause of the behavioral gap vs Hermes Agent.
Changes
extensions/openai/prompt-overlay.ts:OPENAI_GPT5_TOOL_ENFORCEMENTconstant with 8 mandatory tool-use categories:stablePrefixinresolveOpenAISystemPromptContributionHermes Reference
agent/prompt_builder.pylines 207-218 —<mandatory_tool_use>block withinOPENAI_MODEL_EXECUTION_GUIDANCE.Verification
Test GPT 5.4 — all should call tools, not answer from memory:
dateecho -n hello | sha256sumdf -hgit branch --show-currentNote
Currently placed in
stablePrefix. PR 3 (#66348) will promotetool_enforcementto a first-classsectionOverrideskey, at which point this moves fromstablePrefixtosectionOverrides.tool_enforcement.