Skip to content

[GPT 5.4 v3 — PR 1/6] Add mandatory tool-use categories for GPT-5 models #66346

Description

@100yenadmin

Parent: #66345 — GPT 5.4 Enhancement v3: Hermes Parity Sprint

Priority: P0 — CRITICAL
Estimated gap closure: ~35%

Problem

GPT 5.4 on OpenClaw answers factual, computational, and system-state questions from training data instead of calling tools. This produces hallucinated or stale answers and breaks the agentic flow — the model "thinks" it's done when it hasn't actually verified anything.

Hermes Agent solves this with an explicit mandatory tool-use list that GPT-5 receives in its system prompt. OpenClaw has no equivalent.

               User: "What time is it?"
                        │
          ┌─────────────┴──────────────┐
          │                            │
    ┌─────▼──────┐              ┌──────▼──────┐
    │   Hermes    │              │  OpenClaw    │
    │             │              │              │
    │ Has:        │              │ Missing:     │
    │ "NEVER      │              │ No mandatory │
    │  answer     │              │ tool-use     │
    │  from       │              │ categories   │
    │  memory"    │              │              │
    │ + 7 categ.  │              │              │
    └─────┬───────┘              └──────┬───────┘
          │                             │
    ┌─────▼──────┐              ┌───────▼──────┐
    │ Calls       │              │ Answers from  │
    │ `date` tool │              │ training data │
    │ → 2026-04   │              │ → maybe wrong │
    │ ✅           │              │ ❌             │
    └─────────────┘              └──────────────┘

Hermes Reference

agent/prompt_builder.py lines 207-218:

OPENAI_MODEL_EXECUTION_GUIDANCE = (
    "<mandatory_tool_use>\n"
    "NEVER answer these from memory or mental computation — ALWAYS use a tool:\n"
    "- Arithmetic, math, calculations → use terminal or execute_code\n"
    "- Hashes, encodings, checksums → use terminal (e.g. sha256sum, base64)\n"
    "- Current time, date, timezone → use terminal (e.g. date)\n"
    "- System state: OS, CPU, memory, disk, ports, processes → use terminal\n"
    "- File contents, sizes, line counts → use read_file, search_files, or terminal\n"
    "- Git history, branches, diffs → use terminal\n"
    "- Current facts (weather, news, versions) → use web_search\n"
    "</mandatory_tool_use>"
)

Proposed Implementation

File: extensions/openai/prompt-overlay.ts

Add new constant:

export const OPENAI_GPT5_TOOL_ENFORCEMENT = `## Mandatory Tool Use

NEVER answer these from memory or mental computation — ALWAYS use a tool:
- Arithmetic, math, calculations → use terminal or code execution
- Hashes, encodings, checksums → use terminal (e.g. sha256sum, base64)
- Current time, date, timezone → use terminal (e.g. date)
- System state: OS, CPU, memory, disk, ports, processes → use terminal
- File contents, sizes, line counts → use read or search tools, or terminal
- Git history, branches, diffs, status → use terminal
- Current facts (weather, news, package versions) → use web search
- Network checks (port open, DNS, connectivity) → use terminal

Your training data is stale. The execution environment may differ from what you expect. Always ground answers in live tool output.`;

Wire into resolveOpenAISystemPromptContribution:

return {
  stablePrefix: [OPENAI_GPT5_OUTPUT_CONTRACT, OPENAI_GPT5_TOOL_CALL_STYLE].join("\n\n"),
  sectionOverrides: {
    execution_bias: OPENAI_GPT5_EXECUTION_BIAS,
    tool_enforcement: OPENAI_GPT5_TOOL_ENFORCEMENT,  // ← NEW
    ...(params.mode === "friendly" ? { interaction_style: OPENAI_FRIENDLY_PROMPT_OVERLAY } : {}),
  },
};

Note: This uses sectionOverrides.tool_enforcement which requires PR 3 to land first. If shipping independently, place in stablePrefix instead.

File: extensions/openai/index.test.ts

Add test verifying GPT-5 models receive tool enforcement, non-GPT-5 models do not.

Verification

Test GPT 5.4 with these prompts (all should call tools, not answer from memory):

  • "What time is it?" → calls terminal date
  • "What's the sha256 of 'hello'?" → calls terminal echo -n hello | sha256sum
  • "How much free disk space?" → calls terminal df -h
  • "What branch am I on?" → calls terminal git branch --show-current
  • "What's 2^64?" → calls terminal or code execution

Impact

This is the single highest-ROI change. It closes ~35% of the behavioral gap between OpenClaw and Hermes for GPT 5.4 by eliminating the most common failure mode: answering from stale training data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions