Skip to content

feat(openai): add verification checklist to GPT-5 execution bias [v3 5/6]#66375

Closed
100yenadmin wants to merge 3 commits into
openclaw:mainfrom
electricsheephq:gpt5-v3/enhanced-verification-checklist
Closed

feat(openai): add verification checklist to GPT-5 execution bias [v3 5/6]#66375
100yenadmin wants to merge 3 commits into
openclaw:mainfrom
electricsheephq:gpt5-v3/enhanced-verification-checklist

Conversation

@100yenadmin

Copy link
Copy Markdown
Contributor

GPT 5.4 Enhancement v3 — PR 5/6

Tracking: #66345 | Issue: #66351
Priority: P2 — LOW | Type: Quality Polish

Problem

OpenClaw's GPT-5 execution bias says "Act first, then verify if needed" but doesn't specify what to verify. Hermes provides a structured checklist. The critical missing dimension is grounding — checking that claims are backed by tool output, not training data.

Changes

extensions/openai/prompt-overlay.ts — Append verification subsection to OPENAI_GPT5_EXECUTION_BIAS:

### Verification
Before finalizing your response, check:
- Correctness: does the output satisfy every stated requirement?
- Grounding: are factual claims backed by tool outputs, not training data?
- Coverage: did you address every part of the request?
- Formatting: does the output match the requested format or schema?
- Safety: if the next step has side effects, confirm scope before executing.

Verification Flow

  GPT-5 finishes tool calls
           │
           ▼
  ┌─────────────────────┐
  │ Verification Pass    │
  │                      │
  │ ✓ Correctness        │ ← Does output match requirements?
  │ ✓ Grounding          │ ← Claims backed by tool output? (KEY)
  │ ✓ Coverage           │ ← All parts addressed?
  │ ✓ Formatting         │ ← Right format/schema?
  │ ✓ Safety             │ ← Side effects scoped?
  └──────────┬───────────┘
             │
             ▼
  Finalize response

Hermes Reference

agent/prompt_builder.py lines 238-245 — <verification> block.

Impact

Modest quality improvement on multi-step tasks. The Grounding check specifically reinforces mandatory tool use (PR 1) — even if GPT-5 skips a tool call, the verification step prompts it to notice the gap.

Add a structured 5-point verification checklist to the GPT-5
execution bias overlay:
- Correctness: output satisfies requirements
- Grounding: claims backed by tool output, not training data
- Coverage: all parts of request addressed
- Formatting: matches requested format/schema
- Safety: side effects scoped before executing

The "Grounding" check is particularly important — it reinforces
mandatory tool use by prompting GPT-5 to notice when it answered
from memory instead of tool output.

Part of GPT 5.4 Enhancement v3 sprint. Closes #66351.
Tracking issue: #66345.
@greptile-apps

greptile-apps Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR appends a five-point ### Verification checklist to OPENAI_GPT5_EXECUTION_BIAS in extensions/openai/prompt-overlay.ts, adding checks for correctness, grounding, coverage, formatting, and safety before GPT-5 finalizes a response.

  • The Safety bullet ("confirm scope before executing" for side-effecting actions) directly contradicts the pre-existing "Do not pause to summarize or verify before taking the next action" instruction in the same constant. Most consequential tool calls (file writes, API calls, shell commands) have side effects, so this effectively reintroduces a pause-and-confirm signal for the exact actions the execution bias was designed to suppress — likely causing inconsistent model behavior.

Confidence Score: 3/5

  • Not safe to merge as-is; the Safety bullet contradicts the core execution bias directive.
  • There is a clear P1 conflict: the new Safety verification item instructs the model to pause and confirm scope before side-effecting tool calls, while the surrounding prompt text explicitly says not to pause before taking the next action. This internal contradiction within the same exported constant will produce unpredictable model behavior across all GPT-5 sessions that use this overlay.
  • extensions/openai/prompt-overlay.ts — the Safety bullet in the new Verification section
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/openai/prompt-overlay.ts
Line: 72-80

Comment:
**Safety bullet contradicts execution bias principle**

The existing instruction on line 72 explicitly says "Do not pause to summarize or verify before taking the next action," but the new `Safety` bullet says "confirm scope before executing" for side-effecting actions (file writes, commands, API calls). Nearly every consequential tool call has side effects, so this tells the model to pause and confirm before the exact actions the constant is designed to push through without interruption. The two directives are in direct conflict within the same exported string, and a model reading the full text gets contradictory signals about whether to act or ask first.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "feat(openai): add verification checklist..." | Re-trigger Greptile

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a structured verification checklist to the GPT-5 execution bias prompt overlay so GPT-5 does a final self-check (including explicit grounding against tool output) before responding.

Changes:

  • Extend OPENAI_GPT5_EXECUTION_BIAS with a new “### Verification” subsection.
  • Include explicit checks for correctness, grounding, coverage, formatting, and safety prior to finalizing the response.

Comment thread extensions/openai/prompt-overlay.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d23f1862f4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/openai/prompt-overlay.ts
Address review feedback: the safety bullet was too broad ("confirm
scope before executing" for all side effects). This contradicts the
execution bias principle of acting first.

Narrowed to: only confirm scope for destructive operations (deletions,
overwrites, production deploys). Routine reads and writes proceed
without confirmation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread extensions/openai/prompt-overlay.ts Outdated
Parity audit: previous iterations drifted in two ways from Hermes
(prompt_builder.py lines 238-245):

1. Added a "Coverage" bullet that Hermes does not have. REMOVED.
2. Narrowed "Safety: if the next step has side effects (file writes,
   commands, API calls), confirm scope before executing" to only
   destructive ops. RESTORED to Hermes text.

The previous narrowing was driven by a Greptile comment claiming the
broader rule contradicts the execution bias principle. Hermes ships
with both rules simultaneously and works — we must match exactly to
preserve the cross-comparison.
@100yenadmin

Copy link
Copy Markdown
Contributor Author

Superseded by consolidated final-sprint PRs. This PR's changes are incorporated into:

The v3 PRs were split across 6 separate PRs which made review harder and touched the same files redundantly. The final-sprint consolidation combines them into 3 clean, independently-mergeable PRs rebased on current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants