feat(openai): add verification checklist to GPT-5 execution bias [v3 5/6]#66375
feat(openai): add verification checklist to GPT-5 execution bias [v3 5/6]#66375100yenadmin wants to merge 3 commits into
Conversation
Add a structured 5-point verification checklist to the GPT-5 execution bias overlay: - Correctness: output satisfies requirements - Grounding: claims backed by tool output, not training data - Coverage: all parts of request addressed - Formatting: matches requested format/schema - Safety: side effects scoped before executing The "Grounding" check is particularly important — it reinforces mandatory tool use by prompting GPT-5 to notice when it answered from memory instead of tool output. Part of GPT 5.4 Enhancement v3 sprint. Closes #66351. Tracking issue: #66345.
Greptile SummaryThis PR appends a five-point
Confidence Score: 3/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: extensions/openai/prompt-overlay.ts
Line: 72-80
Comment:
**Safety bullet contradicts execution bias principle**
The existing instruction on line 72 explicitly says "Do not pause to summarize or verify before taking the next action," but the new `Safety` bullet says "confirm scope before executing" for side-effecting actions (file writes, commands, API calls). Nearly every consequential tool call has side effects, so this tells the model to pause and confirm before the exact actions the constant is designed to push through without interruption. The two directives are in direct conflict within the same exported string, and a model reading the full text gets contradictory signals about whether to act or ask first.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "feat(openai): add verification checklist..." | Re-trigger Greptile |
There was a problem hiding this comment.
Pull request overview
Adds a structured verification checklist to the GPT-5 execution bias prompt overlay so GPT-5 does a final self-check (including explicit grounding against tool output) before responding.
Changes:
- Extend
OPENAI_GPT5_EXECUTION_BIASwith a new “### Verification” subsection. - Include explicit checks for correctness, grounding, coverage, formatting, and safety prior to finalizing the response.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d23f1862f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address review feedback: the safety bullet was too broad ("confirm
scope before executing" for all side effects). This contradicts the
execution bias principle of acting first.
Narrowed to: only confirm scope for destructive operations (deletions,
overwrites, production deploys). Routine reads and writes proceed
without confirmation.
Parity audit: previous iterations drifted in two ways from Hermes (prompt_builder.py lines 238-245): 1. Added a "Coverage" bullet that Hermes does not have. REMOVED. 2. Narrowed "Safety: if the next step has side effects (file writes, commands, API calls), confirm scope before executing" to only destructive ops. RESTORED to Hermes text. The previous narrowing was driven by a Greptile comment claiming the broader rule contradicts the execution bias principle. Hermes ships with both rules simultaneously and works — we must match exactly to preserve the cross-comparison.
|
Superseded by consolidated final-sprint PRs. This PR's changes are incorporated into:
The v3 PRs were split across 6 separate PRs which made review harder and touched the same files redundantly. The final-sprint consolidation combines them into 3 clean, independently-mergeable PRs rebased on current |
GPT 5.4 Enhancement v3 — PR 5/6
Tracking: #66345 | Issue: #66351
Priority: P2 — LOW | Type: Quality Polish
Problem
OpenClaw's GPT-5 execution bias says "Act first, then verify if needed" but doesn't specify what to verify. Hermes provides a structured checklist. The critical missing dimension is grounding — checking that claims are backed by tool output, not training data.
Changes
extensions/openai/prompt-overlay.ts— Append verification subsection toOPENAI_GPT5_EXECUTION_BIAS:Verification Flow
Hermes Reference
agent/prompt_builder.pylines 238-245 —<verification>block.Impact
Modest quality improvement on multi-step tasks. The Grounding check specifically reinforces mandatory tool use (PR 1) — even if GPT-5 skips a tool call, the verification step prompts it to notice the gap.