Repository navigation
RES-904: add proactive reward-hacking self-check to default prompts - #30
Merged
madeleinesong merged 1 commit intoMay 15, 2026
Conversation
Add a five-item integrity gate paragraph to anthropic, default, and beast system prompts so finalized responses restate literal vs. intended objective, name cheap wins and loopholes, run adversarial self-review, disclose uncertainty, and refuse rather than producing a confidently-worded response the agent cannot honestly back. The scientific-writing block adds rules on speculative pathways, citation confidence, and evidence-kind verbs. Mirrors the canonical spec in get-physics-done (references/shared/reward-hacking-self-check.md) for no-project sessions where the desktop runtime is the only source of the gate. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
📝 WalkthroughWalkthroughThree session prompt files receive parallel additions of a required pre-finalization "Reward-hacking self-check" section. Each prompt now instructs the assistant to validate literal-vs-spirit compliance, identify loopholes and cheap wins, conduct adversarial review, disclose uncertainty, and apply scientific writing constraints before claiming completion. ChangesPre-finalization validation across session prompts
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/opencode/src/session/prompt/default.txt (1)
79-88: ⚡ Quick winAdd an automated prompt-regression guard for this new gate.
Given this block is policy-critical, add a test/snapshot that asserts the section exists (and optionally stays within a token/length budget) to avoid silent truncation/removal in future edits.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/opencode/src/session/prompt/default.txt` around lines 79 - 88, Add an automated test that asserts the "Reward-hacking self-check (required before finalizing)" block is present and unchanged: create a test (e.g., test_reward_hacking_self_check_present) that loads the default prompt text, searches for the exact header string "Reward-hacking self-check (required before finalizing)" and key marker lines (like the five numbered items 1.–5.), and fails if any are missing; optionally include a second assertion that the block length/token count stays under a defined budget to catch accidental expansion or truncation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/opencode/src/session/prompt/default.txt`:
- Around line 79-88: Add an automated test that asserts the "Reward-hacking
self-check (required before finalizing)" block is present and unchanged: create
a test (e.g., test_reward_hacking_self_check_present) that loads the default
prompt text, searches for the exact header string "Reward-hacking self-check
(required before finalizing)" and key marker lines (like the five numbered items
1.–5.), and fails if any are missing; optionally include a second assertion that
the block length/token count stays under a defined budget to catch accidental
expansion or truncation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: cd8b73c5-dad6-4e66-a206-0c9e47801256
📒 Files selected for processing (3)
packages/opencode/src/session/prompt/anthropic.txtpackages/opencode/src/session/prompt/beast.txtpackages/opencode/src/session/prompt/default.txt
cmaloney111
added a commit
that referenced
this pull request
May 15, 2026
PR #30 (RES-904) added the five-item reward-hacking integrity gate to anthropic.txt, default.txt, and beast.txt only — leaving gpt-5.x, gemini-3.x, codex, kimi, and trinity sessions able to bypass the gate by switching the model picker. Close that gap by adding the gate to all five remaining provider prompts in `SystemPrompt.provider()`: - gpt.txt — concise collaborator voice - gemini.txt — bold-prefixed Core-Mandates style - codex.txt — terse, CLI-tooling voice - kimi.txt — general-coding voice, no scientific addendum - trinity.txt — compressed five-item variant to match the brevity-first tone of the prompt Each variant carries the same five items (literal-vs-spirit, cheap wins, adversarial self-review, uncertainty disclosure, revise-or- refuse). gpt / gemini / codex also carry the scientific-writing addendum on speculative-pathway / citation-fabrication / evidence- verb discipline. kimi and trinity skip the addendum because those prompts target general software work, not scientific writing. Sanity checks done: - bun run typecheck — pass - bun test test/session/system — 1/1 pass - bun run build --single — sidecar rebuilds, smoke ok - `strings opencode-cli` on the rebuilt sidecar — eight matches each for "Reward-hacking self-check", "Literal vs. spirit", "Cheap wins" (one per provider prompt baked in)
cmaloney111
added a commit
that referenced
this pull request
May 15, 2026
Reverting in favor of upstream opencode#27163 ("feat: add native
session goals", opened 2026-05-12, still open). Upstream ships:
- Token + wall-clock usage accounting with budget-exhausted state
- Model-facing read/create/mark-complete tools (a real forcing
function — model can't ignore a passive system-prompt block the
way it could here)
- Auto-continue-on-idle + pause-on-abort
- TUI prompt-input / session-header / footer surfaces
- Real SDK codegen output (vs the hand-patched sdk.gen.ts /
types.gen.ts in this PR that next regen would clobber)
- 1243+ lines of test coverage vs ~120 here
PR #31's own "Followups (deferred)" section explicitly listed three
items upstream already provides: SDK regen, footer/status-bar
surface, real budget/time tracking. The implementation here was
display-only prompt injection with no enforcement.
Plan: wait for upstream anomalyco#27163 to merge, then sync via the normal
upstream-merge path. The gpd-cli companion (get-physics-done#228)
is a distinct surface (CLI, not in-runtime) and can land on its
own merits once it consumes the upstream-merged shape.
Cameron's f6a9017 on top (extending the RES-904 reward-hacking
gate to gpt/gemini/codex/kimi/trinity) touches a disjoint file set
and is preserved unchanged.
Reverts the 12 files PR #31 modified/added to their state at the
PR #30 merge (fb230ed):
packages/opencode/migration/20260515120000_add_session_goal/migration.sql (deleted)
packages/opencode/src/cli/cmd/tui/component/dialog-session-goal.tsx (deleted)
packages/opencode/src/cli/cmd/tui/routes/session/index.tsx (restored)
packages/opencode/src/server/instance/session.ts (restored)
packages/opencode/src/session/index.ts (restored)
packages/opencode/src/session/projectors.ts (restored)
packages/opencode/src/session/prompt.ts (restored)
packages/opencode/src/session/session.sql.ts (restored)
packages/opencode/test/cli/tui/dialog-session-goal.test.ts (deleted)
packages/opencode/test/session/goal.test.ts (deleted)
packages/sdk/js/src/v2/gen/sdk.gen.ts (restored)
packages/sdk/js/src/v2/gen/types.gen.ts (restored)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a five-item reward-hacking integrity gate paragraph to the three baseline system prompts (
anthropic.txt,default.txt,beast.txt). The gate restates literal-vs-intended objective, names cheap wins and loopholes, runs adversarial self-review, discloses uncertainty, and refuses rather than producing a confidently-worded response the agent cannot honestly back. The scientific-writing block adds rules on speculative pathways ("could in principle" is not "has been shown to"), citation confidence (drop rather than fabricate when in doubt), and evidence-kind verbs (no blurring of analytic / numerical / experimental with "demonstrated" or "established").This mirrors the canonical spec landing in get-physics-done at
src/gpd/specs/references/shared/reward-hacking-self-check.mdso that no-project sessions in the desktop runtime — where the GPD spec bundle does not load — still get the gate. Project sessions get both: the canonical reference from get-physics-done plus this shorter paragraph.The
beast.txtvariant calls out that autonomy makes the literal-vs-spirit trap larger, not smaller, because that prompt is used for the most autonomous mode where the failure incident occurred.Test plan
bun run typecheck— passesbun test test/session— 259 pass / 11 fail (matches origin/gpd baseline of abort/cancel timeouts; none introduced)anthropic.txt-routed Claude sessionsLinked: RES-904. Companion PR in get-physics-done adds the canonical spec, agent wiring, and the
reward_hacking_integrity_gateworkflow step.Co-Authored-By: Claude Opus 4.7 (1M context) [email protected]
Summary by CodeRabbit
Release Notes