Skip to content

fix(cli): require verification for no-change patches - #1020

Merged
mldangelo-oai merged 5 commits into
openai:mainfrom
arunimshukla:fix/no-change-verification
Sep 27, 2026
Merged

mldangelo-oai merged 5 commits into
openai:mainfrom
arunimshukla:fix/no-change-verification

Conversation

@arunimshukla

@arunimshukla arunimshukla commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

scan --patch could return success after the model reported no_change without explaining how it verified the finding. That result now becomes failed with exit code 2, matching the existing check for verified results.

Fixes #1019. Original fix by @arunimshukla.

Changes

  • Apply the existing nonblank verification check to both successful statuses.
  • Ask for verification in the model's response format for both statuses.
  • Extend the existing tests for missing and whitespace-only verification, and preserve valid no_change, blocked, and failed behavior.

Testing

  • Focused patch tests: 72 passed with seed 12345 and the existing 30-second timeout.
  • Controlled before/after comparison: the same tests fail twice on current main because missing and blank no_change verification return success. All eight selected cases pass with this fix. These tests supply model responses explicitly.
  • Built CLI scan with a real model on a benign local fixture: completed successfully, with complete coverage, no findings, and no source changes. This checks scan compatibility; it did not exercise the missing-verification case.
  • Plugin build, CI-script build, SDK build, type checks, and formatting passed.

Risk and rollout

This checks whether a verification explanation is present. It cannot establish whether that explanation is true. Valid no_change results remain accepted; missing or blank verification now fails. No CLI options or output fields change.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the bug Something isn't working label Sep 24, 2026
@arunimshukla
arunimshukla marked this pull request as ready for review September 24, 2026 03:54
@arunimshukla arunimshukla changed the title fix(cli): require verification for no-change patches - State in the patch result contract that verification is required for verified and no_change outcomes, so the requested result format matches the enforced check. Sep 24, 2026
@arunimshukla arunimshukla changed the title - State in the patch result contract that verification is required for verified and no_change outcomes, so the requested result format matches the enforced check. State in the patch result contract that verification is required for verified and no_change outcomes, so the requested result format matches the enforced check. Sep 24, 2026
@arunimshukla arunimshukla changed the title State in the patch result contract that verification is required for verified and no_change outcomes, so the requested result format matches the enforced check. fix(cli): require verification for no-change patches Sep 24, 2026
@mldangelo-oai

Copy link
Copy Markdown
Collaborator

Thanks, Arunim. I merged current main and kept the production fix to the existing verification check and its response instructions.

The before/after comparison confirms the bug: main returns success for no_change with missing or blank verification; this branch returns exit 2. Valid no_change results still work without creating a pull request.

Validation is complete: 72 focused patch tests passed, and both full SDK runs passed with 3,242 tests passing, 50 skipped, and no failures in each run. Builds, type checks, formatting, and GitHub CI passed. The review found no further changes needed.

I also ran the built CLI with a real model on a benign local fixture. It completed with full coverage, no findings, and unchanged source files. That scan did not produce the bad patch response; the controlled regression tests establish that part of the fix.

@mldangelo-oai mldangelo-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@mldangelo-oai
mldangelo-oai merged commit 9abe68c into openai:main Sep 27, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

scan --patch accepts no_change without verification evidence

2 participants