Skip to content

test: cover versioned inline eval approval [AI]#97833

Closed
pgondhi987 wants to merge 1 commit into
openclaw:mainfrom
pgondhi987:fix/fix-790
Closed

test: cover versioned inline eval approval [AI]#97833
pgondhi987 wants to merge 1 commit into
openclaw:mainfrom
pgondhi987:fix/fix-790

Conversation

@pgondhi987

@pgondhi987 pgondhi987 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

The strict inline-eval approval boundary needs source-level and gateway-path coverage for version-suffixed interpreter names and inline flags so those command forms continue to require explicit approval under strictInlineEval=true.

Why This Change Was Made

This makes versioned interpreter matching explicit in the inline-eval detector specs for the affected interpreter families, adds direct detector coverage for a versioned PHP binary, and broadens approval-path regression coverage for the reported carrier forms. It also verifies those carriers do not become reusable allow-always entries in strict mode.

This PR was AI-assisted.

User Impact

Operators keep the same strict inline-eval guardrail behavior for interpreter inline-code commands. The change is limited to detector matching and regression coverage; it does not add config, migration, provider, plugin, channel, storage, or protocol behavior.

Evidence

  • node scripts/run-vitest.mjs src/infra/command-analysis/inline-eval.test.ts src/infra/command-analysis/risks.test.ts src/infra/exec-approvals-allow-always.test.ts src/agents/bash-tools.exec-host-gateway.test.ts

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS maintainer Maintainer-authored PR labels Jun 29, 2026
@clawsweeper

clawsweeper Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 29, 2026, 11:29 AM ET / 15:29 UTC.

Summary
The PR changes strict inline-eval interpreter matching and adds regression coverage for versioned Python/PyPy, PHP inline flags, Rscript eval, gateway denial, and strict allow-always persistence.

PR surface: Source +13, Tests +120. Total +133 across 4 files.

Reproducibility: yes. from source and related proof: the original gap was exact interpreter matching feeding strictInlineEval, and #96216 already demonstrates the fixed node-host denial and one-shot allow-always path. I did not run tests because this review is read-only.

Review metrics: 1 noteworthy metric.

  • Approval detector surface: 1 runtime detector changed after current-main fix. The branch is not tests-only against current main, so maintainers need to review a security-sensitive detector shape change rather than only added coverage.

Stored data model
Persistent data-model change detected: serialized state: src/infra/exec-approvals-allow-always.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: partial_overlap
Canonical: #96216
Summary: The merged PR owns the runtime detector hardening; this PR overlaps that work and adds additional test coverage on adjacent gateway and persistence paths.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Refresh on current main and remove the duplicate production detector changes unless a maintainer chooses that contract.
  • [P1] Add redacted terminal, log, or live-output proof from a real gateway or node-host denial path, with private endpoints, IPs, keys, and phone numbers redacted.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The latest evidence lists tests, CI, review artifacts, and GHSA gates, but it explicitly says live interpreter execution through a real gateway or node host was not tested; external runtime-boundary PRs still need redacted terminal/log/live output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The runtime detector edit adds a second per-spec versioned-name path even though current main already handles version suffixes centrally, so future interpreter additions could drift between the two concepts.
  • [P1] The PR changes a strict exec approval boundary with compatibility and security impact; strictInlineEval users can see different approval prompts or allow-always persistence behavior if the detector shape changes.
  • [P1] The latest contributor proof is still test/CI/GHSA-gate focused and explicitly says no live interpreter execution through a real gateway or node host was tested for this branch.

Maintainer options:

  1. Refresh to unique regression coverage (recommended)
    Remove the duplicate versionedNames runtime path, keep only tests that add coverage beyond current main, and add redacted terminal output from a real gateway or node-host denial path.
  2. Adopt per-spec version matching deliberately
    If maintainers want versionedNames as the new detector contract, document why it replaces the current central suffix helper and add compatibility coverage for the existing strictInlineEval cases before merge.
  3. Pause as superseded after extracting tests
    If the extra tests are not worth adopting, close this branch as overlap with the merged detector fix after linking fix(exec): gate versioned inline interpreters #96216.

Next step before merge

  • [P1] The protected maintainer label, security-sensitive approval boundary, and choice between coverage-only refresh versus a new detector contract need maintainer handling rather than an automated repair lane.

Security
Needs attention: The diff has no supply-chain issue, but it changes a security-sensitive exec approval detector in a way that duplicates current-main behavior and needs maintainer attention before merge.

Review findings

  • [P1] Drop the duplicate versioned interpreter path — src/infra/command-analysis/inline-eval.ts:18-20
Review details

Best possible solution:

Refresh the branch on current main, remove the duplicate production detector reshaping unless a maintainer explicitly wants that new contract, and keep only the useful php8.3, gateway denial, and allow-always regression coverage with real terminal proof.

Do we have a high-confidence way to reproduce the issue?

Yes from source and related proof: the original gap was exact interpreter matching feeding strictInlineEval, and #96216 already demonstrates the fixed node-host denial and one-shot allow-always path. I did not run tests because this review is read-only.

Is this the best way to solve the issue?

No as currently shaped: the extra tests are useful, but the production detector reshaping duplicates behavior already present on current main. The safer fix is to refresh this branch into a narrow coverage-only PR unless maintainers intentionally choose the new per-spec detector contract.

Full review comments:

  • [P1] Drop the duplicate versioned interpreter path — src/infra/command-analysis/inline-eval.ts:18-20
    Current main already normalizes versioned interpreter names through interpreterNameVariants, and the merged fix(exec): gate versioned inline interpreters #96216 owns the runtime detector hardening. Adding a new versionedNames field here creates a second source of truth for the same approval-boundary behavior; refresh on current main and keep the unique tests unless maintainers explicitly want to replace the detector contract.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1052652a7168.

Label changes

Label justifications:

  • P2: This is a normal-priority exec approval hardening/test PR with limited blast radius because the central runtime fix is already on current main.
  • merge-risk: 🚨 compatibility: Changing strictInlineEval matching can alter which existing allowlisted interpreter commands require explicit approval or remain one-shot.
  • merge-risk: 🚨 security-boundary: The diff changes command classification that gates exec approval and allow-always persistence on a security-sensitive path.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The latest evidence lists tests, CI, review artifacts, and GHSA gates, but it explicitly says live interpreter execution through a real gateway or node host was not tested; external runtime-boundary PRs still need redacted terminal/log/live output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +13, Tests +120. Total +133 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 1 27 14 +13
Tests 3 122 2 +120
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 149 16 +133

Security concerns:

  • [medium] Duplicate detector source of truth on approval boundary — src/infra/command-analysis/inline-eval.ts:18
    Adding per-spec versioned interpreter metadata creates a second way to decide strictInlineEval coverage, which can drift from the current central suffix-normalization helper and affect exec approval enforcement.
    Confidence: 0.82

What I checked:

Likely related people:

  • eleqtrizit: Merged fix(exec): gate versioned inline interpreters #96216, which added the version-suffixed interpreter, PHP flag, and R/Rscript detection that this branch overlaps. (role: introduced related detector behavior; confidence: high; commits: 3168987b288f, 55c3bff6ae73, f420505b85ba; files: src/infra/command-analysis/inline-eval.ts, src/infra/command-analysis/inline-eval.test.ts, src/node-host/invoke-system-run.test.ts)
  • jesse-merhi: Authored broad command authorization planner work touching allow-always persistence, gateway approval tests, and strict inline-eval no-persistence behavior. (role: adjacent exec approval owner; confidence: medium; commits: 87372b6057d6, ee369d68e13e, 40ace0806537; files: src/infra/exec-approvals-allowlist.ts, src/infra/exec-approvals-allow-always.test.ts, src/agents/bash-tools.exec-host-gateway.test.ts)
  • pgondhi987: Besides authoring this PR, previously merged work in fix: detect carried exec command forms [AI] #81000 touched carried exec command detection in the same command-analysis area. (role: prior adjacent contributor; confidence: medium; commits: 65d7232218a9, bf5358020684; files: src/infra/command-analysis/risks.ts, src/infra/command-analysis/risks.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 29, 2026
@pgondhi987

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Updated after the earlier review: this head now includes the production inline-eval detector change in src/infra/command-analysis/inline-eval.ts, direct detector coverage, strict allow-always persistence coverage, and gateway denial coverage for the reported carrier forms. The PR body evidence has been refreshed for head 07357d6ffeaa5ed757b0ae65f700b86764f7a590.

@clawsweeper

clawsweeper Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 29, 2026
@pgondhi987

Copy link
Copy Markdown
Contributor Author

Verification before merge:

Behavior addressed: strict inline-eval detection now recognizes versioned Python/PyPy interpreter names, PHP inline flags, and R/Rscript eval forms so those inline-code commands stay on the existing approval path.
Real environment tested: local source checkout and GitHub Actions PR checks; no delegated remote provider was used.
Exact steps or command run after this patch:

  • node scripts/run-vitest.mjs src/infra/command-analysis/inline-eval.test.ts src/infra/command-analysis/risks.test.ts src/infra/exec-approvals-allow-always.test.ts src/agents/bash-tools.exec-host-gateway.test.ts
  • auto-pr.sh issue 790 --run-stage ghsa_dry_run
  • scripts/pr review-validate-artifacts 97833
  • timeout 1800 .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
  • auto-pr.sh issue 790 --run-stage ghsa_real_gate
    Evidence after fix:
  • Focused local validation passed: 3 Vitest shards, 152 tests.
  • review-pr: passed, READY FOR /prepare-pr with 0 findings; autoreview: passed with no accepted/actionable findings.
  • GHSA dry-run: passed; GHSA real gate: passed.
  • GitHub CI for head 07357d6ffeaa5ed757b0ae65f700b86764f7a590: no failing or pending status checks.
    Observed result after fix: the detector, allow-always persistence, and gateway denial paths all cover the reported interpreter forms without allowing reusable strict-inline-eval approval persistence.
    What was not tested: live interpreter execution through a real gateway/node host; no known untested critical path for the detector and approval-boundary logic covered here.

Regression Risk:
Low. The affected boundary is the existing exec approval guardrail under strictInlineEval; this does not change auth, sandboxing, config, storage, provider, plugin, channel, dependency, migration, or protocol surfaces. SECURITY.md treats this class as hardening inside the trusted-operator exec approval model, and targeted tests plus CI bound the compatibility risk.

Best fix verdict: appropriate. The detector table owns interpreter spelling and inline-flag classification, while gateway tests verify enforcement after detection; moving this into auth, sandbox, provider, plugin, or channel policy would be the wrong layer.

User behavior change:
Before, some inline interpreter spellings could avoid strict inline-eval classification. After, versioned Python/PyPy forms, PHP inline flags, and R/Rscript eval forms are classified consistently; operators may see approval prompts for those commands when strictInlineEval is enabled. No config shape, default setting, provider/plugin/channel behavior, storage, migration, or persisted credential behavior changes.

@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jun 29, 2026
@pgondhi987 pgondhi987 closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant