Skip to content

fix: stop evaluating step backticks by default#2075

Merged
yottahmd merged 1 commit intomainfrom
fix/disable-step-backtick-substitution
Apr 30, 2026
Merged

fix: stop evaluating step backticks by default#2075
yottahmd merged 1 commit intomainfrom
fix/disable-step-backtick-substitution

Conversation

@yottahmd
Copy link
Copy Markdown
Collaborator

@yottahmd yottahmd commented Apr 30, 2026

Summary

Stop evaluating backtick command substitution in step-owned fields by default.

What Changed

  • add field-aware evaluator hooks for executor capabilities so command, script, and config fields can declare their own evaluation policy
  • disable backtick substitution by default for step command, script, config, sub-DAG, parallel, and other step-owned text fields while preserving condition evaluation behavior
  • update affected runtime call sites and executor eval-option wiring to use the new step-field contract
  • add regression coverage for harness prompt/script paths and generic executor config with literal fenced content

Why

Step-owned text was being routed through the generic evaluator pipeline, which treats backticks as command substitution. That made prompts, templates, config strings, and other non-shell step fields brittle: a rendered value containing fenced Markdown or YAML could be executed accidentally during step setup instead of being passed through literally.

Validation

  • go test ./internal/core ./internal/runtime ./internal/runtime/builtin/command ./internal/runtime/builtin/docker ./internal/runtime/builtin/ssh -run 'Test(Step_EvalOptions|EvalExecutorConfig_DefaultPreservesLiteralCodeFencesInData|SetupExecutor_Harness(Command|Script)PreservesLiteralCodeFences|CommandExecutor_GetEvalOptions|DockerExecutor_GetEvalOptions|SSHExecutor_GetEvalOptions)' -count=1
  • go test ./internal/core/... ./internal/runtime/... -count=1
  • go test ./... -run ^ -count=1

Summary by CodeRabbit

  • Refactor

    • Improved internal evaluation consistency across command, script, and configuration steps to ensure more appropriate variable substitution and character escaping behavior in different execution contexts.
  • Bug Fixes

    • Enhanced preservation of literal code content and backticks during step execution across various executor types.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 30, 2026

📝 Walkthrough

Walkthrough

The changes introduce field-specific eval-option hooks to ExecutorCapabilities (CommandEvalOptions, ScriptEvalOptions, ConfigEvalOptions) and new Step methods that compute eval options by defaulting to WithoutSubstitute() then applying executor-provided options. A new EvalStepString function enforces substitution-disabled evaluation for step fields, with the legacy EvalOptions method deprecated.

Changes

Cohort / File(s) Summary
Core Capability Types
internal/core/capabilities.go, internal/core/capabilities_test.go
Extended ExecutorCapabilities with three new hooks (GetCommandEvalOptions, GetScriptEvalOptions, GetConfigEvalOptions); added corresponding Step methods that default to WithoutSubstitute() and apply executor options; deprecated Step.EvalOptions() in favor of CommandEvalOptions().
Eval Runtime Infrastructure
internal/runtime/eval.go
Added EvalStepString() function that forces WithoutSubstitute() for step field evaluation; removed legacy evalObjectTreatingOmittedParamsAsEmpty helper.
Node Execution & Data Setup
internal/runtime/node.go, internal/runtime/data.go
Updated sub-DAG and command/script evaluation to use EvalStepString() and field-specific eval option methods (CommandEvalOptions, ScriptEvalOptions, ConfigEvalOptions); reworked executor config evaluation paths to use ConfigEvalOptions().
Node Execution Tests
internal/runtime/node_internal_test.go
Added three new integration tests validating literal fenced code preservation during executor setup and config evaluation for both command and script execution paths.
Chat & Agent Step Executors
internal/runtime/builtin/chat/executor.go, internal/runtime/builtin/agentstep/executor.go
Replaced EvalString with EvalStepString for message content evaluation across chat messages, API keys, and base URLs.
Command Executor
internal/runtime/builtin/command/command.go, internal/runtime/builtin/command/eval_options_test.go
Updated executor hook registration from GetEvalOptions to GetCommandEvalOptions and GetScriptEvalOptions; test helper now uses CommandEvalOptions() with added Substitute=false assertion.
Docker Executor
internal/runtime/builtin/docker/executor.go, internal/runtime/builtin/docker/eval_options_test.go
Replaced EvalString with EvalStepString for container field and slice element evaluation; updated hook registration from GetEvalOptions to GetCommandEvalOptions; test helper now derives options via CommandEvalOptions().
JQ Executor
internal/runtime/builtin/jq/jq.go
Replaced EvalString with EvalStepString for config.input path evaluation.
SSH Executor
internal/runtime/builtin/ssh/ssh.go, internal/runtime/builtin/ssh/ssh_test.go
Updated hook registration from GetEvalOptions to GetCommandEvalOptions with shell-aware dollar escaping; test now uses CommandEvalOptions() with added Substitute=false assertion.
Template Executor
internal/runtime/builtin/template/template.go
Replaced GetEvalOptions hook with GetScriptEvalOptions, preserving WithNoExpansion() behavior.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 51.85% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately summarizes the main change: disabling backtick (command substitution) evaluation by default for step-owned fields, which is the core objective.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/disable-step-backtick-substitution

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/runtime/builtin/docker/executor.go (1)

471-484: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use step-aware evaluation for container.env values too.

ct.Env still flows through evalEnvSequentially, and Line 521 there still uses runtime.EvalString(...). That leaves backtick substitution enabled for step-owned container env values, so inputs like FOO=\uname`can still execute even though the other container fields were moved toEvalStepString`.

Suggested fix
 func evalEnvSequentially(ctx context.Context, envs []string) ([]string, error) {
 	if len(envs) == 0 {
 		return envs, nil
 	}
 	resolved := make(map[string]string, len(envs))
 	result := make([]string, 0, len(envs))
 	for _, entry := range envs {
 		key, rawVal, found := strings.Cut(entry, "=")
 		if !found {
 			result = append(result, entry)
 			continue
 		}
-		val, err := runtime.EvalString(ctx, rawVal, eval.WithVariables(resolved))
+		val, err := runtime.EvalStepString(ctx, rawVal, eval.WithVariables(resolved))
 		if err != nil {
 			return nil, fmt.Errorf("env %s: %w", key, err)
 		}
 		resolved[key] = val
 		result = append(result, key+"="+val)
 	}
 	return result, nil
 }

Also applies to: 490-498

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@internal/runtime/builtin/docker/executor.go` around lines 471 - 484, ct.Env
is still evaluated via evalEnvSequentially which uses runtime.EvalString and
thus allows backtick substitution for step-owned env values; change the env
evaluation to the step-aware evaluator used for other container fields (i.e.,
use EvalStepString instead of runtime.EvalString). Update evalEnvSequentially
(or create a new evalEnvStepSequentially) to call runtime.EvalStepString for
each env value so container environment variables are evaluated with step
scope/escaping consistent with ct.Volumes, ct.Ports, ct.Command and ct.Shell.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@internal/runtime/builtin/docker/executor.go`:
- Around line 471-484: ct.Env is still evaluated via evalEnvSequentially which
uses runtime.EvalString and thus allows backtick substitution for step-owned env
values; change the env evaluation to the step-aware evaluator used for other
container fields (i.e., use EvalStepString instead of runtime.EvalString).
Update evalEnvSequentially (or create a new evalEnvStepSequentially) to call
runtime.EvalStepString for each env value so container environment variables are
evaluated with step scope/escaping consistent with ct.Volumes, ct.Ports,
ct.Command and ct.Shell.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: efea73cf-4246-4ae1-b270-a485a17f99ff

📥 Commits

Reviewing files that changed from the base of the PR and between 5d9abfe and 894c829.

📒 Files selected for processing (16)
  • internal/core/capabilities.go
  • internal/core/capabilities_test.go
  • internal/runtime/builtin/agentstep/executor.go
  • internal/runtime/builtin/chat/executor.go
  • internal/runtime/builtin/command/command.go
  • internal/runtime/builtin/command/eval_options_test.go
  • internal/runtime/builtin/docker/eval_options_test.go
  • internal/runtime/builtin/docker/executor.go
  • internal/runtime/builtin/jq/jq.go
  • internal/runtime/builtin/ssh/ssh.go
  • internal/runtime/builtin/ssh/ssh_test.go
  • internal/runtime/builtin/template/template.go
  • internal/runtime/data.go
  • internal/runtime/eval.go
  • internal/runtime/node.go
  • internal/runtime/node_internal_test.go

@yottahmd yottahmd merged commit d60fa49 into main Apr 30, 2026
10 checks passed
@yottahmd yottahmd deleted the fix/disable-step-backtick-substitution branch April 30, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant