Skip to content

fix: resolve {{ recipe_dir }} in nested sub-recipe paths during secret discovery#7797

Merged
DOsinga merged 1 commit intomainfrom
fix/5040-nested-sub-recipe-secret-discovery
Mar 11, 2026
Merged

fix: resolve {{ recipe_dir }} in nested sub-recipe paths during secret discovery#7797
DOsinga merged 1 commit intomainfrom
fix/5040-nested-sub-recipe-secret-discovery

Conversation

@DOsinga
Copy link
Copy Markdown
Collaborator

@DOsinga DOsinga commented Mar 11, 2026

Summary

Fixes #5040

When a recipe has nested sub-recipes that use {{ recipe_dir }} in their paths, the secret discovery phase would try to load files with the literal unresolved template string as the path. This caused confusing output like:

📦 Looking for recipe "{{ recipe_dir }}/leaf.yaml" in github repo: squareup/goose-recipes
fatal: not a valid object name: origin/main:{{ recipe_dir }}/leaf.yaml

It also meant secrets from deeply nested sub-recipes were never discovered.

Root Cause

discover_recipe_secrets_recursive loads sub-recipe files as raw YAML (no template rendering). When those sub-recipes themselves have sub_recipes entries with {{ recipe_dir }} in the path, the recursive call uses the literal unrendered string as a file path, which fails local lookup and falls through to a confusing GitHub repository search.

Fix

When recursing into a loaded sub-recipe, resolve {{ recipe_dir }} (with any whitespace variation) to the sub-recipe file's actual parent directory before the recursive call. This is done via a new discover_sub_recipe_secrets helper that uses a regex replacement with NoExpand to treat the directory path as a literal string (preventing $ in paths from being misinterpreted as capture group references).

Testing

Validated with a 3-level recipe hierarchy (top → middle → leaf) where the middle and leaf recipes use {{ recipe_dir }} in sub-recipe paths:

Before (main): Found only 1 of 2 secrets, produced confusing GitHub lookup output for {{ recipe_dir }}/leaf.yaml

After (this PR): Found all 2 secrets, no confusing output

Reproduction script
#!/usr/bin/env bash
#
# Test for https://github.com/block/goose/issues/5040
#
# Verifies that nested sub-recipes using {{ recipe_dir }} in their paths
# don't produce confusing "Looking for recipe" / GitHub lookup output
# during secret discovery.
#
set -euo pipefail

GOOSE_DIR="${GOOSE_DIR:-$HOME/proj/wt-goose-3}"
RECIPE_DIR=$(mktemp -d)
trap 'rm -rf "$RECIPE_DIR"' EXIT

echo "=== Setting up test recipes in $RECIPE_DIR ==="

# Level 2: a leaf sub-recipe with an extension that declares env_keys
cat > "$RECIPE_DIR/leaf.yaml" <<'EOF'
version: "1.0.0"
title: "Leaf Sub-Recipe"
description: "A leaf recipe with secrets"
instructions: "You are a helper."
extensions:
  - name: "leaf-service"
    type: "streamable_http"
    uri: "http://localhost:9999/mcp"
    env_keys:
      - "LEAF_SECRET_TOKEN"
EOF

# Level 1: a mid-level sub-recipe that references the leaf via {{ recipe_dir }}
cat > "$RECIPE_DIR/middle.yaml" <<'EOF'
version: "1.0.0"
title: "Middle Sub-Recipe"
description: "A mid-level recipe that nests another sub-recipe"
instructions: "You are a helper."
extensions:
  - name: "middle-service"
    type: "streamable_http"
    uri: "http://localhost:9998/mcp"
    env_keys:
      - "MIDDLE_SECRET_TOKEN"
sub_recipes:
  - name: "leaf"
    path: "{{ recipe_dir }}/leaf.yaml"
EOF

# Level 0: the top-level recipe that references the middle recipe
cat > "$RECIPE_DIR/top.yaml" <<EOF
version: "1.0.0"
title: "Top-Level Recipe"
description: "A recipe that tests nested sub-recipe secret discovery"
prompt: "Say exactly: done"
extensions:
  - name: "developer"
    type: "builtin"
sub_recipes:
  - name: "middle"
    path: "{{ recipe_dir }}/middle.yaml"
EOF

echo "=== Recipe files ==="
ls -la "$RECIPE_DIR"
echo

echo "=== Building goose ==="
cd "$GOOSE_DIR"
source bin/activate-hermit
cargo build --bin goose 2>&1 | tail -3
echo

echo "=== Running recipe ==="
OUTPUT_FILE=$(mktemp)
# Run the recipe with max-turns 1 so it finishes quickly.
# Capture all output (stdout + stderr) to check for the bug.
set +e
GOOSE_PROVIDER="${GOOSE_PROVIDER:-anthropic}" \
GOOSE_MODEL="${GOOSE_MODEL:-claude-sonnet-4-20250514}" \
cargo run --bin goose -- run \
    --recipe "$RECIPE_DIR/top.yaml" \
    --no-profile \
    --max-turns 1 \
    --quiet \
    > "$OUTPUT_FILE" 2>&1
EXIT_CODE=$?
set -e

echo
echo "=== Checking output for issue #5040 symptoms ==="
FAILED=0

# The bug manifests as lines like:
#   📦 Looking for recipe "{{ recipe_dir }}/leaf.yaml" in github repo: ...
#   fatal: not a valid object name: origin/main:{{ recipe_dir }}/...
if grep -q "Looking for recipe.*recipe_dir" "$OUTPUT_FILE"; then
    echo "❌ FAIL: Found confusing GitHub lookup for unresolved {{ recipe_dir }} path"
    echo "   Matching lines:"
    grep "Looking for recipe.*recipe_dir" "$OUTPUT_FILE" | sed 's/^/   /'
    FAILED=1
fi

if grep -q "fatal:.*recipe_dir" "$OUTPUT_FILE"; then
    echo "❌ FAIL: Found git fatal error for unresolved {{ recipe_dir }} path"
    echo "   Matching lines:"
    grep "fatal:.*recipe_dir" "$OUTPUT_FILE" | sed 's/^/   /'
    FAILED=1
fi

if grep -q '{{.*recipe_dir.*}}' "$OUTPUT_FILE"; then
    echo "❌ FAIL: Found unresolved {{ recipe_dir }} in output"
    echo "   Matching lines:"
    grep '{{.*recipe_dir.*}}' "$OUTPUT_FILE" | sed 's/^/   /'
    FAILED=1
fi

if [ "$FAILED" -eq 0 ]; then
    echo "✅ PASS: No confusing {{ recipe_dir }} lookup output detected"
fi

echo
echo "=== Full output (for reference) ==="
cat "$OUTPUT_FILE"
rm -f "$OUTPUT_FILE"

exit "$FAILED"

Changes

  • crates/goose-cli/src/recipes/secret_discovery.rs:
    • load_sub_recipe now returns the parent directory alongside the recipe
    • New discover_sub_recipe_secrets resolves {{ recipe_dir }} before recursing
    • Uses regex::NoExpand for safe literal replacement

…t discovery

When a recipe has nested sub-recipes that use {{ recipe_dir }} in their
paths, the secret discovery phase would try to load files with the literal
unresolved template string as the path. This caused confusing output like:

  📦 Looking for recipe "{{ recipe_dir }}/leaf.yaml" in github repo: ...
  fatal: not a valid object name: origin/main:{{ recipe_dir }}/...

The fix resolves {{ recipe_dir }} to the sub-recipe's actual parent
directory before recursing into nested sub-recipes. This eliminates the
confusing output and also correctly discovers secrets from all nesting
levels.

Uses regex::NoExpand to treat the replacement path as a literal string,
preventing $ characters in directory paths from being misinterpreted as
capture group references.

Fixes #5040
@DOsinga DOsinga requested a review from wpfleger96 March 11, 2026 00:18
parent_dir: &str,
visited_recipes: &mut HashSet<String>,
) -> Vec<SecretRequirement> {
let re = Regex::new(r"\{\{\s*recipe_dir\s*\}\}").expect("valid regex");
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be put in a OnceCell so it isn't compiled every time

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, but is it worth the mental overhead? this doesn't happen very often at all

Copy link
Copy Markdown
Collaborator

@wpfleger96 wpfleger96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice thank you!

@DOsinga DOsinga added this pull request to the merge queue Mar 11, 2026
Merged via the queue into main with commit 58f3cc9 Mar 11, 2026
20 checks passed
@DOsinga DOsinga deleted the fix/5040-nested-sub-recipe-secret-discovery branch March 11, 2026 01:53
lifeizhou-ap added a commit that referenced this pull request Mar 11, 2026
* main: (45 commits)
  fix: resolve {{ recipe_dir }} in nested sub-recipe paths during secret discovery (#7797)
  Add @DOsinga as CODEOWNER for documentation (#7799)
  feat: Add summarize tool for deterministic reads (#7054)
  fix(api): use camelCase in CallToolResponse and add type discriminators to ContentBlock (#7487)
  feat: ACP providers for claude code and codex (#6605)
  chore(deps): bump express-rate-limit from 8.2.1 to 8.3.0 in /evals/open-model-gym/mcp-harness (#7703)
  feat(openai): capture reasoning summaries from responses API (#7375)
  Fix some dependencies (#7794)
  fix: improve keyring availability error detection (#7766)
  feat: add MiniMax provider with Anthropic-compatible API (#7640)
  feat: add Tensorix as a declarative provider (#7712)
  fix(security): remove insecure default secret from GOOSE_EXTERNAL_BACKEND (#7783)
  refactor: Convert Tanzu provider to declarative JSON config (#7124)
  replaces https://github.com/block/goose/pull/7340/changes (#7786)
  feat(summon): make skill supporting files individually loadable via load() (#7583)
  Keep toast open on failed extension (#7771)
  fix(ui-desktop): unify path resolution around GOOSE_PATH_ROOT (#7335)
  fix: pass OAuth scopes to DCR and extract granted_scopes from token response (#7571)
  fix: write to real file if config.yaml is symlink (#7669)
  fix: preserve pairings when stopping gateway (#7733)
  ...
lifeizhou-ap added a commit that referenced this pull request Mar 11, 2026
* main: (69 commits)
  fix: resolve {{ recipe_dir }} in nested sub-recipe paths during secret discovery (#7797)
  Add @DOsinga as CODEOWNER for documentation (#7799)
  feat: Add summarize tool for deterministic reads (#7054)
  fix(api): use camelCase in CallToolResponse and add type discriminators to ContentBlock (#7487)
  feat: ACP providers for claude code and codex (#6605)
  chore(deps): bump express-rate-limit from 8.2.1 to 8.3.0 in /evals/open-model-gym/mcp-harness (#7703)
  feat(openai): capture reasoning summaries from responses API (#7375)
  Fix some dependencies (#7794)
  fix: improve keyring availability error detection (#7766)
  feat: add MiniMax provider with Anthropic-compatible API (#7640)
  feat: add Tensorix as a declarative provider (#7712)
  fix(security): remove insecure default secret from GOOSE_EXTERNAL_BACKEND (#7783)
  refactor: Convert Tanzu provider to declarative JSON config (#7124)
  replaces https://github.com/block/goose/pull/7340/changes (#7786)
  feat(summon): make skill supporting files individually loadable via load() (#7583)
  Keep toast open on failed extension (#7771)
  fix(ui-desktop): unify path resolution around GOOSE_PATH_ROOT (#7335)
  fix: pass OAuth scopes to DCR and extract granted_scopes from token response (#7571)
  fix: write to real file if config.yaml is symlink (#7669)
  fix: preserve pairings when stopping gateway (#7733)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Confusing Output When Loading Nested Sub-Recipes

3 participants