Skip to content

fix(llm-cli): address post-review issues in CLI integration#1097

Merged
muddlebee merged 10 commits intoTracer-Cloud:mainfrom
muddlebee:fix/llm-cli-review-followup
Apr 30, 2026
Merged

fix(llm-cli): address post-review issues in CLI integration#1097
muddlebee merged 10 commits intoTracer-Cloud:mainfrom
muddlebee:fix/llm-cli-review-followup

Conversation

@muddlebee
Copy link
Copy Markdown
Collaborator

Follows up on #781 and #795 with fixes surfaced during code review.

Summary

  • Bug fix: _run_cli_llm_onboarding retry loop now sets os.environ[env_key] after writing to .env so the in-process binary resolver immediately sees the user-provided path — without this, the retry always re-detected "not found" even after the user entered a valid path
  • Security: user-supplied binary path is now validated (isfile + executable check) before being persisted to .env
  • Reliability: bumped npm_prefix_bin_dirs subprocess timeout from 0.3 s → 2.0 s to avoid false-empty @lru_cache on slow nvm/volta shim startup
  • Testability: resolve_cli_binary defaults for which_resolver / runnable_check changed from bound-at-definition-time function objects to None sentinels with call-time lookup — this makes patch("binary_resolver.shutil.which") work without callers passing explicit overrides
  • Cleanup: dropped redundant shutil import and explicit default kwargs from CodexAdapter._resolve_binary; consolidated four separate binary_resolver import blocks into one
  • Correctness: _ver_tuple switched to re.findall so segments like "1.2a.3" or pre-release suffixes ("1.2.3-beta.4") parse correctly
  • Observability: CLI stderr is now logged at DEBUG after a successful parse so warnings (e.g. deprecation notices from codex) are visible in debug output
  • Tests: updated all shutil.which patch targets from codex.shutil.whichbinary_resolver.shutil.which to match where resolution now lives

Test plan

  • make lint passes
  • make format-check passes
  • make typecheck passes
  • make test-cov passes (4149 passed locally)
  • tests/integrations/llm_cli/ — all 16 tests pass

- fix: set os.environ[env_key] after sync_env_values so in-process retry
  loop sees the user-provided binary path immediately (Tracer-Cloud#781 follow-up)
- fix: validate user-supplied binary path (isfile + executable) before
  persisting to .env to prevent silently storing broken paths
- fix: bump npm_prefix_bin_dirs subprocess timeout from 0.3s to 2.0s to
  avoid false-empty cache on slow nvm/volta shim startup
- refactor: use None sentinels for resolve_cli_binary defaults so
  shutil.which and is_runnable_binary are looked up at call time, making
  the function patchable in tests without explicit overrides
- refactor: drop redundant shutil import and explicit which_resolver /
  runnable_check kwargs from CodexAdapter._resolve_binary
- refactor: consolidate four separate binary_resolver import blocks into one
- fix: _ver_tuple now uses re.findall to correctly parse versions with
  non-digit segments (e.g. "1.2a.3", "1.2.3-beta.4")
- feat: log CLI stderr at DEBUG level after successful parse so warnings
  from the CLI are not silently discarded
- test: update patch targets from codex.shutil.which to
  binary_resolver.shutil.which after resolution moved to shared module
…S.md

- Add three-state logged_in contract table (True/False/None) with wizard behaviour
- Document _classify_*_auth pattern and negative-phrases-first rule
- Warn about _SAFE_SUBPROCESS_ENV_PREFIXES coupling in runner.py for new CLIs
- Update provider checklist with auth probe and env forwarding steps
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 30, 2026

Greptile Summary

This follow-up PR addresses several post-review issues in the CLI LLM integration: the retry loop now correctly propagates a user-entered binary path into the in-process environment, _ver_tuple is fixed to handle pre-release semver correctly (the old split+isdigit approach mapped "1.2.3-beta.4" to (1,2,4) rather than (1,2,3)), and the resolve_cli_binary default arguments are switched to None-sentinel call-time lookups to make shutil.which patches transparent to tests.

Confidence Score: 5/5

Safe to merge; all findings are P2 style/consistency suggestions with no blocking correctness issues.

The core bug fixes (env propagation, _ver_tuple, sentinel defaults) are all correct. Remaining comments are a Windows-only edge case for executable validation (low real-world impact) and an incomplete import consolidation. No P0 or P1 issues found.

app/cli/wizard/flow.py — Windows executable check uses os.access instead of is_runnable_binary; app/integrations/llm_cli/codex.py — imports could be further consolidated into one block

Important Files Changed

Filename Overview
app/cli/wizard/flow.py Adds isfile + X_OK validation before persisting user-supplied binary path, and sets os.environ[env_key] so the retry loop re-detects successfully; Windows executable validation may diverge from is_runnable_binary
app/integrations/llm_cli/binary_resolver.py Timeout raised 0.3→2.0 s for npm_prefix_bin_dirs subprocess; resolve_cli_binary switches which_resolver/runnable_check defaults to None sentinels with call-time lookup, enabling monkeypatching in tests without explicit overrides
app/integrations/llm_cli/codex.py Removes redundant shutil import and explicit kwargs; _ver_tuple switched from split+isdigit to re.findall fixing pre-release version parsing (e.g. "1.2.3-beta.4" previously mapped to (1,2,4))
app/integrations/llm_cli/runner.py Adds a DEBUG-level log of CLI stderr (truncated to 500 chars) after a successful parse, surfacing deprecation warnings from tools like codex
tests/integrations/llm_cli/test_codex_adapter.py All shutil.which and is_runnable_binary patch targets updated from codex.* to binary_resolver.* to match the new resolution location; no logic changes to test assertions

Sequence Diagram

sequenceDiagram
    participant W as Wizard (flow.py)
    participant A as CodexAdapter
    participant R as resolve_cli_binary
    participant E as os.environ / .env

    loop Up to 10 attempts
        W->>A: adapter.detect()
        A->>R: resolve_cli_binary(explicit_env_key, ...)
        R->>E: os.getenv(explicit_env_key)
        E-->>R: saved path or empty string
        alt env path set and is_runnable_binary passes
            R-->>A: binary path
        else shutil.which finds binary on PATH
            R-->>A: binary path
        else fallback paths scanned
            R-->>A: binary path or None
        end
        A-->>W: CLIProbe(installed, logged_in)

        alt installed and logged_in is True
            W-->>W: return ok
        else installed but not logged in
            W->>W: prompt → retry or repick
        else not installed
            W->>W: prompt → retry / enter path / repick
            opt user enters path
                W->>W: os.path.isfile check
                W->>W: os.access X_OK check
                W->>E: sync_env_values to .env
                W->>E: os.environ update in-process
                note right of W: next detect() sees the path immediately
            end
        end
    end
    W-->>W: return abort (max retries exceeded)
Loading

Comments Outside Diff (1)

  1. app/integrations/llm_cli/codex.py, line 10-18 (link)

    P2 Import blocks not fully consolidated

    The PR description says "consolidated four separate binary_resolver import blocks into one," but there are still three separate from app.integrations.llm_cli.binary_resolver import (...) blocks. Consider collapsing them into a single block:

    from app.integrations.llm_cli.binary_resolver import (
        candidate_binary_names as _candidate_binary_names,
        default_cli_fallback_paths as _default_cli_fallback_paths,
        resolve_cli_binary,
    )

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "docs(llm-cli): document auth probe patte..." | Re-trigger Greptile

Comment thread app/cli/wizard/flow.py Outdated
@muddlebee muddlebee changed the title fix(llm-cli): address post-review issues in CLI integration WIP : fix(llm-cli): address post-review issues in CLI integration Apr 30, 2026
…orms

Replace os.path.isfile + os.access(X_OK) with shared is_runnable_binary
from binary_resolver, which already handles Windows extension checks
(.cmd, .exe, .ps1, .bat). os.access(X_OK) returns False for valid
Windows executables that lack Unix-style execute bits.
@muddlebee
Copy link
Copy Markdown
Collaborator Author

Audit


Binary Resolution (binary_resolver.py)

Scenario Covered? How
Linux — PATH lookup shutil.which("codex") + fallback paths
macOS — Homebrew /opt/homebrew/bin darwin-only fallback path
Windows.cmd/.exe/.ps1/.bat extensions candidate_binary_names + is_runnable_binary extension check
WindowsAPPDATA/npm, LOCALAPPDATA/Programs default_cli_fallback_paths win32 branch
Windows — npm prefix uses root (no /bin) npm_prefix_bin_dirs win32 branch + test
nvm/volta/pnpm install dirs .volta/bin, PNPM_HOME, XDG_DATA_HOME/pnpm, npm prefix
CODEX_BIN set but invalid Falls through to PATH/fallbacks + test
CODEX_BIN set and valid Short-circuits + test
CODEX_BIN empty/blank .strip() makes it falsy, falls through
npm not installed OSError caught, returns ()
npm hangs/slow shim 2.0s timeout + TimeoutExpired caught
Duplicate fallback paths Deduplication via seen set
Spaces in path Path objects handle this; expanduser for ~
npm_config_prefix case-insensitive env Explicit .lower() scan of os.environ
Symlinks added checks diagnose_binary_path( )

Tests: Linux, macOS, Windows fallback paths all have dedicated tests. npm prefix has win32/unix tests.


Auth Probe (codex.py)

Edge case Covered? How
"Not logged in" vs "logged in" substring Negative phrases checked first
Exit 0 but "Not logged in" text Dedicated test test_detect_not_logged_in_exit_zero
Expired/invalid token _classify_codex_auth checks both
Rate-limited but auth'd Returns True with warning message
Network unreachable Returns None (unknown), not False
Version parsing: pre-release suffix re.findall(r"\d+") handles 1.2.3-beta.4
Version parsing: non-digit segment "1.2a.3" → extracts 1, 2, 3 correctly now
Version below minimum _ver_tuple comparison + upgrade note
--version times out TimeoutExpiredinstalled=False
login status times out TimeoutExpiredlogged_in=None
Binary exists but crashes on --version Non-zero returncode → installed=False
Empty version string ⚠️ Mild gap _parse_semver returns None, _ver_tuple never called — safe but version=None in probe

Wizard Onboarding (flow.py)

Edge case Covered? How
User provides valid path is_runnable_binary + persist + os.environ set
User provides non-existent path is_runnable_binaryis_file() check → rejection
User provides non-executable file is_runnable_binaryos.access(X_OK) on Unix, extension check on Windows
Windows .cmd file is_runnable_binary extension check
Path with spaces/quotes Path handles this; written as-is to .env
Max retries (10) Returns "abort"
Repick from installed-but-not-authed Returns "repick" → outer loop re-prompts
Repick from not-installed Same
Adapter factory is None Early abort with error message
In-process env propagation os.environ[env_key] = path after sync_env_values

Subprocess Environment (runner.py)

Edge case Covered? How
API keys not leaked ANTHROPIC_API_KEY, OPENAI_API_KEY excluded from env + test assertion
CODEX_* forwarded Prefix "CODEX_" in allowlist
Proxy vars forwarded HTTP_PROXY, HTTPS_PROXY, etc. in allowlist
XDG dirs forwarded XDG_CONFIG_HOME, XDG_CACHE_HOME, etc.
Adapter env overrides merged overrides dict applied after base env
Subprocess timeout TimeoutExpiredRuntimeError
Subprocess OSError (binary vanishes) OSErrorRuntimeError
Non-zero exit Delegates to adapter.explain_failure
ANSI escape codes in output Stripped from both stdout and stderr
Probe caching (thread-safe) Double-checked locking with threading.Lock
Future CLI env prefix (e.g. CLAUDE_*) ⚠️ Gap (documented) Must manually add to _SAFE_SUBPROCESS_ENV_PREFIXES — now documented in AGENTS.md

- add diagnose_binary_path() to binary_resolver: distinguishes broken
  symlinks (with target path in message) from missing files and
  non-executable files, returning an actionable string or None
- resolve_cli_binary now logs a WARNING with the diagnostic when an
  explicit *_BIN env var is set but unusable, instead of silently
  falling through to PATH lookup
- wizard flow uses diagnose_binary_path so users see e.g. "'~/bin/codex'
  is a broken symlink (points to '/usr/local/bin/codex'). Remove or fix it."
  instead of the generic "not a valid executable" message
- tests: missing file, broken symlink, valid exe, non-executable,
  and resolver warning on broken symlink
Comment thread tests/integrations/llm_cli/test_codex_adapter.py Fixed
Comment thread tests/integrations/llm_cli/test_codex_adapter.py Fixed
…esolver

- document resolve_cli_binary, diagnose_binary_path, and is_runnable_binary
  at the top of binary_resolver.py with path state -> message reference table
  and platform notes (Windows extension check, lru_cache, readlink fallback)
- move inline `import sys` and `import logging` to top-level in test file
- guard symlink-creation tests against Windows hosts without elevation:
  catch OSError/NotImplementedError and skip rather than fail, preserving
  full coverage on Linux/macOS CI and graceful no-op on restricted runners
Comment thread tests/integrations/llm_cli/test_codex_adapter.py Fixed
- set executable test fixture permissions to 0700
- set non-executable test fixture permissions to 0600
@muddlebee muddlebee changed the title WIP : fix(llm-cli): address post-review issues in CLI integration fix(llm-cli): address post-review issues in CLI integration Apr 30, 2026
Copy link
Copy Markdown
Collaborator

@yashksaini-coder yashksaini-coder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran locally: 21/21 pass, lint clean, no new mypy errors.

The three core fixes are all correct — the os.environ propagation bug in particular was a real problem (without it, every retry in the wizard would re-read the stale env and keep reporting "not found" even after the user entered a valid path). Good catch.


diagnose_binary_path and is_runnable_binary diverge on Windows

is_runnable_binary passes a Windows file when its suffix is .cmd, .exe, .ps1, or .bat. diagnose_binary_path skips the sys.platform != "win32" block entirely, so on Windows it returns None (valid) for any existing file regardless of extension — a .txt file would pass wizard validation and get written to .env, then silently fail at resolve_cli_binary time when is_runnable_binary rejects it.

Easy fix: add the extension check to diagnose_binary_path for Windows:

if sys.platform == "win32":
    if p.suffix.lower() not in {".cmd", ".exe", ".ps1", ".bat"} and not os.access(p, os.X_OK):
        return f"'{path}' is not a recognised executable (expected .cmd, .exe, .ps1, or .bat)."

Low real-world impact since most Windows users will point to a real binary, but the inconsistency between the two functions is a correctness gap worth closing.


Import blocks in codex.py not fully collapsed

The PR description says "consolidated four separate binary_resolver import blocks into one" but there are still three separate from app.integrations.llm_cli.binary_resolver import (...) blocks (lines 10–18). Should be:

from app.integrations.llm_cli.binary_resolver import (
    candidate_binary_names as _candidate_binary_names,
    default_cli_fallback_paths as _default_cli_fallback_paths,
    resolve_cli_binary,
)

No test for the os.environ propagation path

The retry-loop env fix is the most impactful change here, and it has no targeted test. _run_cli_llm_onboarding is interactive but the propagation can be tested in isolation: verify that after sync_env_values + os.environ[env_key] = path, a subsequent os.getenv(env_key) returns the new value in the same process. Not blocking, but worth adding since this was a silent regression.


Aside from these, everything else looks solid — _ver_tuple with re.findall is more robust, the sentinel defaults correctly fix test patchability, the npm timeout bump is pragmatic, and the AGENTS.md updates are genuinely useful for the next contributor adding a CLI integration. The patch target changes in the test file are all correct.

Comment thread app/integrations/llm_cli/binary_resolver.py
Comment thread app/integrations/llm_cli/codex.py
Comment thread app/cli/wizard/flow.py
- add Windows executable extension validation in diagnose_binary_path
- expand binary_resolver docstring path-state table for Windows mismatch case
- harden onboarding path override test with real executable and env cleanup
- assert in-process CODEX_BIN propagation without leaking env across tests
@muddlebee
Copy link
Copy Markdown
Collaborator Author

@yashksaini-coder thank you for the catch. nice

@muddlebee
Copy link
Copy Markdown
Collaborator Author

Import blocks in codex.py not fully collapsed

skipped this one, not an issue imho functional or santiy wise rn. also it opens up new lint errors..

- reformat Windows executable diagnostic branch to satisfy ruff format-check
- add registry mapping LLM_PROVIDER to adapter factory and model env key
- wire _create_llm_client through registry instead of codex-only branch
- derive saved-summary credential line from adapter auth_hint
- drop unused prompt_delivery from LLMCLIAdapter; document prompt via build()
- update AGENTS.md wiring checklist; add registry and summary tests
- defer get_cli_provider_registration import until _create_llm_client runs
- avoid loading llm_cli package __init__ (runner) during llm_client import
@muddlebee muddlebee merged commit 33b478f into Tracer-Cloud:main Apr 30, 2026
10 of 13 checks passed
@github-actions
Copy link
Copy Markdown
Contributor

🎊 Achievement unlocked: PR Merged. @muddlebee passed code review, survived CI, and shipped. Respect. 🤝


👋 Join us on Discord - OpenSRE : hang out, contribute, or hunt for features and issues. Everyone's welcome.

gitsofaryan pushed a commit to gitsofaryan/opensre that referenced this pull request May 3, 2026
…loud#1097)

* fix(llm-cli): address post-review issues in CLI integration

- fix: set os.environ[env_key] after sync_env_values so in-process retry
  loop sees the user-provided binary path immediately (Tracer-Cloud#781 follow-up)
- fix: validate user-supplied binary path (isfile + executable) before
  persisting to .env to prevent silently storing broken paths
- fix: bump npm_prefix_bin_dirs subprocess timeout from 0.3s to 2.0s to
  avoid false-empty cache on slow nvm/volta shim startup
- refactor: use None sentinels for resolve_cli_binary defaults so
  shutil.which and is_runnable_binary are looked up at call time, making
  the function patchable in tests without explicit overrides
- refactor: drop redundant shutil import and explicit which_resolver /
  runnable_check kwargs from CodexAdapter._resolve_binary
- refactor: consolidate four separate binary_resolver import blocks into one
- fix: _ver_tuple now uses re.findall to correctly parse versions with
  non-digit segments (e.g. "1.2a.3", "1.2.3-beta.4")
- feat: log CLI stderr at DEBUG level after successful parse so warnings
  from the CLI are not silently discarded
- test: update patch targets from codex.shutil.which to
  binary_resolver.shutil.which after resolution moved to shared module

* docs(llm-cli): document auth probe pattern and env allowlist in AGENTS.md

- Add three-state logged_in contract table (True/False/None) with wizard behaviour
- Document _classify_*_auth pattern and negative-phrases-first rule
- Warn about _SAFE_SUBPROCESS_ENV_PREFIXES coupling in runner.py for new CLIs
- Update provider checklist with auth probe and env forwarding steps

* fix(llm-cli): use is_runnable_binary for path validation on all platforms

Replace os.path.isfile + os.access(X_OK) with shared is_runnable_binary
from binary_resolver, which already handles Windows extension checks
(.cmd, .exe, .ps1, .bat). os.access(X_OK) returns False for valid
Windows executables that lack Unix-style execute bits.

* fix(llm-cli): diagnose broken symlinks in binary path resolution

- add diagnose_binary_path() to binary_resolver: distinguishes broken
  symlinks (with target path in message) from missing files and
  non-executable files, returning an actionable string or None
- resolve_cli_binary now logs a WARNING with the diagnostic when an
  explicit *_BIN env var is set but unusable, instead of silently
  falling through to PATH lookup
- wizard flow uses diagnose_binary_path so users see e.g. "'~/bin/codex'
  is a broken symlink (points to '/usr/local/bin/codex'). Remove or fix it."
  instead of the generic "not a valid executable" message
- tests: missing file, broken symlink, valid exe, non-executable,
  and resolver warning on broken symlink

* docs(llm-cli): add module docstring with path-state table to binary_resolver

- document resolve_cli_binary, diagnose_binary_path, and is_runnable_binary
  at the top of binary_resolver.py with path state -> message reference table
  and platform notes (Windows extension check, lru_cache, readlink fallback)
- move inline `import sys` and `import logging` to top-level in test file
- guard symlink-creation tests against Windows hosts without elevation:
  catch OSError/NotImplementedError and skip rather than fail, preserving
  full coverage on Linux/macOS CI and graceful no-op on restricted runners

* fix(tests): tighten chmod masks for CodeQL

- set executable test fixture permissions to 0700
- set non-executable test fixture permissions to 0600

* fix(llm-cli): align windows diagnostics and onboarding test

- add Windows executable extension validation in diagnose_binary_path
- expand binary_resolver docstring path-state table for Windows mismatch case
- harden onboarding path override test with real executable and env cleanup
- assert in-process CODEX_BIN propagation without leaking env across tests

* style(llm-cli): format binary_resolver for CI

- reformat Windows executable diagnostic branch to satisfy ruff format-check

* feat(llm-cli): add CLI registry and generic wizard summary

- add registry mapping LLM_PROVIDER to adapter factory and model env key
- wire _create_llm_client through registry instead of codex-only branch
- derive saved-summary credential line from adapter auth_hint
- drop unused prompt_delivery from LLMCLIAdapter; document prompt via build()
- update AGENTS.md wiring checklist; add registry and summary tests

* fix(llm-cli): lazy-load CLI registry to break import cycle

- defer get_cli_provider_registration import until _create_llm_client runs
- avoid loading llm_cli package __init__ (runner) during llm_client import
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants