fix(llm-cli): address post-review issues in CLI integration#1097
fix(llm-cli): address post-review issues in CLI integration#1097muddlebee merged 10 commits intoTracer-Cloud:mainfrom
Conversation
- fix: set os.environ[env_key] after sync_env_values so in-process retry loop sees the user-provided binary path immediately (Tracer-Cloud#781 follow-up) - fix: validate user-supplied binary path (isfile + executable) before persisting to .env to prevent silently storing broken paths - fix: bump npm_prefix_bin_dirs subprocess timeout from 0.3s to 2.0s to avoid false-empty cache on slow nvm/volta shim startup - refactor: use None sentinels for resolve_cli_binary defaults so shutil.which and is_runnable_binary are looked up at call time, making the function patchable in tests without explicit overrides - refactor: drop redundant shutil import and explicit which_resolver / runnable_check kwargs from CodexAdapter._resolve_binary - refactor: consolidate four separate binary_resolver import blocks into one - fix: _ver_tuple now uses re.findall to correctly parse versions with non-digit segments (e.g. "1.2a.3", "1.2.3-beta.4") - feat: log CLI stderr at DEBUG level after successful parse so warnings from the CLI are not silently discarded - test: update patch targets from codex.shutil.which to binary_resolver.shutil.which after resolution moved to shared module
…S.md - Add three-state logged_in contract table (True/False/None) with wizard behaviour - Document _classify_*_auth pattern and negative-phrases-first rule - Warn about _SAFE_SUBPROCESS_ENV_PREFIXES coupling in runner.py for new CLIs - Update provider checklist with auth probe and env forwarding steps
Greptile SummaryThis follow-up PR addresses several post-review issues in the CLI LLM integration: the retry loop now correctly propagates a user-entered binary path into the in-process environment, Confidence Score: 5/5Safe to merge; all findings are P2 style/consistency suggestions with no blocking correctness issues. The core bug fixes (env propagation, _ver_tuple, sentinel defaults) are all correct. Remaining comments are a Windows-only edge case for executable validation (low real-world impact) and an incomplete import consolidation. No P0 or P1 issues found. app/cli/wizard/flow.py — Windows executable check uses os.access instead of is_runnable_binary; app/integrations/llm_cli/codex.py — imports could be further consolidated into one block Important Files Changed
Sequence DiagramsequenceDiagram
participant W as Wizard (flow.py)
participant A as CodexAdapter
participant R as resolve_cli_binary
participant E as os.environ / .env
loop Up to 10 attempts
W->>A: adapter.detect()
A->>R: resolve_cli_binary(explicit_env_key, ...)
R->>E: os.getenv(explicit_env_key)
E-->>R: saved path or empty string
alt env path set and is_runnable_binary passes
R-->>A: binary path
else shutil.which finds binary on PATH
R-->>A: binary path
else fallback paths scanned
R-->>A: binary path or None
end
A-->>W: CLIProbe(installed, logged_in)
alt installed and logged_in is True
W-->>W: return ok
else installed but not logged in
W->>W: prompt → retry or repick
else not installed
W->>W: prompt → retry / enter path / repick
opt user enters path
W->>W: os.path.isfile check
W->>W: os.access X_OK check
W->>E: sync_env_values to .env
W->>E: os.environ update in-process
note right of W: next detect() sees the path immediately
end
end
end
W-->>W: return abort (max retries exceeded)
|
…orms Replace os.path.isfile + os.access(X_OK) with shared is_runnable_binary from binary_resolver, which already handles Windows extension checks (.cmd, .exe, .ps1, .bat). os.access(X_OK) returns False for valid Windows executables that lack Unix-style execute bits.
AuditBinary Resolution (
|
| Scenario | Covered? | How |
|---|---|---|
| Linux — PATH lookup | ✅ | shutil.which("codex") + fallback paths |
macOS — Homebrew /opt/homebrew/bin |
✅ | darwin-only fallback path |
Windows — .cmd/.exe/.ps1/.bat extensions |
✅ | candidate_binary_names + is_runnable_binary extension check |
Windows — APPDATA/npm, LOCALAPPDATA/Programs |
✅ | default_cli_fallback_paths win32 branch |
Windows — npm prefix uses root (no /bin) |
✅ | npm_prefix_bin_dirs win32 branch + test |
| nvm/volta/pnpm install dirs | ✅ | .volta/bin, PNPM_HOME, XDG_DATA_HOME/pnpm, npm prefix |
CODEX_BIN set but invalid |
✅ | Falls through to PATH/fallbacks + test |
CODEX_BIN set and valid |
✅ | Short-circuits + test |
CODEX_BIN empty/blank |
✅ | .strip() makes it falsy, falls through |
| npm not installed | ✅ | OSError caught, returns () |
| npm hangs/slow shim | ✅ | 2.0s timeout + TimeoutExpired caught |
| Duplicate fallback paths | ✅ | Deduplication via seen set |
| Spaces in path | ✅ | Path objects handle this; expanduser for ~ |
npm_config_prefix case-insensitive env |
✅ | Explicit .lower() scan of os.environ |
| Symlinks | ✅ | added checks diagnose_binary_path( ) |
Tests: Linux, macOS, Windows fallback paths all have dedicated tests. npm prefix has win32/unix tests.
Auth Probe (codex.py)
| Edge case | Covered? | How |
|---|---|---|
| "Not logged in" vs "logged in" substring | ✅ | Negative phrases checked first |
| Exit 0 but "Not logged in" text | ✅ | Dedicated test test_detect_not_logged_in_exit_zero |
| Expired/invalid token | ✅ | _classify_codex_auth checks both |
| Rate-limited but auth'd | ✅ | Returns True with warning message |
| Network unreachable | ✅ | Returns None (unknown), not False |
| Version parsing: pre-release suffix | ✅ | re.findall(r"\d+") handles 1.2.3-beta.4 |
| Version parsing: non-digit segment | ✅ | "1.2a.3" → extracts 1, 2, 3 correctly now |
| Version below minimum | ✅ | _ver_tuple comparison + upgrade note |
--version times out |
✅ | TimeoutExpired → installed=False |
login status times out |
✅ | TimeoutExpired → logged_in=None |
Binary exists but crashes on --version |
✅ | Non-zero returncode → installed=False |
| Empty version string | _parse_semver returns None, _ver_tuple never called — safe but version=None in probe |
Wizard Onboarding (flow.py)
| Edge case | Covered? | How |
|---|---|---|
| User provides valid path | ✅ | is_runnable_binary + persist + os.environ set |
| User provides non-existent path | ✅ | is_runnable_binary → is_file() check → rejection |
| User provides non-executable file | ✅ | is_runnable_binary → os.access(X_OK) on Unix, extension check on Windows |
Windows .cmd file |
✅ | is_runnable_binary extension check |
| Path with spaces/quotes | ✅ | Path handles this; written as-is to .env |
| Max retries (10) | ✅ | Returns "abort" |
| Repick from installed-but-not-authed | ✅ | Returns "repick" → outer loop re-prompts |
| Repick from not-installed | ✅ | Same |
| Adapter factory is None | ✅ | Early abort with error message |
| In-process env propagation | ✅ | os.environ[env_key] = path after sync_env_values |
Subprocess Environment (runner.py)
| Edge case | Covered? | How |
|---|---|---|
| API keys not leaked | ✅ | ANTHROPIC_API_KEY, OPENAI_API_KEY excluded from env + test assertion |
CODEX_* forwarded |
✅ | Prefix "CODEX_" in allowlist |
| Proxy vars forwarded | ✅ | HTTP_PROXY, HTTPS_PROXY, etc. in allowlist |
| XDG dirs forwarded | ✅ | XDG_CONFIG_HOME, XDG_CACHE_HOME, etc. |
| Adapter env overrides merged | ✅ | overrides dict applied after base env |
| Subprocess timeout | ✅ | TimeoutExpired → RuntimeError |
| Subprocess OSError (binary vanishes) | ✅ | OSError → RuntimeError |
| Non-zero exit | ✅ | Delegates to adapter.explain_failure |
| ANSI escape codes in output | ✅ | Stripped from both stdout and stderr |
| Probe caching (thread-safe) | ✅ | Double-checked locking with threading.Lock |
Future CLI env prefix (e.g. CLAUDE_*) |
Must manually add to _SAFE_SUBPROCESS_ENV_PREFIXES — now documented in AGENTS.md |
- add diagnose_binary_path() to binary_resolver: distinguishes broken symlinks (with target path in message) from missing files and non-executable files, returning an actionable string or None - resolve_cli_binary now logs a WARNING with the diagnostic when an explicit *_BIN env var is set but unusable, instead of silently falling through to PATH lookup - wizard flow uses diagnose_binary_path so users see e.g. "'~/bin/codex' is a broken symlink (points to '/usr/local/bin/codex'). Remove or fix it." instead of the generic "not a valid executable" message - tests: missing file, broken symlink, valid exe, non-executable, and resolver warning on broken symlink
…esolver - document resolve_cli_binary, diagnose_binary_path, and is_runnable_binary at the top of binary_resolver.py with path state -> message reference table and platform notes (Windows extension check, lru_cache, readlink fallback) - move inline `import sys` and `import logging` to top-level in test file - guard symlink-creation tests against Windows hosts without elevation: catch OSError/NotImplementedError and skip rather than fail, preserving full coverage on Linux/macOS CI and graceful no-op on restricted runners
- set executable test fixture permissions to 0700 - set non-executable test fixture permissions to 0600
yashksaini-coder
left a comment
There was a problem hiding this comment.
Ran locally: 21/21 pass, lint clean, no new mypy errors.
The three core fixes are all correct — the os.environ propagation bug in particular was a real problem (without it, every retry in the wizard would re-read the stale env and keep reporting "not found" even after the user entered a valid path). Good catch.
diagnose_binary_path and is_runnable_binary diverge on Windows
is_runnable_binary passes a Windows file when its suffix is .cmd, .exe, .ps1, or .bat. diagnose_binary_path skips the sys.platform != "win32" block entirely, so on Windows it returns None (valid) for any existing file regardless of extension — a .txt file would pass wizard validation and get written to .env, then silently fail at resolve_cli_binary time when is_runnable_binary rejects it.
Easy fix: add the extension check to diagnose_binary_path for Windows:
if sys.platform == "win32":
if p.suffix.lower() not in {".cmd", ".exe", ".ps1", ".bat"} and not os.access(p, os.X_OK):
return f"'{path}' is not a recognised executable (expected .cmd, .exe, .ps1, or .bat)."Low real-world impact since most Windows users will point to a real binary, but the inconsistency between the two functions is a correctness gap worth closing.
Import blocks in codex.py not fully collapsed
The PR description says "consolidated four separate binary_resolver import blocks into one" but there are still three separate from app.integrations.llm_cli.binary_resolver import (...) blocks (lines 10–18). Should be:
from app.integrations.llm_cli.binary_resolver import (
candidate_binary_names as _candidate_binary_names,
default_cli_fallback_paths as _default_cli_fallback_paths,
resolve_cli_binary,
)No test for the os.environ propagation path
The retry-loop env fix is the most impactful change here, and it has no targeted test. _run_cli_llm_onboarding is interactive but the propagation can be tested in isolation: verify that after sync_env_values + os.environ[env_key] = path, a subsequent os.getenv(env_key) returns the new value in the same process. Not blocking, but worth adding since this was a silent regression.
Aside from these, everything else looks solid — _ver_tuple with re.findall is more robust, the sentinel defaults correctly fix test patchability, the npm timeout bump is pragmatic, and the AGENTS.md updates are genuinely useful for the next contributor adding a CLI integration. The patch target changes in the test file are all correct.
- add Windows executable extension validation in diagnose_binary_path - expand binary_resolver docstring path-state table for Windows mismatch case - harden onboarding path override test with real executable and env cleanup - assert in-process CODEX_BIN propagation without leaking env across tests
|
@yashksaini-coder thank you for the catch. nice |
skipped this one, not an issue imho functional or santiy wise rn. also it opens up new lint errors.. |
- reformat Windows executable diagnostic branch to satisfy ruff format-check
- add registry mapping LLM_PROVIDER to adapter factory and model env key - wire _create_llm_client through registry instead of codex-only branch - derive saved-summary credential line from adapter auth_hint - drop unused prompt_delivery from LLMCLIAdapter; document prompt via build() - update AGENTS.md wiring checklist; add registry and summary tests
- defer get_cli_provider_registration import until _create_llm_client runs - avoid loading llm_cli package __init__ (runner) during llm_client import
|
🎊 Achievement unlocked: PR Merged. @muddlebee passed code review, survived CI, and shipped. Respect. 🤝 👋 Join us on Discord - OpenSRE : hang out, contribute, or hunt for features and issues. Everyone's welcome. |
…loud#1097) * fix(llm-cli): address post-review issues in CLI integration - fix: set os.environ[env_key] after sync_env_values so in-process retry loop sees the user-provided binary path immediately (Tracer-Cloud#781 follow-up) - fix: validate user-supplied binary path (isfile + executable) before persisting to .env to prevent silently storing broken paths - fix: bump npm_prefix_bin_dirs subprocess timeout from 0.3s to 2.0s to avoid false-empty cache on slow nvm/volta shim startup - refactor: use None sentinels for resolve_cli_binary defaults so shutil.which and is_runnable_binary are looked up at call time, making the function patchable in tests without explicit overrides - refactor: drop redundant shutil import and explicit which_resolver / runnable_check kwargs from CodexAdapter._resolve_binary - refactor: consolidate four separate binary_resolver import blocks into one - fix: _ver_tuple now uses re.findall to correctly parse versions with non-digit segments (e.g. "1.2a.3", "1.2.3-beta.4") - feat: log CLI stderr at DEBUG level after successful parse so warnings from the CLI are not silently discarded - test: update patch targets from codex.shutil.which to binary_resolver.shutil.which after resolution moved to shared module * docs(llm-cli): document auth probe pattern and env allowlist in AGENTS.md - Add three-state logged_in contract table (True/False/None) with wizard behaviour - Document _classify_*_auth pattern and negative-phrases-first rule - Warn about _SAFE_SUBPROCESS_ENV_PREFIXES coupling in runner.py for new CLIs - Update provider checklist with auth probe and env forwarding steps * fix(llm-cli): use is_runnable_binary for path validation on all platforms Replace os.path.isfile + os.access(X_OK) with shared is_runnable_binary from binary_resolver, which already handles Windows extension checks (.cmd, .exe, .ps1, .bat). os.access(X_OK) returns False for valid Windows executables that lack Unix-style execute bits. * fix(llm-cli): diagnose broken symlinks in binary path resolution - add diagnose_binary_path() to binary_resolver: distinguishes broken symlinks (with target path in message) from missing files and non-executable files, returning an actionable string or None - resolve_cli_binary now logs a WARNING with the diagnostic when an explicit *_BIN env var is set but unusable, instead of silently falling through to PATH lookup - wizard flow uses diagnose_binary_path so users see e.g. "'~/bin/codex' is a broken symlink (points to '/usr/local/bin/codex'). Remove or fix it." instead of the generic "not a valid executable" message - tests: missing file, broken symlink, valid exe, non-executable, and resolver warning on broken symlink * docs(llm-cli): add module docstring with path-state table to binary_resolver - document resolve_cli_binary, diagnose_binary_path, and is_runnable_binary at the top of binary_resolver.py with path state -> message reference table and platform notes (Windows extension check, lru_cache, readlink fallback) - move inline `import sys` and `import logging` to top-level in test file - guard symlink-creation tests against Windows hosts without elevation: catch OSError/NotImplementedError and skip rather than fail, preserving full coverage on Linux/macOS CI and graceful no-op on restricted runners * fix(tests): tighten chmod masks for CodeQL - set executable test fixture permissions to 0700 - set non-executable test fixture permissions to 0600 * fix(llm-cli): align windows diagnostics and onboarding test - add Windows executable extension validation in diagnose_binary_path - expand binary_resolver docstring path-state table for Windows mismatch case - harden onboarding path override test with real executable and env cleanup - assert in-process CODEX_BIN propagation without leaking env across tests * style(llm-cli): format binary_resolver for CI - reformat Windows executable diagnostic branch to satisfy ruff format-check * feat(llm-cli): add CLI registry and generic wizard summary - add registry mapping LLM_PROVIDER to adapter factory and model env key - wire _create_llm_client through registry instead of codex-only branch - derive saved-summary credential line from adapter auth_hint - drop unused prompt_delivery from LLMCLIAdapter; document prompt via build() - update AGENTS.md wiring checklist; add registry and summary tests * fix(llm-cli): lazy-load CLI registry to break import cycle - defer get_cli_provider_registration import until _create_llm_client runs - avoid loading llm_cli package __init__ (runner) during llm_client import

Follows up on #781 and #795 with fixes surfaced during code review.
Summary
_run_cli_llm_onboardingretry loop now setsos.environ[env_key]after writing to.envso the in-process binary resolver immediately sees the user-provided path — without this, the retry always re-detected "not found" even after the user entered a valid pathisfile+ executable check) before being persisted to.envnpm_prefix_bin_dirssubprocess timeout from 0.3 s → 2.0 s to avoid false-empty@lru_cacheon slow nvm/volta shim startupresolve_cli_binarydefaults forwhich_resolver/runnable_checkchanged from bound-at-definition-time function objects toNonesentinels with call-time lookup — this makespatch("binary_resolver.shutil.which")work without callers passing explicit overridesshutilimport and explicit default kwargs fromCodexAdapter._resolve_binary; consolidated four separatebinary_resolverimport blocks into one_ver_tupleswitched tore.findallso segments like"1.2a.3"or pre-release suffixes ("1.2.3-beta.4") parse correctlystderris now logged atDEBUGafter a successful parse so warnings (e.g. deprecation notices from codex) are visible in debug outputshutil.whichpatch targets fromcodex.shutil.which→binary_resolver.shutil.whichto match where resolution now livesTest plan
make lintpassesmake format-checkpassesmake typecheckpassesmake test-covpasses (4149 passed locally)tests/integrations/llm_cli/— all 16 tests pass