Repository navigation
Description
Describe the bug
On Windows, adk eval incorrectly parses absolute eval-set file paths.
The CLI supports optionally selecting individual cases using:
<eval-set path or ID>:<comma-separated case IDs>
The parser treats every colon as the case-selector separator, so the colon in a Windows drive prefix such as C:\\ is mistaken for that separator.
For example, C:\tmp\agent\eval.evalset.json is parsed approximately as:
{"C": [r"\tmp\agent\eval.evalset.json"]}instead of:
{r"C:\tmp\agent\eval.evalset.json": []}The problem also prevents case selectors from working with Windows absolute paths.
Steps to reproduce
uv run pytest -q -p no:cacheprovider `
tests/unittests/cli/utils/test_cli_tools_click.py::test_cli_eval_with_eval_set_file_pathThe existing regression fails because the CLI writes no eval result:
assert len(eval_set_results) == 1
E assert 0 == 1
Direct parser demonstration:
uv run python -c "from google.adk.cli.cli_eval import parse_and_get_evals_to_run as parse; inputs=[r'C:\tmp\agent\eval.evalset.json',r'C:\tmp\agent\eval.evalset.json:case1',r'C:\tmp\agent\eval.evalset.json:case1,case2']; [print(repr(value), '->', repr(parse([value]))) for value in inputs]"Actual result
'C:\\tmp\\agent\\eval.evalset.json'
-> {'C': ['\\tmp\\agent\\eval.evalset.json']}
'C:\\tmp\\agent\\eval.evalset.json:case1'
-> {'C': ['\\tmp\\agent\\eval.evalset.json']}
'C:\\tmp\\agent\\eval.evalset.json:case1,case2'
-> {'C': ['\\tmp\\agent\\eval.evalset.json']}
The drive letter becomes the eval-set identifier and requested case IDs are lost.
Expected result
{r"C:\tmp\agent\eval.evalset.json": []}
{r"C:\tmp\agent\eval.evalset.json": ["case1"]}
{r"C:\tmp\agent\eval.evalset.json": ["case1", "case2"]}Existing behavior should remain unchanged:
{"/tmp/agent/eval.evalset.json": ["case1", "case2"]}
{"my_eval_set": ["case1", "case2"]}Environment
- OS: Windows 10, build 19045
- Python: 3.11.15
- uv: 0.11.29
- Local
google-adk: 2.4.0 - Latest upstream
main - Commit:
4a84d8a459853b039d3774da27e00b57b4d51aa5
No model, credentials, or external service is involved.
Root cause
parse_and_get_evals_to_run() uses unrestricted splitting:
eval_set = input_eval_set.split(":")[0]
evals = input_eval_set.split(":")[1].split(",")The first colon in a Windows path is the drive-letter separator. When a selector colon is also present, that selector is beyond index 1 and is ignored.
Proposed narrow solution
Make parsing aware of a leading Windows drive prefix:
- Preserve the drive-designator colon.
- Split only at a subsequent selector delimiter.
- Preserve the full drive-qualified path when no selector is present.
- Keep POSIX-path and eval-set-ID behavior unchanged.
The solution should not depend solely on the file already existing, so nonexistent paths retain useful file-not-found diagnostics.
Regression-test plan
Add platform-independent parser tests for Windows absolute paths without selectors, with one selector, and with multiple selectors, plus POSIX paths and eval-set IDs with selectors. Use Windows-style strings even on Linux so existing CI covers the regression. Retain the existing CLI-level test shown above.
Conflict check
I searched open and closed issues and pull requests for the parser name, Windows absolute eval-set paths, adk eval Windows paths, drive-letter colon parsing, and the focused test name. I found no issue or PR addressing this root cause.
PR #6166 is a separate Windows wildcard-expansion change and explicitly notes that this eval test fails independently on unmodified main. Issue #2602 concerns pytest-driven result persistence and has a different root cause.
Contribution
I am willing to contribute a small, focused fix with these regression tests after maintainer approval.
Activity
surajksharma07 commentedon Jul 17, 2026
@sgoel2be24-cyber
We tried a narrow fix in parse_and_get_evals_to_run: check for a leading drive prefix (letter + : + \ or /) and only look for the selector colon after that.
That handles all five cases you listed while leaving POSIX paths and eval-set IDs untouched.
Could you try that same approach locally and confirm it holds for you ?
sgoel2be24-cyber commentedon Jul 17, 2026
Thanks, @surajksharma07 — I tested the suggested approach locally on Windows against upstream commit
fd006db9.I implemented the proposed narrow approach: detect a leading alphabetic drive prefix followed by
:\or:/, then search for the selector delimiter only after that prefix.The results hold for me:
test_cli_eval_with_eval_set_file_pathnow passesThe verified cases include:
Existing blank-selector filtering, multiple-input handling, and duplicate merging remain unchanged. The implementation does not depend on the file existing.
One local tooling caveat: two repository pre-commit hooks use POSIX executable/shebang paths that native-Windows pre-commit could not launch. Their underlying checks passed when run directly through Git Bash and
uv; Ruff, isort, Pyink, whitespace checks, end-of-file checks, andgit diff --checkall passed.I currently have the focused implementation and tests on a local branch, uncommitted and unpushed. I can open a PR if this implementation direction looks good.
fix: handle Windows paths in `adk eval`
fix: handle Windows paths in `adk eval`