Skip to content

bug: adk eval misparses Windows absolute eval-set paths as case selectors #6415

Closed

Description

@sgoel2be24-cyber

Describe the bug

On Windows, adk eval incorrectly parses absolute eval-set file paths.

The CLI supports optionally selecting individual cases using:

<eval-set path or ID>:<comma-separated case IDs>

The parser treats every colon as the case-selector separator, so the colon in a Windows drive prefix such as C:\\ is mistaken for that separator.

For example, C:\tmp\agent\eval.evalset.json is parsed approximately as:

{"C": [r"\tmp\agent\eval.evalset.json"]}

instead of:

{r"C:\tmp\agent\eval.evalset.json": []}

The problem also prevents case selectors from working with Windows absolute paths.

Steps to reproduce

uv run pytest -q -p no:cacheprovider `
  tests/unittests/cli/utils/test_cli_tools_click.py::test_cli_eval_with_eval_set_file_path

The existing regression fails because the CLI writes no eval result:

assert len(eval_set_results) == 1
E assert 0 == 1

Direct parser demonstration:

uv run python -c "from google.adk.cli.cli_eval import parse_and_get_evals_to_run as parse; inputs=[r'C:\tmp\agent\eval.evalset.json',r'C:\tmp\agent\eval.evalset.json:case1',r'C:\tmp\agent\eval.evalset.json:case1,case2']; [print(repr(value), '->', repr(parse([value]))) for value in inputs]"

Actual result

'C:\\tmp\\agent\\eval.evalset.json'
-> {'C': ['\\tmp\\agent\\eval.evalset.json']}

'C:\\tmp\\agent\\eval.evalset.json:case1'
-> {'C': ['\\tmp\\agent\\eval.evalset.json']}

'C:\\tmp\\agent\\eval.evalset.json:case1,case2'
-> {'C': ['\\tmp\\agent\\eval.evalset.json']}

The drive letter becomes the eval-set identifier and requested case IDs are lost.

Expected result

{r"C:\tmp\agent\eval.evalset.json": []}
{r"C:\tmp\agent\eval.evalset.json": ["case1"]}
{r"C:\tmp\agent\eval.evalset.json": ["case1", "case2"]}

Existing behavior should remain unchanged:

{"/tmp/agent/eval.evalset.json": ["case1", "case2"]}
{"my_eval_set": ["case1", "case2"]}

Environment

  • OS: Windows 10, build 19045
  • Python: 3.11.15
  • uv: 0.11.29
  • Local google-adk: 2.4.0
  • Latest upstream main
  • Commit: 4a84d8a459853b039d3774da27e00b57b4d51aa5

No model, credentials, or external service is involved.

Root cause

parse_and_get_evals_to_run() uses unrestricted splitting:

eval_set = input_eval_set.split(":")[0]
evals = input_eval_set.split(":")[1].split(",")

The first colon in a Windows path is the drive-letter separator. When a selector colon is also present, that selector is beyond index 1 and is ignored.

Proposed narrow solution

Make parsing aware of a leading Windows drive prefix:

  1. Preserve the drive-designator colon.
  2. Split only at a subsequent selector delimiter.
  3. Preserve the full drive-qualified path when no selector is present.
  4. Keep POSIX-path and eval-set-ID behavior unchanged.

The solution should not depend solely on the file already existing, so nonexistent paths retain useful file-not-found diagnostics.

Regression-test plan

Add platform-independent parser tests for Windows absolute paths without selectors, with one selector, and with multiple selectors, plus POSIX paths and eval-set IDs with selectors. Use Windows-style strings even on Linux so existing CI covers the regression. Retain the existing CLI-level test shown above.

Conflict check

I searched open and closed issues and pull requests for the parser name, Windows absolute eval-set paths, adk eval Windows paths, drive-letter colon parsing, and the focused test name. I found no issue or PR addressing this root cause.

PR #6166 is a separate Windows wildcard-expansion change and explicitly notes that this eval test fails independently on unmodified main. Issue #2602 concerns pytest-driven result persistence and has a different root cause.

Contribution

I am willing to contribute a small, focused fix with these regression tests after maintainer approval.

Activity

  1. added
    eval[Component] This issue is related to evaluation
    on Jul 17, 2026
  2. surajksharma07 commented on Jul 17, 2026

    @surajksharma07
    Collaborator

    @sgoel2be24-cyber

    We tried a narrow fix in parse_and_get_evals_to_run: check for a leading drive prefix (letter + : + \ or /) and only look for the selector colon after that.
    That handles all five cases you listed while leaving POSIX paths and eval-set IDs untouched.

    Could you try that same approach locally and confirm it holds for you ?

  3. sgoel2be24-cyber commented on Jul 17, 2026

    @sgoel2be24-cyber
    ContributorAuthor

    Thanks, @surajksharma07 — I tested the suggested approach locally on Windows against upstream commit fd006db9.

    I implemented the proposed narrow approach: detect a leading alphabetic drive prefix followed by :\ or :/, then search for the selector delimiter only after that prefix.

    The results hold for me:

    • 9 direct parser regression cases passed
    • the previously failing test_cli_eval_with_eval_set_file_path now passes
    • the complete parser test module passes: 11 tests
    • the adjacent CLI test module passes: 42 tests

    The verified cases include:

    • Windows backslash paths with and without selectors
    • Windows forward-slash paths
    • lowercase drive letters
    • one and multiple selectors
    • POSIX paths
    • eval-set IDs

    Existing blank-selector filtering, multiple-input handling, and duplicate merging remain unchanged. The implementation does not depend on the file existing.

    One local tooling caveat: two repository pre-commit hooks use POSIX executable/shebang paths that native-Windows pre-commit could not launch. Their underlying checks passed when run directly through Git Bash and uv; Ruff, isort, Pyink, whitespace checks, end-of-file checks, and git diff --check all passed.

    I currently have the focused implementation and tests on a local branch, uncommitted and unpushed. I can open a PR if this implementation direction looks good.

  4. added a commit that references this issue on Jul 20, 2026
    6f6106f
  5. added a commit that references this issue on Aug 11, 2026
    5a06476
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

eval[Component] This issue is related to evaluationrequest clarification[Status] The maintainer need clarification or more information from the author

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions