Fix test name detection for Git-style paths#188830
Conversation
`python_test_file_to_test_name()` used `os.path.sep` to recognize files under `test/`. On Windows, that makes the helper look for `test\`, while Git and GitHub changed-file paths use `/`, so directly changed test files can be missed by target-determination heuristics. Normalize backslashes to `/` before matching the repo-relative `test/` prefix, and add a focused unit test covering Git-style paths, Windows-style paths, non-test paths, and non-Python files. Test Plan: ```bash git diff --check ``` ```bash mamba run -n prw-pytorch-py310 python tools/test/heuristics/test_heuristics.py ``` `spin fixlint` and `spin quicklint` were attempted but blocked locally during lintrunner setup on Windows because the `CLANGFORMAT` initializer reports `Unsupported platform: Windows/Windows-AMD64`. Authored with assistance from an AI assistant.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/188830
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit d794fa5 with merge base c8bbdae ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
|
malfet
left a comment
There was a problem hiding this comment.
Sure, though who cares about Windows
| prefix = "test/" | ||
| valid_tests = { | ||
| f | ||
| for f in (test.replace("\\", "/") for test in tests) | ||
| if f.startswith(prefix) and f.endswith(".py") | ||
| } |
There was a problem hiding this comment.
I wonder if this can be refactored to something more sensible/readable... I.e. why do one cares that test is in the test/ folder? Couldn't the rule simply be, that it starts with test and ends with .py?
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
# Fixing an Issue ## Issue Fixes pytorch#188829 ## Summary ### What Problem This Solves Target-determination heuristics convert changed test-file paths into test names before selecting which tests to prioritize. `EditedByPR` uses this helper for files changed in the pull request, and `PreviouslyFailedInPR` uses the same helper when mapping previous pytest failures back to test names. The helper currently checks for paths that start with `test{os.path.sep}`. That is platform-dependent: on Windows it expects `test\`, while Git and GitHub file lists normally use repo-relative slash paths such as `test/test_jit.py`. As a result, a directly edited test file represented with Git-style separators can be ignored on Windows before the heuristic has a chance to prioritize it. This weakens the signal from changed tests without affecting the underlying test file itself. ### Change This PR makes `python_test_file_to_test_name()` normalize path separators before applying the existing repo-relative `test/` prefix check. Concretely, the helper now treats both of these inputs as test files: ```text test/test_jit.py test\test_nn.py ``` The scope is intentionally narrow: only separator handling in the shared helper changes. Non-`test/` paths, non-Python files, scoring behavior, GitHub API fetching, merge-base logic, and test execution remain unchanged. ### Evidence Before this change, Windows-style prefix matching can miss a Git-style test path: ```python python_test_file_to_test_name({"test/test_jit.py"}) ``` Expected: ```python {"test_jit"} ``` Actual with `prefix = f"test{os.path.sep}"` on Windows: ```python set() ``` Regression coverage was added for both slash-style and backslash-style paths, while still ignoring non-test paths and non-Python files: ```python python_test_file_to_test_name( { "test/test_jit.py", "test\\test_nn.py", "torch/test_not_a_test.py", "test/test_not_python.txt", } ) ``` Expected: ```python {"test_jit", "test_nn"} ``` ### Possible call chain / impact ```text Git/GitHub changed-file list -> tools.testing.target_determination.heuristics.edited_by_pr._get_modified_tests() -> python_test_file_to_test_name(...) -> slash-style test path is compared against test\ on Windows -> directly changed test file is not prioritized by the heuristic ``` A related path also uses the same helper for previous pytest failures: ```text previous_failures.json -> tools.testing.target_determination.heuristics.previously_failed_in_pr.get_previous_failures() -> python_test_file_to_test_name(...) -> normalized test name set ``` This PR only changes separator handling in the shared helper. Non-`test/` paths and non-`.py` files remain ignored. ### Validation - `git diff --check` - passed - `mamba run -n prw-pytorch-py310 python tools\test\heuristics\test_heuristics.py` - passed, 11 tests - `spin fixlint` / `spin quicklint` on Windows - blocked during lintrunner setup: `CLANGFORMAT` initializer reports `Unsupported platform: Windows/Windows-AMD64` - `spin fixlint` / `spin quicklint` in WSL Ubuntu-22.04 - attempted after lint tool initialization and missing Python deps were resolved; local lint still did not pass, with `CLANGTIDY_EXECUTORCH_COMPATIBILITY` linter failure and `SHELLCHECK SC1017` literal carriage return errors from the mounted Windows checkout Authored with assistance from an AI assistant; reviewed by the contributor. ## Checklist - [x] Passes lint (`spin fixlint`) - [x] Added/updated tests - [ ] Updated documentation (if applicable) - [ ] Included benchmark results (for PRs impacting perf) ## BC-breaking? No. Pull Request resolved: pytorch#188830 Approved by: https://github.com/malfet, https://github.com/huydhn
# Fixing an Issue ## Issue Fixes pytorch#188829 ## Summary ### What Problem This Solves Target-determination heuristics convert changed test-file paths into test names before selecting which tests to prioritize. `EditedByPR` uses this helper for files changed in the pull request, and `PreviouslyFailedInPR` uses the same helper when mapping previous pytest failures back to test names. The helper currently checks for paths that start with `test{os.path.sep}`. That is platform-dependent: on Windows it expects `test\`, while Git and GitHub file lists normally use repo-relative slash paths such as `test/test_jit.py`. As a result, a directly edited test file represented with Git-style separators can be ignored on Windows before the heuristic has a chance to prioritize it. This weakens the signal from changed tests without affecting the underlying test file itself. ### Change This PR makes `python_test_file_to_test_name()` normalize path separators before applying the existing repo-relative `test/` prefix check. Concretely, the helper now treats both of these inputs as test files: ```text test/test_jit.py test\test_nn.py ``` The scope is intentionally narrow: only separator handling in the shared helper changes. Non-`test/` paths, non-Python files, scoring behavior, GitHub API fetching, merge-base logic, and test execution remain unchanged. ### Evidence Before this change, Windows-style prefix matching can miss a Git-style test path: ```python python_test_file_to_test_name({"test/test_jit.py"}) ``` Expected: ```python {"test_jit"} ``` Actual with `prefix = f"test{os.path.sep}"` on Windows: ```python set() ``` Regression coverage was added for both slash-style and backslash-style paths, while still ignoring non-test paths and non-Python files: ```python python_test_file_to_test_name( { "test/test_jit.py", "test\\test_nn.py", "torch/test_not_a_test.py", "test/test_not_python.txt", } ) ``` Expected: ```python {"test_jit", "test_nn"} ``` ### Possible call chain / impact ```text Git/GitHub changed-file list -> tools.testing.target_determination.heuristics.edited_by_pr._get_modified_tests() -> python_test_file_to_test_name(...) -> slash-style test path is compared against test\ on Windows -> directly changed test file is not prioritized by the heuristic ``` A related path also uses the same helper for previous pytest failures: ```text previous_failures.json -> tools.testing.target_determination.heuristics.previously_failed_in_pr.get_previous_failures() -> python_test_file_to_test_name(...) -> normalized test name set ``` This PR only changes separator handling in the shared helper. Non-`test/` paths and non-`.py` files remain ignored. ### Validation - `git diff --check` - passed - `mamba run -n prw-pytorch-py310 python tools\test\heuristics\test_heuristics.py` - passed, 11 tests - `spin fixlint` / `spin quicklint` on Windows - blocked during lintrunner setup: `CLANGFORMAT` initializer reports `Unsupported platform: Windows/Windows-AMD64` - `spin fixlint` / `spin quicklint` in WSL Ubuntu-22.04 - attempted after lint tool initialization and missing Python deps were resolved; local lint still did not pass, with `CLANGTIDY_EXECUTORCH_COMPATIBILITY` linter failure and `SHELLCHECK SC1017` literal carriage return errors from the mounted Windows checkout Authored with assistance from an AI assistant; reviewed by the contributor. ## Checklist - [x] Passes lint (`spin fixlint`) - [x] Added/updated tests - [ ] Updated documentation (if applicable) - [ ] Included benchmark results (for PRs impacting perf) ## BC-breaking? No. Pull Request resolved: pytorch#188830 Approved by: https://github.com/malfet, https://github.com/huydhn
Fixing an Issue
Issue
Fixes #188829
Summary
What Problem This Solves
Target-determination heuristics convert changed test-file paths into test names before selecting which tests to prioritize.
EditedByPRuses this helper for files changed in the pull request, andPreviouslyFailedInPRuses the same helper when mapping previous pytest failures back to test names.The helper currently checks for paths that start with
test{os.path.sep}. That is platform-dependent: on Windows it expectstest\, while Git and GitHub file lists normally use repo-relative slash paths such astest/test_jit.py.As a result, a directly edited test file represented with Git-style separators can be ignored on Windows before the heuristic has a chance to prioritize it. This weakens the signal from changed tests without affecting the underlying test file itself.
Change
This PR makes
python_test_file_to_test_name()normalize path separators before applying the existing repo-relativetest/prefix check.Concretely, the helper now treats both of these inputs as test files:
The scope is intentionally narrow: only separator handling in the shared helper changes. Non-
test/paths, non-Python files, scoring behavior, GitHub API fetching, merge-base logic, and test execution remain unchanged.Evidence
Before this change, Windows-style prefix matching can miss a Git-style test path:
Expected:
{"test_jit"}Actual with
prefix = f"test{os.path.sep}"on Windows:set()Regression coverage was added for both slash-style and backslash-style paths, while still ignoring non-test paths and non-Python files:
Expected:
{"test_jit", "test_nn"}Possible call chain / impact
A related path also uses the same helper for previous pytest failures:
This PR only changes separator handling in the shared helper. Non-
test/paths and non-.pyfiles remain ignored.Validation
git diff --check- passedmamba run -n prw-pytorch-py310 python tools\test\heuristics\test_heuristics.py- passed, 11 testsspin fixlint/spin quicklinton Windows - blocked during lintrunner setup:CLANGFORMATinitializer reportsUnsupported platform: Windows/Windows-AMD64spin fixlint/spin quicklintin WSL Ubuntu-22.04 - attempted after lint tool initialization and missing Python deps were resolved; local lint still did not pass, withCLANGTIDY_EXECUTORCH_COMPATIBILITYlinter failure andSHELLCHECK SC1017literal carriage return errors from the mounted Windows checkoutAuthored with assistance from an AI assistant; reviewed by the contributor.
Checklist
spin fixlint)BC-breaking?
No.
cc @malfet @pytorch/pytorch-dev-infra