Skip to content

Fix test name detection for Git-style paths#188830

Closed
VectorPeak wants to merge 1 commit into
pytorch:mainfrom
VectorPeak:codex/git-style-test-paths
Closed

Fix test name detection for Git-style paths#188830
VectorPeak wants to merge 1 commit into
pytorch:mainfrom
VectorPeak:codex/git-style-test-paths

Conversation

@VectorPeak

@VectorPeak VectorPeak commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Fixing an Issue

Issue

Fixes #188829

Summary

What Problem This Solves

Target-determination heuristics convert changed test-file paths into test names before selecting which tests to prioritize. EditedByPR uses this helper for files changed in the pull request, and PreviouslyFailedInPR uses the same helper when mapping previous pytest failures back to test names.

The helper currently checks for paths that start with test{os.path.sep}. That is platform-dependent: on Windows it expects test\, while Git and GitHub file lists normally use repo-relative slash paths such as test/test_jit.py.

As a result, a directly edited test file represented with Git-style separators can be ignored on Windows before the heuristic has a chance to prioritize it. This weakens the signal from changed tests without affecting the underlying test file itself.

Change

This PR makes python_test_file_to_test_name() normalize path separators before applying the existing repo-relative test/ prefix check.

Concretely, the helper now treats both of these inputs as test files:

test/test_jit.py
test\test_nn.py

The scope is intentionally narrow: only separator handling in the shared helper changes. Non-test/ paths, non-Python files, scoring behavior, GitHub API fetching, merge-base logic, and test execution remain unchanged.

Evidence

Before this change, Windows-style prefix matching can miss a Git-style test path:

python_test_file_to_test_name({"test/test_jit.py"})

Expected:

{"test_jit"}

Actual with prefix = f"test{os.path.sep}" on Windows:

set()

Regression coverage was added for both slash-style and backslash-style paths, while still ignoring non-test paths and non-Python files:

python_test_file_to_test_name(
    {
        "test/test_jit.py",
        "test\\test_nn.py",
        "torch/test_not_a_test.py",
        "test/test_not_python.txt",
    }
)

Expected:

{"test_jit", "test_nn"}

Possible call chain / impact

Git/GitHub changed-file list
  -> tools.testing.target_determination.heuristics.edited_by_pr._get_modified_tests()
  -> python_test_file_to_test_name(...)
  -> slash-style test path is compared against test\ on Windows
  -> directly changed test file is not prioritized by the heuristic

A related path also uses the same helper for previous pytest failures:

previous_failures.json
  -> tools.testing.target_determination.heuristics.previously_failed_in_pr.get_previous_failures()
  -> python_test_file_to_test_name(...)
  -> normalized test name set

This PR only changes separator handling in the shared helper. Non-test/ paths and non-.py files remain ignored.

Validation

  • git diff --check - passed
  • mamba run -n prw-pytorch-py310 python tools\test\heuristics\test_heuristics.py - passed, 11 tests
  • spin fixlint / spin quicklint on Windows - blocked during lintrunner setup: CLANGFORMAT initializer reports Unsupported platform: Windows/Windows-AMD64
  • spin fixlint / spin quicklint in WSL Ubuntu-22.04 - attempted after lint tool initialization and missing Python deps were resolved; local lint still did not pass, with CLANGTIDY_EXECUTORCH_COMPATIBILITY linter failure and SHELLCHECK SC1017 literal carriage return errors from the mounted Windows checkout

Authored with assistance from an AI assistant; reviewed by the contributor.

Checklist

  • Passes lint (spin fixlint)
  • Added/updated tests
  • Updated documentation (if applicable)
  • Included benchmark results (for PRs impacting perf)

BC-breaking?

No.

cc @malfet @pytorch/pytorch-dev-infra

`python_test_file_to_test_name()` used `os.path.sep` to recognize files under `test/`. On Windows, that makes the helper look for `test\`, while Git and GitHub changed-file paths use `/`, so directly changed test files can be missed by target-determination heuristics.

Normalize backslashes to `/` before matching the repo-relative `test/` prefix, and add a focused unit test covering Git-style paths, Windows-style paths, non-test paths, and non-Python files.

Test Plan:

```bash
git diff --check
```

```bash
mamba run -n prw-pytorch-py310 python tools/test/heuristics/test_heuristics.py
```

`spin fixlint` and `spin quicklint` were attempted but blocked locally during lintrunner setup on Windows because the `CLANGFORMAT` initializer reports `Unsupported platform: Windows/Windows-AMD64`.

Authored with assistance from an AI assistant.
@pytorch-bot

pytorch-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/188830

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d794fa5 with merge base c8bbdae (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot Bot added the topic: not user facing topic category label Jul 2, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 2, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: VectorPeak / name: VectorPeak (d794fa5)

@VectorPeak
VectorPeak marked this pull request as ready for review July 2, 2026 17:22
@malfet malfet added module: ci Related to continuous integration triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jul 2, 2026

@malfet malfet left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, though who cares about Windows

Comment on lines +92 to +97
prefix = "test/"
valid_tests = {
f
for f in (test.replace("\\", "/") for test in tests)
if f.startswith(prefix) and f.endswith(".py")
}

@malfet malfet Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this can be refactored to something more sensible/readable... I.e. why do one cares that test is in the test/ folder? Couldn't the rule simply be, that it starts with test and ends with .py?

@huydhn huydhn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@huydhn

huydhn commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@pytorchbot merge

@pytorch-bot pytorch-bot Bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 2, 2026
@pytorchmergebot

Copy link
Copy Markdown
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

vishalgoyal316 pushed a commit to vishalgoyal316/pytorch that referenced this pull request Jul 16, 2026
# Fixing an Issue

## Issue

Fixes pytorch#188829

## Summary

### What Problem This Solves

Target-determination heuristics convert changed test-file paths into test names before selecting which tests to prioritize. `EditedByPR` uses this helper for files changed in the pull request, and `PreviouslyFailedInPR` uses the same helper when mapping previous pytest failures back to test names.

The helper currently checks for paths that start with `test{os.path.sep}`. That is platform-dependent: on Windows it expects `test\`, while Git and GitHub file lists normally use repo-relative slash paths such as `test/test_jit.py`.

As a result, a directly edited test file represented with Git-style separators can be ignored on Windows before the heuristic has a chance to prioritize it. This weakens the signal from changed tests without affecting the underlying test file itself.

### Change

This PR makes `python_test_file_to_test_name()` normalize path separators before applying the existing repo-relative `test/` prefix check.

Concretely, the helper now treats both of these inputs as test files:

```text
test/test_jit.py
test\test_nn.py
```

The scope is intentionally narrow: only separator handling in the shared helper changes. Non-`test/` paths, non-Python files, scoring behavior, GitHub API fetching, merge-base logic, and test execution remain unchanged.

### Evidence

Before this change, Windows-style prefix matching can miss a Git-style test path:

```python
python_test_file_to_test_name({"test/test_jit.py"})
```

Expected:

```python
{"test_jit"}
```

Actual with `prefix = f"test{os.path.sep}"` on Windows:

```python
set()
```

Regression coverage was added for both slash-style and backslash-style paths, while still ignoring non-test paths and non-Python files:

```python
python_test_file_to_test_name(
    {
        "test/test_jit.py",
        "test\\test_nn.py",
        "torch/test_not_a_test.py",
        "test/test_not_python.txt",
    }
)
```

Expected:

```python
{"test_jit", "test_nn"}
```

### Possible call chain / impact

```text
Git/GitHub changed-file list
  -> tools.testing.target_determination.heuristics.edited_by_pr._get_modified_tests()
  -> python_test_file_to_test_name(...)
  -> slash-style test path is compared against test\ on Windows
  -> directly changed test file is not prioritized by the heuristic
```

A related path also uses the same helper for previous pytest failures:

```text
previous_failures.json
  -> tools.testing.target_determination.heuristics.previously_failed_in_pr.get_previous_failures()
  -> python_test_file_to_test_name(...)
  -> normalized test name set
```

This PR only changes separator handling in the shared helper. Non-`test/` paths and non-`.py` files remain ignored.

### Validation

- `git diff --check` - passed
- `mamba run -n prw-pytorch-py310 python tools\test\heuristics\test_heuristics.py` - passed, 11 tests
- `spin fixlint` / `spin quicklint` on Windows - blocked during lintrunner setup: `CLANGFORMAT` initializer reports `Unsupported platform: Windows/Windows-AMD64`
- `spin fixlint` / `spin quicklint` in WSL Ubuntu-22.04 - attempted after lint tool initialization and missing Python deps were resolved; local lint still did not pass, with `CLANGTIDY_EXECUTORCH_COMPATIBILITY` linter failure and `SHELLCHECK SC1017` literal carriage return errors from the mounted Windows checkout

Authored with assistance from an AI assistant; reviewed by the contributor.

## Checklist

- [x] Passes lint (`spin fixlint`)
- [x] Added/updated tests
- [ ] Updated documentation (if applicable)
- [ ] Included benchmark results (for PRs impacting perf)

## BC-breaking?

No.

Pull Request resolved: pytorch#188830
Approved by: https://github.com/malfet, https://github.com/huydhn
DamJanusz pushed a commit to DamJanusz/pytorch that referenced this pull request Jul 21, 2026
# Fixing an Issue

## Issue

Fixes pytorch#188829

## Summary

### What Problem This Solves

Target-determination heuristics convert changed test-file paths into test names before selecting which tests to prioritize. `EditedByPR` uses this helper for files changed in the pull request, and `PreviouslyFailedInPR` uses the same helper when mapping previous pytest failures back to test names.

The helper currently checks for paths that start with `test{os.path.sep}`. That is platform-dependent: on Windows it expects `test\`, while Git and GitHub file lists normally use repo-relative slash paths such as `test/test_jit.py`.

As a result, a directly edited test file represented with Git-style separators can be ignored on Windows before the heuristic has a chance to prioritize it. This weakens the signal from changed tests without affecting the underlying test file itself.

### Change

This PR makes `python_test_file_to_test_name()` normalize path separators before applying the existing repo-relative `test/` prefix check.

Concretely, the helper now treats both of these inputs as test files:

```text
test/test_jit.py
test\test_nn.py
```

The scope is intentionally narrow: only separator handling in the shared helper changes. Non-`test/` paths, non-Python files, scoring behavior, GitHub API fetching, merge-base logic, and test execution remain unchanged.

### Evidence

Before this change, Windows-style prefix matching can miss a Git-style test path:

```python
python_test_file_to_test_name({"test/test_jit.py"})
```

Expected:

```python
{"test_jit"}
```

Actual with `prefix = f"test{os.path.sep}"` on Windows:

```python
set()
```

Regression coverage was added for both slash-style and backslash-style paths, while still ignoring non-test paths and non-Python files:

```python
python_test_file_to_test_name(
    {
        "test/test_jit.py",
        "test\\test_nn.py",
        "torch/test_not_a_test.py",
        "test/test_not_python.txt",
    }
)
```

Expected:

```python
{"test_jit", "test_nn"}
```

### Possible call chain / impact

```text
Git/GitHub changed-file list
  -> tools.testing.target_determination.heuristics.edited_by_pr._get_modified_tests()
  -> python_test_file_to_test_name(...)
  -> slash-style test path is compared against test\ on Windows
  -> directly changed test file is not prioritized by the heuristic
```

A related path also uses the same helper for previous pytest failures:

```text
previous_failures.json
  -> tools.testing.target_determination.heuristics.previously_failed_in_pr.get_previous_failures()
  -> python_test_file_to_test_name(...)
  -> normalized test name set
```

This PR only changes separator handling in the shared helper. Non-`test/` paths and non-`.py` files remain ignored.

### Validation

- `git diff --check` - passed
- `mamba run -n prw-pytorch-py310 python tools\test\heuristics\test_heuristics.py` - passed, 11 tests
- `spin fixlint` / `spin quicklint` on Windows - blocked during lintrunner setup: `CLANGFORMAT` initializer reports `Unsupported platform: Windows/Windows-AMD64`
- `spin fixlint` / `spin quicklint` in WSL Ubuntu-22.04 - attempted after lint tool initialization and missing Python deps were resolved; local lint still did not pass, with `CLANGTIDY_EXECUTORCH_COMPATIBILITY` linter failure and `SHELLCHECK SC1017` literal carriage return errors from the mounted Windows checkout

Authored with assistance from an AI assistant; reviewed by the contributor.

## Checklist

- [x] Passes lint (`spin fixlint`)
- [x] Added/updated tests
- [ ] Updated documentation (if applicable)
- [ ] Included benchmark results (for PRs impacting perf)

## BC-breaking?

No.

Pull Request resolved: pytorch#188830
Approved by: https://github.com/malfet, https://github.com/huydhn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged module: ci Related to continuous integration open source topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows target determination can miss Git-style test paths

5 participants