Build --subprocess pytest test ids from public fields by ezyang · Pull Request #202406 · pytorch/pytorch

ezyang · 2026-10-28T15:48:17Z

Human Note

The comment explains it

Agent Report

get_pytest_test_cases builds the test ids that --subprocess --use-pytest
hands to each child (--pytest-single-test <id>). A node id's path part is
relative to pytest's rootdir (the repo root, where pytest.ini lives), while
children run from the parent's cwd (test/ under run_test.py), so it called
config.cwd_relative_nodeid to rebase the path. pytest 9.2 changed that method
to take a NodeId object, so collection hit an INTERNALERROR whenever cwd
differs from the rootdir.

The id is now item.path made relative to config.invocation_params.dir, plus
the Class::name part of the node id: the same string as before, from public
API. (A relative path, not an absolute one, because pytest splits arguments at
the first [, so an absolute path breaks under a parent directory like
ws[1].) When the test file and the cwd are on different Windows drives,
os.path.relpath raises ValueError; the id then falls back to the absolute
path, which is what cwd_relative_nodeid (via bestrelpath) produced there.

TestGetPytestTestCases (test/torchci/test_testing.py) calls
get_pytest_test_cases in-process: test_ids_resolve_from_cwd checks that the
returned id names the test and that its path resolves from the cwd (it fails on
pytest 9.2 without this change), and
test_ids_fall_back_to_absolute_path_across_drives makes relpath raise for
the test file and checks for the absolute-path id. The class skips unless the
file runs as __main__, because --use-main-module collects from there.

cd test && CUDA_VISIBLE_DEVICES= python torchci/test_testing.py -k TestGetPytestTestCases

Test plan: a file with a passing, a failing and a parametrized test (one value
containing a space) run with --subprocess --use-pytest under pytest 7.3.2,
9.1.1 and 9.2.0, from test/, from the repo root, by absolute path from /tmp,
via python -m, and from inside a ws[1]/test directory: every combination
reports exactly the one failing test, with no "not found" or INTERNALERROR.
Before, pytest 9.2.0 hit an INTERNALERROR whenever cwd differs from the rootdir.

cd <dir> && PYTHONPATH=<pytest==$V overlay> python <script> --subprocess --use-pytest

Authored with an AI assistant (Claude).

Agent Worklog

2026-10-28: B2 split from B (id part only). Verified 3 pytest versions x 4 launch forms.
review: absolute path breaks under '[' dirs -> switched to relpath(item.path, invocation_params.dir); re-verified incl. ws[1].
2026-10-29: addressed automated review B2 (in-process TestGetPytestTestCases.test_ids_resolve_from_cwd). B1 (Windows cross-drive relpath fallback) deliberately not done per Edward. Not pushed.
2026-10-30: per Edward, did the Windows cross-drive fallback after all (to clear the automated review): relpath ValueError -> str(item.path), plus test_ids_fall_back_to_absolute_path_across_drives (patches os.path.relpath to raise for the test file; errors without the fix). Rebased onto origin/main 99003a3 (the 14 failing builds were the stale build-external-packages@main action). Fail --subprocess pytest runs when test collection fails #202411's job (eff5ee) rebased onto the new branch (trivial test-file adjacency conflict, own diff unchanged). Not pushed.

This PR was generated by ptq with human review.

pytorch-bot · 2026-10-28T15:48:28Z

PR Status: ready for maintainer review

Waiting for a maintainer with merge rights for the changed paths to review this PR. Please address any feedback they leave.

✅ AUTOMATED REVIEW - Ready for human review:

The PR builds `--subprocess --use-pytest` child test ids from
`os.path.relpath(item.path, config.invocation_params.dir)` plus the node id's
`Class::name` part, replacing `config.cwd_relative_nodeid`, which pytest 9.2
broke. If the file and cwd are on different Windows drives, the id falls back to
the absolute path. Recommendation: Approve. The remaining points are all
non-blocking test improvements, though the review recommends adding the
`os.path.isabs` assertion before landing.

Findings (4):

[minor] test/torchci/test_testing.py:616
Non-blocking (continuing from the earlier review): nothing checks the
relative-path behaviour. The only path assertion is
`os.path.samefile(ids[0].split("::", 1)[0], __file__)`, which also passes for an
absolute path. If `common_utils.py:1439-1442` were reverted to `path =
str(item.path)`, both new tests would still pass. The relative path is the whole
reason for this design: pytest splits arguments at the first `[`, so ids under a
directory like `ws[1]` break when absolute. Fix: `path = ids[0].split("::",
1)[0]`, then `self.assertFalse(os.path.isabs(path), ids)`. I recommend adding
this one-line assertion before landing.

[minor] test/torchci/test_testing.py:610
Non-blocking: the tests don't use the same input as production (here and at
:627). Both tests pass `__file__` as the collection argument, and for a script
run as `__main__` that is absolute on Python 3.9+. Production passes the
relative `argv[0]` (`test/run_test.py:687`  `common_utils.py:1242`  `:1568`). In
a checkout whose path contains `[`, the tests' own inner collection would hit
the splitting problem this PR describes, even though production works there.
Passing `sys.argv[0]` (or `os.path.relpath(__file__)`) would match production.

[minor] test/torchci/test_testing.py:621
Non-blocking: the fake `relpath` affects the whole process.
`unittest.mock.patch("os.path.relpath", ...)` replaces `relpath` for every
caller while the nested `pytest.main` runs, not just for
`get_pytest_test_cases`. The fake calls `os.path.samefile(path, __file__)` on
every call. Real `relpath` handles a nonexistent path fine, but `samefile`
raises `FileNotFoundError` on one. No caller in the repo hits this today;
pytest's internals weren't checked. Comparing `os.path.abspath(path) ==
os.path.abspath(__file__)` would avoid the stat. So would moving the id
construction into a small helper and testing that directly, which would also
remove the nested `pytest.main`.

[minor] test/torchci/test_testing.py:604
Non-blocking: the new class skips none of the configs that the file's other
pytest-driving classes skip. The classes at lines 166-172, 440-444, 522-528 and
659-665 carry `@ skipIfTorchDynamo`, `skipIf(TEST_WITH_CROSSREF)` and
`skipIf(TEST_CUDA or TEST_WITH_ROCM)`, but `TestGetPytestTestCases` has only the
`__name__` skip. So it runs on the Dynamo-wrapped, crossref and GPU shards
(`.ci/pytorch/test.sh:768`, `:276-277`). That costs a little time but causes no
known failure. Leaving out the Windows skip makes sense, since the class tests
the Windows fallback; the other skips are worth matching for consistency.

Reviewed commit: b6237d8

Review run

🐛 Report a bad review

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/202406

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b6237d8 with merge base 99003a3 ():
💚 Looks good so far! There are no failures yet. 💚

🟡 GREEN LIGHT - PR requires human review:

Triviality
- Rewrites how get_pytest_test_cases in torch/testing/_internal/common_utils.py builds ids, replacing pytest's cwd_relative_nodeid with hand-built relpath(item.path, invocation_params.dir)
- That shared runner code feeds every run_tests --subprocess --use-pytest child via --pytest-single-test, so it fits no trivial class
Review questions
- Whether item.path always matches rootdir plus the nodeid path part (symlinks, files outside rootdir) is a judgment pytest previously owned
- TestGetPytestTestCases runs a nested pytest.main inside the test process, with os.path.relpath patched globally during collection
Test coverage
- CI pins pytest==7.3.2 in .ci/docker/requirements-ci.txt, so the pytest 9.2 failure the PR fixes is not exercised in CI
- The cross-drive test does exercise the new ValueError fallback branch

reason: not_trivial
Reviewed commit: b6237d8

Inference job

🐛 Report a wrong verdict

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ezyang · 2026-10-28T22:20:19Z

mergedog handoff

Note

This PR is being shepherded by mergedog, an automation tool run by a maintainer. It watches CI and may push fixes; the PR author does not need to act unless a maintainer asks.

All CI is green (or skipped). Ready for human review and @pytorchbot merge.

Current PR head: 957f776de5ee7795d96f4b5234f46721fa2a3bb5.
No mergedog-authored commits were pushed during this run.

No LLM was invoked during this run.

`get_pytest_test_cases` builds the test ids that `--subprocess --use-pytest` hands to each child (`--pytest-single-test <id>`). A node id's path part is relative to pytest's rootdir (the repo root, where pytest.ini lives), while children run from the parent's cwd (`test/` under run_test.py), so it called `config.cwd_relative_nodeid` to rebase the path. pytest 9.2 changed that method to take a `NodeId` object, so collection hit an INTERNALERROR whenever cwd differs from the rootdir. The id is now `item.path` made relative to `config.invocation_params.dir`, plus the `Class::name` part of the node id: the same string as before, from public API. (A relative path, not an absolute one, because pytest splits arguments at the first `[`, so an absolute path breaks under a parent directory like `ws[1]`.) Test plan: a file with a passing, a failing and a parametrized test (one value containing a space) run with `--subprocess --use-pytest` under pytest 7.3.2, 9.1.1 and 9.2.0, from `test/`, from the repo root, by absolute path from /tmp, via `python -m`, and from inside a `ws[1]/test` directory: every combination reports exactly the one failing test, with no "not found" or INTERNALERROR. Before, pytest 9.2.0 hit an INTERNALERROR whenever cwd differs from the rootdir. ``` cd <dir> && PYTHONPATH=<pytest==$V overlay> python <script> --subprocess --use-pytest ``` Authored with an AI assistant (Claude).

Address the automated review: add TestGetPytestTestCases.test_ids_resolve_from_cwd, which calls get_pytest_test_cases in-process and checks that the returned id names the test and that its path resolves from the cwd (node ids are relative to the rootdir instead). On pytest 9.2 it fails without the id change. It skips unless the file runs as __main__, because --use-main-module collects from there. When the test file and the cwd are on different Windows drives, relpath raises ValueError; fall back to the absolute path there, as pytest's own cwd_relative_nodeid (via bestrelpath) did. test_ids_fall_back_to_absolute_path_across_drives covers it by making relpath raise for the test file. ``` cd test && CUDA_VISIBLE_DEVICES= python torchci/test_testing.py -k TestGetPytestTestCases cd test && CUDA_VISIBLE_DEVICES= python torchci/test_testing.py --use-pytest -k TestGetPytestTestCases ``` Run under pytest 7.3.2 and 9.2.0, from test/ and from the repo root; about 1.5s. Authored with an AI assistant (Claude).

## Human Note I tried getting rid of the custom class entirely but Claude persuaded me it was useful. So we're just going to subclass from pytest so we can track upstream more easily. ## Agent Report ### Problem With pytest 9.2.0, any test run that writes `--junit-xml-reruns` (every CI test shard) aborts with an INTERNALERROR at the first test's teardown: ``` File "test/conftest.py", line 254, in pytest_runtest_logreport super().pytest_runtest_logreport(report) File "_pytest/junitxml.py", line 544, in finalize reporter = self.node_reporters.pop((node_id, workernode)) KeyError: (NodeId(path='test/test_pytree.py', names=('TestGenericPytree', 'test_aligned_public_apis'), params=None), None) ``` ### Root cause `LogXMLReruns.node_reporter` was a copy of pytest's `LogXML.node_reporter`. The copy only differed in building a `_NodeReporterReruns` and stored reporters under `(report.nodeid, workernode)`, with `nodeid` as a string. pytest 9.2 keys `node_reporters` by a new `NodeId` dataclass (`report.id`), and `LogXML.finalize` pops with that key. A `NodeId` never equals a string, so the pop fails. ### Fix Call `super().node_reporter(report)` so pytest builds the key and the reporter in whatever form that version uses, check that `_NodeReporterReruns` subclasses the returned type, then set the reporter's `__class__` to `_NodeReporterReruns`. The class docstring now says it may only override methods, not add state or change how instance data is represented. The subclass only overrides output methods (`_prepare_content`, `_write_content`, `append_skipped`) and adds no state, so switching the class in place is safe, and it removes the copy of pytest internals that went stale. I also added a docstring to `_NodeReporterReruns` explaining why it exists. pytest.ini sets `junit_logging_reruns = all`, and with the stock `_NodeReporter` every testcase, including passing tests that print nothing, gets a headed `<system-out>` and `<system-err>` element (skipped tests get both twice). The subclass writes captured output raw and omits empty streams. Regenerating the goldens without the subclass shows exactly that difference. The other option, branching on the pytest version to build the right key type, would keep the copy and break again the next time pytest changes the key. This also reverts #202385, which pinned `pytest==9.1.1` in the inductor-benchmarks images as a stopgap for this crash (TorchBench's requirements pull in an unpinned pytest). The other pytest 9.2 fixes still open shouldn't affect those images: #202406 (`--subprocess` test ids) only matters for the few test modules that `run_test.py` runs with `--subprocess`, which are CUDA-init and distributed tests the inductor configs don't run, and #202412 only restores the teardown part of the RERUNS terminal summary. ### Test `TestJunitXml` (test/test_testing.py) used to skip on every pytest version except the goldens' (the CI pins), so no test could see this crash. `setUpClass` now records that skip reason instead of raising and still runs the fixture suite; only the exact golden comparisons skip on other versions. The tag counts move out of `test_pytest_outcome_shapes` into `test_pytest_outcome_counts`, which runs on every version and also checks for INTERNALERROR. On pytest 9.2.0 the counts test passes with this change and fails without it; on 7.3.2 (the CI pin) all three tests pass. #### Where the versions come from Running the fixture suite on every version means `setUpClass` now depends on the environment in ways the old blanket skip hid, so the version checks have to describe the process that actually runs the suite. That process is a child (`sys.executable -m pytest`) with `PYTHONPATH` removed, so the parent's view can be wrong. Asking the parent with `importlib.metadata` would mean: - With pytest only on the parent's `PYTHONPATH`, the parent sees pytest, the child doesn't, writes no XML, and `setUpClass` raises instead of skipping (the case the automated review hit, where pytest is missing entirely, is the simplest instance). - With a different pytest on the parent's `PYTHONPATH`, the golden header check compares the goldens against one version while the child produces XML with another, and the golden tests fail. So `setUpClass` first runs a small probe, `_junit_child_versions()`, in the child's interpreter and environment: it prints the pytest and pytest-rerunfailures versions (or null) as JSON via `importlib.metadata`, without importing torch. That one result decides all three things: the class skips when pytest is missing, the counts test skips when pytest-rerunfailures is missing, and the golden header is built from it. The probe costs about 60 ms once per class, next to roughly 3.5 s for the fixture suite run the class already does. The alternatives were matching `No module named pytest` in the fixture run's stderr (covers only the missing-pytest case and leaves the golden mismatch) or parsing the `plugins:` line from pytest's console header (fragile, and only saves those 60 ms). The child also drops `PYTEST_PLUGINS` and `PYTEST_DISABLE_PLUGIN_AUTOLOAD`, as it already drops `PYTEST_ADDOPTS`, so it loads the installed plugins as CI does; with autoload off, pytest-rerunfailures was importable but never loaded and the counts were wrong. Checked in throwaway environments that really have each version (a `PYTHONPATH` overlay only changes the parent): | Child environment | Result | |---|---| | no pytest | class skipped | | pytest 9.1.1, no pytest-rerunfailures | all three skip | | pytest 9.2.0 + pytest-rerunfailures 14.0 | counts pass, goldens skip; counts fail with the old conftest | | pytest 7.3.2 + pytest-rerunfailures 14.0 | all three pass | | job venv with pytest 7.3.2 on the parent's `PYTHONPATH`, or `PYTEST_DISABLE_PLUGIN_AUTOLOAD=1` | counts pass, goldens skip | ``` cd test && CUDA_VISIBLE_DEVICES= python test_testing.py -k TestJunitXml ``` ### Validation A minimal test file covering pass, skip, a flaky test that passes on rerun, a consistent failure with reruns, and an expected failure, run with `--reruns=2 --junit-xml-reruns`, serially and with `-n 2`, under pytest 7.3.2 (requirements-ci pin), 9.1.1 and 9.2.0. pytest-rerunfailures 14.0 and pytest-xdist 3.3.1 are the CI versions. In all six configurations the testcase/rerun/skipped/failure elements of the rerun XML match those the unmodified conftest produces under 9.1.1. Unmodified conftest under 9.2.0 crashes with the KeyError above. ``` PYTHONPATH=<overlay with pytest==$V pytest-rerunfailures==14.0 pytest-xdist==3.3.1> python -m pytest -q -p no:cacheprovider test_mini.py --reruns=2 [-n 2] --junit-xml-reruns=out/r.xml ``` Real test file with each pytest version: 251 passed and 251 testcases in the XML, no INTERNALERROR. The same command with the old conftest on 9.2.0 crashes after the first test. ``` cd test && PYTHONPATH=<overlay pytest==$V> python -m pytest -q -p no:cacheprovider test_pytree.py --reruns=2 -x --junit-xml-reruns=/tmp/rr/r.xml ``` The existing golden fixture `test/junit_xml_testdata/pytest_suite.py` (13 outcome shapes, normalized the same way `TestJunitXml` in `test/test_testing.py` does): - pytest 7.3.2: the new conftest output matches the committed golden `expected/pytest.xml`, which was recorded with pytest 7.3.2. - pytest 9.1.1: the new conftest output is identical to the old conftest's. - pytest 9.2.0: the new conftest output is identical to the old conftest's output under 9.1.1. The old conftest crashes with 81 INTERNALERRORs. The change was reviewed by two fresh agents (correctness and quality) before this update; their findings are addressed above. `lintrunner test/conftest.py` is clean. <details> <summary>Agent Worklog</summary> # Worklog - 2026-10-28: Diffed pytest 9.1.1 and 9.2.0 `_pytest/junitxml.py`. `node_reporters` is now keyed by a `NodeId` dataclass (`report.id`); `finalize` pops `(report.id, workernode)`. conftest's copied `node_reporter` used string nodeids, so it fails at the first teardown, not only on reruns. - Reproduced with a minimal test file under overlays of pytest 7.3.2/9.1.1/9.2.0 on the job venv's torch. - Fix: delegate to `super().node_reporter` and set `__class__ = _NodeReporterReruns`. The XML matches the old behavior under all three versions, serial and with xdist. `test_pytree.py` passes under all three. lintrunner is clean. - Golden fixture junit_xml_testdata/pytest_suite.py: new==golden on 7.3.2, new==old on 9.1.1, new on 9.2.0 == old on 9.1.1. - 2026-10-29: (superseded) C1 test change moved to its own PR, see below. - 2026-10-29: split the C1 test change into job 20261029-pytorch-adhoc-7dcc5b (new PR below this one); this branch now sits on ptq/20261029-pytorch-adhoc-7dcc5b. Counts test passes on 9.2.0 with this fix. Not pushed. - 2026-10-29: now on the refactor 7879242; this PR moves the version skip onto the goldens and runs the counts (plus an INTERNALERROR check) on every version. Commit d334d9d. 9.2.0: counts pass with the fix, fail without. - 2026-10-29: shrank the test change: setUpClass unchanged except it records the version-mismatch reason (golden_skip) instead of raising, and keeps proc output; goldens skip on it; counts add an INTERNALERROR check. Commit 980d40eddfd. 7.3.2 all pass; 9.2.0 counts pass (fail without the conftest fix), goldens skip. - 2026-10-29: consolidated (Edward OK): #202597 closed, this PR retargeted to main, single commit de4b8db pushed, body Test section updated. - 2026-10-29: automated review flagged the pytest==9.1.1 pin from #202385 in install_inductor_benchmark_deps.sh as stale. Rebased onto origin/main 92264e2 and removed the pin (exact revert of #202385). TestJunitXml: 9.1.1 venv and 9.2.0 overlay pass (goldens skip); the 7.3.2 golden diffs I saw were my overlay run missing the repo-root pytest.ini; the PR-review agent confirmed old and new conftest match the goldens on 7.3.2 run in place. Not pushed. - 2026-10-30: Edward approved; pushed 1d11f93 via git (ssh). PR body not yet regenerated: gh on devgpu013 is logged out (~/.config/gh/hosts.yml emptied 2026-10-30 06:20). Body regenerated with ptq pr on 2026-10-30 after gh re-auth. - 2026-10-30: automated review of 1d11f93: without pytest installed, TestJunitXml errored (subprocess ran, no XML) instead of skipping. setUpClass now raises SkipTest first when pytest is missing. Verified with _junit_provenance patched to "pytest missing" (class skipped, subprocess not run); 9.1.1 venv and 9.2.0 overlay unchanged; lint clean. - 2026-10-30: per Edward, replaced the provenance-string checks: setUpClass skips when the child stderr has "No module named pytest" (no XML), and the counts test probes `import pytest_rerunfailures` with the child env. Note: earlier PYTHONPATH-overlay runs of TestJunitXml only changed the parent; the child drops PYTHONPATH and used the venv pytest 9.1.1. Re-verified with symlinked throwaway venvs: 9.2.0 counts pass (fail with origin/main conftest), 7.3.2 all pass, no pytest -> class skipped, no rerunfailures -> all skip. - 2026-10-30: replaced the stderr match and import probe with one child probe (_junit_child_versions, JSON of pytest/rerunfailures versions under _JUNIT_CHILD_ENV): it drives the class skip, has_rerunfailures, and the golden provenance (fixes parent/child mismatch under PYTHONPATH). Child env also drops PYTEST_PLUGINS/PYTEST_DISABLE_PLUGIN_AUTOLOAD. Verified: no pytest -> class skip; no rerunfailures -> 3 skips; 9.2.0 counts pass (fail with origin/main conftest); 7.3.2 all pass; job venv with PYTHONPATH=ov-7.3.2 or PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 -> OK. </details> --- *This PR was generated by [ptq](https://github.com/drisspg/pt_job_queue) with human review.* Pull Request resolved: #202390 Approved by: https://github.com/nWEIdia, https://github.com/atalman, https://github.com/huydhn, https://github.com/jeanschmidt

…h#202390) ## Human Note I tried getting rid of the custom class entirely but Claude persuaded me it was useful. So we're just going to subclass from pytest so we can track upstream more easily. ## Agent Report ### Problem With pytest 9.2.0, any test run that writes `--junit-xml-reruns` (every CI test shard) aborts with an INTERNALERROR at the first test's teardown: ``` File "test/conftest.py", line 254, in pytest_runtest_logreport super().pytest_runtest_logreport(report) File "_pytest/junitxml.py", line 544, in finalize reporter = self.node_reporters.pop((node_id, workernode)) KeyError: (NodeId(path='test/test_pytree.py', names=('TestGenericPytree', 'test_aligned_public_apis'), params=None), None) ``` ### Root cause `LogXMLReruns.node_reporter` was a copy of pytest's `LogXML.node_reporter`. The copy only differed in building a `_NodeReporterReruns` and stored reporters under `(report.nodeid, workernode)`, with `nodeid` as a string. pytest 9.2 keys `node_reporters` by a new `NodeId` dataclass (`report.id`), and `LogXML.finalize` pops with that key. A `NodeId` never equals a string, so the pop fails. ### Fix Call `super().node_reporter(report)` so pytest builds the key and the reporter in whatever form that version uses, check that `_NodeReporterReruns` subclasses the returned type, then set the reporter's `__class__` to `_NodeReporterReruns`. The class docstring now says it may only override methods, not add state or change how instance data is represented. The subclass only overrides output methods (`_prepare_content`, `_write_content`, `append_skipped`) and adds no state, so switching the class in place is safe, and it removes the copy of pytest internals that went stale. I also added a docstring to `_NodeReporterReruns` explaining why it exists. pytest.ini sets `junit_logging_reruns = all`, and with the stock `_NodeReporter` every testcase, including passing tests that print nothing, gets a headed `<system-out>` and `<system-err>` element (skipped tests get both twice). The subclass writes captured output raw and omits empty streams. Regenerating the goldens without the subclass shows exactly that difference. The other option, branching on the pytest version to build the right key type, would keep the copy and break again the next time pytest changes the key. This also reverts pytorch#202385, which pinned `pytest==9.1.1` in the inductor-benchmarks images as a stopgap for this crash (TorchBench's requirements pull in an unpinned pytest). The other pytest 9.2 fixes still open shouldn't affect those images: pytorch#202406 (`--subprocess` test ids) only matters for the few test modules that `run_test.py` runs with `--subprocess`, which are CUDA-init and distributed tests the inductor configs don't run, and pytorch#202412 only restores the teardown part of the RERUNS terminal summary. ### Test `TestJunitXml` (test/test_testing.py) used to skip on every pytest version except the goldens' (the CI pins), so no test could see this crash. `setUpClass` now records that skip reason instead of raising and still runs the fixture suite; only the exact golden comparisons skip on other versions. The tag counts move out of `test_pytest_outcome_shapes` into `test_pytest_outcome_counts`, which runs on every version and also checks for INTERNALERROR. On pytest 9.2.0 the counts test passes with this change and fails without it; on 7.3.2 (the CI pin) all three tests pass. #### Where the versions come from Running the fixture suite on every version means `setUpClass` now depends on the environment in ways the old blanket skip hid, so the version checks have to describe the process that actually runs the suite. That process is a child (`sys.executable -m pytest`) with `PYTHONPATH` removed, so the parent's view can be wrong. Asking the parent with `importlib.metadata` would mean: - With pytest only on the parent's `PYTHONPATH`, the parent sees pytest, the child doesn't, writes no XML, and `setUpClass` raises instead of skipping (the case the automated review hit, where pytest is missing entirely, is the simplest instance). - With a different pytest on the parent's `PYTHONPATH`, the golden header check compares the goldens against one version while the child produces XML with another, and the golden tests fail. So `setUpClass` first runs a small probe, `_junit_child_versions()`, in the child's interpreter and environment: it prints the pytest and pytest-rerunfailures versions (or null) as JSON via `importlib.metadata`, without importing torch. That one result decides all three things: the class skips when pytest is missing, the counts test skips when pytest-rerunfailures is missing, and the golden header is built from it. The probe costs about 60 ms once per class, next to roughly 3.5 s for the fixture suite run the class already does. The alternatives were matching `No module named pytest` in the fixture run's stderr (covers only the missing-pytest case and leaves the golden mismatch) or parsing the `plugins:` line from pytest's console header (fragile, and only saves those 60 ms). The child also drops `PYTEST_PLUGINS` and `PYTEST_DISABLE_PLUGIN_AUTOLOAD`, as it already drops `PYTEST_ADDOPTS`, so it loads the installed plugins as CI does; with autoload off, pytest-rerunfailures was importable but never loaded and the counts were wrong. Checked in throwaway environments that really have each version (a `PYTHONPATH` overlay only changes the parent): | Child environment | Result | |---|---| | no pytest | class skipped | | pytest 9.1.1, no pytest-rerunfailures | all three skip | | pytest 9.2.0 + pytest-rerunfailures 14.0 | counts pass, goldens skip; counts fail with the old conftest | | pytest 7.3.2 + pytest-rerunfailures 14.0 | all three pass | | job venv with pytest 7.3.2 on the parent's `PYTHONPATH`, or `PYTEST_DISABLE_PLUGIN_AUTOLOAD=1` | counts pass, goldens skip | ``` cd test && CUDA_VISIBLE_DEVICES= python test_testing.py -k TestJunitXml ``` ### Validation A minimal test file covering pass, skip, a flaky test that passes on rerun, a consistent failure with reruns, and an expected failure, run with `--reruns=2 --junit-xml-reruns`, serially and with `-n 2`, under pytest 7.3.2 (requirements-ci pin), 9.1.1 and 9.2.0. pytest-rerunfailures 14.0 and pytest-xdist 3.3.1 are the CI versions. In all six configurations the testcase/rerun/skipped/failure elements of the rerun XML match those the unmodified conftest produces under 9.1.1. Unmodified conftest under 9.2.0 crashes with the KeyError above. ``` PYTHONPATH=<overlay with pytest==$V pytest-rerunfailures==14.0 pytest-xdist==3.3.1> python -m pytest -q -p no:cacheprovider test_mini.py --reruns=2 [-n 2] --junit-xml-reruns=out/r.xml ``` Real test file with each pytest version: 251 passed and 251 testcases in the XML, no INTERNALERROR. The same command with the old conftest on 9.2.0 crashes after the first test. ``` cd test && PYTHONPATH=<overlay pytest==$V> python -m pytest -q -p no:cacheprovider test_pytree.py --reruns=2 -x --junit-xml-reruns=/tmp/rr/r.xml ``` The existing golden fixture `test/junit_xml_testdata/pytest_suite.py` (13 outcome shapes, normalized the same way `TestJunitXml` in `test/test_testing.py` does): - pytest 7.3.2: the new conftest output matches the committed golden `expected/pytest.xml`, which was recorded with pytest 7.3.2. - pytest 9.1.1: the new conftest output is identical to the old conftest's. - pytest 9.2.0: the new conftest output is identical to the old conftest's output under 9.1.1. The old conftest crashes with 81 INTERNALERRORs. The change was reviewed by two fresh agents (correctness and quality) before this update; their findings are addressed above. `lintrunner test/conftest.py` is clean. <details> <summary>Agent Worklog</summary> # Worklog - 2026-10-28: Diffed pytest 9.1.1 and 9.2.0 `_pytest/junitxml.py`. `node_reporters` is now keyed by a `NodeId` dataclass (`report.id`); `finalize` pops `(report.id, workernode)`. conftest's copied `node_reporter` used string nodeids, so it fails at the first teardown, not only on reruns. - Reproduced with a minimal test file under overlays of pytest 7.3.2/9.1.1/9.2.0 on the job venv's torch. - Fix: delegate to `super().node_reporter` and set `__class__ = _NodeReporterReruns`. The XML matches the old behavior under all three versions, serial and with xdist. `test_pytree.py` passes under all three. lintrunner is clean. - Golden fixture junit_xml_testdata/pytest_suite.py: new==golden on 7.3.2, new==old on 9.1.1, new on 9.2.0 == old on 9.1.1. - 2026-10-29: (superseded) C1 test change moved to its own PR, see below. - 2026-10-29: split the C1 test change into job 20261029-pytorch-adhoc-7dcc5b (new PR below this one); this branch now sits on ptq/20261029-pytorch-adhoc-7dcc5b. Counts test passes on 9.2.0 with this fix. Not pushed. - 2026-10-29: now on the refactor 7879242; this PR moves the version skip onto the goldens and runs the counts (plus an INTERNALERROR check) on every version. Commit d334d9d. 9.2.0: counts pass with the fix, fail without. - 2026-10-29: shrank the test change: setUpClass unchanged except it records the version-mismatch reason (golden_skip) instead of raising, and keeps proc output; goldens skip on it; counts add an INTERNALERROR check. Commit 980d40eddfd. 7.3.2 all pass; 9.2.0 counts pass (fail without the conftest fix), goldens skip. - 2026-10-29: consolidated (Edward OK): pytorch#202597 closed, this PR retargeted to main, single commit de4b8db pushed, body Test section updated. - 2026-10-29: automated review flagged the pytest==9.1.1 pin from pytorch#202385 in install_inductor_benchmark_deps.sh as stale. Rebased onto origin/main 92264e2 and removed the pin (exact revert of pytorch#202385). TestJunitXml: 9.1.1 venv and 9.2.0 overlay pass (goldens skip); the 7.3.2 golden diffs I saw were my overlay run missing the repo-root pytest.ini; the PR-review agent confirmed old and new conftest match the goldens on 7.3.2 run in place. Not pushed. - 2026-10-30: Edward approved; pushed 1d11f93 via git (ssh). PR body not yet regenerated: gh on devgpu013 is logged out (~/.config/gh/hosts.yml emptied 2026-10-30 06:20). Body regenerated with ptq pr on 2026-10-30 after gh re-auth. - 2026-10-30: automated review of 1d11f93: without pytest installed, TestJunitXml errored (subprocess ran, no XML) instead of skipping. setUpClass now raises SkipTest first when pytest is missing. Verified with _junit_provenance patched to "pytest missing" (class skipped, subprocess not run); 9.1.1 venv and 9.2.0 overlay unchanged; lint clean. - 2026-10-30: per Edward, replaced the provenance-string checks: setUpClass skips when the child stderr has "No module named pytest" (no XML), and the counts test probes `import pytest_rerunfailures` with the child env. Note: earlier PYTHONPATH-overlay runs of TestJunitXml only changed the parent; the child drops PYTHONPATH and used the venv pytest 9.1.1. Re-verified with symlinked throwaway venvs: 9.2.0 counts pass (fail with origin/main conftest), 7.3.2 all pass, no pytest -> class skipped, no rerunfailures -> all skip. - 2026-10-30: replaced the stderr match and import probe with one child probe (_junit_child_versions, JSON of pytest/rerunfailures versions under _JUNIT_CHILD_ENV): it drives the class skip, has_rerunfailures, and the golden provenance (fixes parent/child mismatch under PYTHONPATH). Child env also drops PYTEST_PLUGINS/PYTEST_DISABLE_PLUGIN_AUTOLOAD. Verified: no pytest -> class skip; no rerunfailures -> 3 skips; 9.2.0 counts pass (fail with origin/main conftest); 7.3.2 all pass; job venv with PYTHONPATH=ov-7.3.2 or PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 -> OK. </details> --- *This PR was generated by [ptq](https://github.com/drisspg/pt_job_queue) with human review.* Pull Request resolved: pytorch#202390 Approved by: https://github.com/nWEIdia, https://github.com/atalman, https://github.com/huydhn, https://github.com/jeanschmidt

pytest 9.2 (pytest-dev/pytest#14758) made Config.cwd_relative_nodeid take a NodeId; run_test.py's --subprocess collection still passes item.nodeid (a str), so test_cuda_primary_ctx and test_cuda_trace end in INTERNALERROR and run no test. Upstream fix: pytorch#202406 (open). Drop the bound once this checkout has it. Assisted-by: Claude

…der pytest 9.2 20261027.0, 20261030.0 and 20261030.1 list test_cuda_primary_ctx and test_cuda_trace as passed; their attached logs show a pytest 9.2.0 INTERNALERROR in collection and no test run. See pytest-dev/pytest#14758, pytorch#202406. Assisted-by: Claude

GPU_TUNED_PYTEST_SPEC ("pytest<9.2") is the single pytest cap for every tuned repo. pytest 9.2.0 (pytest-dev/pytest#14758) breaks PyTorch's run_test.py collection so whole modules ran no tests; the cap stays until pytorch/pytorch#202406 is in our checkout. - gpu_tuned_pip_install_requirements passes the spec as a pip constraints file in both the locked and UNLOCKED branches, so it caps pytest only where a requirements file asks for it, and refuses a lock that pins pytest outside the spec before pip runs. - gpu_tuned_spec_allows <python> <spec> <version> evaluates a spec with pip's vendored packaging. - gpu_tuned_check_pytest <python> verifies the installed pytest. - 11 new tests; ARCHITECTURE H13. Assisted-by: Claude

…h#202390) ## Human Note I tried getting rid of the custom class entirely but Claude persuaded me it was useful. So we're just going to subclass from pytest so we can track upstream more easily. ## Agent Report ### Problem With pytest 9.2.0, any test run that writes `--junit-xml-reruns` (every CI test shard) aborts with an INTERNALERROR at the first test's teardown: ``` File "test/conftest.py", line 254, in pytest_runtest_logreport super().pytest_runtest_logreport(report) File "_pytest/junitxml.py", line 544, in finalize reporter = self.node_reporters.pop((node_id, workernode)) KeyError: (NodeId(path='test/test_pytree.py', names=('TestGenericPytree', 'test_aligned_public_apis'), params=None), None) ``` ### Root cause `LogXMLReruns.node_reporter` was a copy of pytest's `LogXML.node_reporter`. The copy only differed in building a `_NodeReporterReruns` and stored reporters under `(report.nodeid, workernode)`, with `nodeid` as a string. pytest 9.2 keys `node_reporters` by a new `NodeId` dataclass (`report.id`), and `LogXML.finalize` pops with that key. A `NodeId` never equals a string, so the pop fails. ### Fix Call `super().node_reporter(report)` so pytest builds the key and the reporter in whatever form that version uses, check that `_NodeReporterReruns` subclasses the returned type, then set the reporter's `__class__` to `_NodeReporterReruns`. The class docstring now says it may only override methods, not add state or change how instance data is represented. The subclass only overrides output methods (`_prepare_content`, `_write_content`, `append_skipped`) and adds no state, so switching the class in place is safe, and it removes the copy of pytest internals that went stale. I also added a docstring to `_NodeReporterReruns` explaining why it exists. pytest.ini sets `junit_logging_reruns = all`, and with the stock `_NodeReporter` every testcase, including passing tests that print nothing, gets a headed `<system-out>` and `<system-err>` element (skipped tests get both twice). The subclass writes captured output raw and omits empty streams. Regenerating the goldens without the subclass shows exactly that difference. The other option, branching on the pytest version to build the right key type, would keep the copy and break again the next time pytest changes the key. This also reverts pytorch#202385, which pinned `pytest==9.1.1` in the inductor-benchmarks images as a stopgap for this crash (TorchBench's requirements pull in an unpinned pytest). The other pytest 9.2 fixes still open shouldn't affect those images: pytorch#202406 (`--subprocess` test ids) only matters for the few test modules that `run_test.py` runs with `--subprocess`, which are CUDA-init and distributed tests the inductor configs don't run, and pytorch#202412 only restores the teardown part of the RERUNS terminal summary. ### Test `TestJunitXml` (test/test_testing.py) used to skip on every pytest version except the goldens' (the CI pins), so no test could see this crash. `setUpClass` now records that skip reason instead of raising and still runs the fixture suite; only the exact golden comparisons skip on other versions. The tag counts move out of `test_pytest_outcome_shapes` into `test_pytest_outcome_counts`, which runs on every version and also checks for INTERNALERROR. On pytest 9.2.0 the counts test passes with this change and fails without it; on 7.3.2 (the CI pin) all three tests pass. #### Where the versions come from Running the fixture suite on every version means `setUpClass` now depends on the environment in ways the old blanket skip hid, so the version checks have to describe the process that actually runs the suite. That process is a child (`sys.executable -m pytest`) with `PYTHONPATH` removed, so the parent's view can be wrong. Asking the parent with `importlib.metadata` would mean: - With pytest only on the parent's `PYTHONPATH`, the parent sees pytest, the child doesn't, writes no XML, and `setUpClass` raises instead of skipping (the case the automated review hit, where pytest is missing entirely, is the simplest instance). - With a different pytest on the parent's `PYTHONPATH`, the golden header check compares the goldens against one version while the child produces XML with another, and the golden tests fail. So `setUpClass` first runs a small probe, `_junit_child_versions()`, in the child's interpreter and environment: it prints the pytest and pytest-rerunfailures versions (or null) as JSON via `importlib.metadata`, without importing torch. That one result decides all three things: the class skips when pytest is missing, the counts test skips when pytest-rerunfailures is missing, and the golden header is built from it. The probe costs about 60 ms once per class, next to roughly 3.5 s for the fixture suite run the class already does. The alternatives were matching `No module named pytest` in the fixture run's stderr (covers only the missing-pytest case and leaves the golden mismatch) or parsing the `plugins:` line from pytest's console header (fragile, and only saves those 60 ms). The child also drops `PYTEST_PLUGINS` and `PYTEST_DISABLE_PLUGIN_AUTOLOAD`, as it already drops `PYTEST_ADDOPTS`, so it loads the installed plugins as CI does; with autoload off, pytest-rerunfailures was importable but never loaded and the counts were wrong. Checked in throwaway environments that really have each version (a `PYTHONPATH` overlay only changes the parent): | Child environment | Result | |---|---| | no pytest | class skipped | | pytest 9.1.1, no pytest-rerunfailures | all three skip | | pytest 9.2.0 + pytest-rerunfailures 14.0 | counts pass, goldens skip; counts fail with the old conftest | | pytest 7.3.2 + pytest-rerunfailures 14.0 | all three pass | | job venv with pytest 7.3.2 on the parent's `PYTHONPATH`, or `PYTEST_DISABLE_PLUGIN_AUTOLOAD=1` | counts pass, goldens skip | ``` cd test && CUDA_VISIBLE_DEVICES= python test_testing.py -k TestJunitXml ``` ### Validation A minimal test file covering pass, skip, a flaky test that passes on rerun, a consistent failure with reruns, and an expected failure, run with `--reruns=2 --junit-xml-reruns`, serially and with `-n 2`, under pytest 7.3.2 (requirements-ci pin), 9.1.1 and 9.2.0. pytest-rerunfailures 14.0 and pytest-xdist 3.3.1 are the CI versions. In all six configurations the testcase/rerun/skipped/failure elements of the rerun XML match those the unmodified conftest produces under 9.1.1. Unmodified conftest under 9.2.0 crashes with the KeyError above. ``` PYTHONPATH=<overlay with pytest==$V pytest-rerunfailures==14.0 pytest-xdist==3.3.1> python -m pytest -q -p no:cacheprovider test_mini.py --reruns=2 [-n 2] --junit-xml-reruns=out/r.xml ``` Real test file with each pytest version: 251 passed and 251 testcases in the XML, no INTERNALERROR. The same command with the old conftest on 9.2.0 crashes after the first test. ``` cd test && PYTHONPATH=<overlay pytest==$V> python -m pytest -q -p no:cacheprovider test_pytree.py --reruns=2 -x --junit-xml-reruns=/tmp/rr/r.xml ``` The existing golden fixture `test/junit_xml_testdata/pytest_suite.py` (13 outcome shapes, normalized the same way `TestJunitXml` in `test/test_testing.py` does): - pytest 7.3.2: the new conftest output matches the committed golden `expected/pytest.xml`, which was recorded with pytest 7.3.2. - pytest 9.1.1: the new conftest output is identical to the old conftest's. - pytest 9.2.0: the new conftest output is identical to the old conftest's output under 9.1.1. The old conftest crashes with 81 INTERNALERRORs. The change was reviewed by two fresh agents (correctness and quality) before this update; their findings are addressed above. `lintrunner test/conftest.py` is clean. <details> <summary>Agent Worklog</summary> # Worklog - 2026-10-28: Diffed pytest 9.1.1 and 9.2.0 `_pytest/junitxml.py`. `node_reporters` is now keyed by a `NodeId` dataclass (`report.id`); `finalize` pops `(report.id, workernode)`. conftest's copied `node_reporter` used string nodeids, so it fails at the first teardown, not only on reruns. - Reproduced with a minimal test file under overlays of pytest 7.3.2/9.1.1/9.2.0 on the job venv's torch. - Fix: delegate to `super().node_reporter` and set `__class__ = _NodeReporterReruns`. The XML matches the old behavior under all three versions, serial and with xdist. `test_pytree.py` passes under all three. lintrunner is clean. - Golden fixture junit_xml_testdata/pytest_suite.py: new==golden on 7.3.2, new==old on 9.1.1, new on 9.2.0 == old on 9.1.1. - 2026-10-29: (superseded) C1 test change moved to its own PR, see below. - 2026-10-29: split the C1 test change into job 20261029-pytorch-adhoc-7dcc5b (new PR below this one); this branch now sits on ptq/20261029-pytorch-adhoc-7dcc5b. Counts test passes on 9.2.0 with this fix. Not pushed. - 2026-10-29: now on the refactor 7879242; this PR moves the version skip onto the goldens and runs the counts (plus an INTERNALERROR check) on every version. Commit d334d9d. 9.2.0: counts pass with the fix, fail without. - 2026-10-29: shrank the test change: setUpClass unchanged except it records the version-mismatch reason (golden_skip) instead of raising, and keeps proc output; goldens skip on it; counts add an INTERNALERROR check. Commit 980d40eddfd. 7.3.2 all pass; 9.2.0 counts pass (fail without the conftest fix), goldens skip. - 2026-10-29: consolidated (Edward OK): pytorch#202597 closed, this PR retargeted to main, single commit de4b8db pushed, body Test section updated. - 2026-10-29: automated review flagged the pytest==9.1.1 pin from pytorch#202385 in install_inductor_benchmark_deps.sh as stale. Rebased onto origin/main 92264e2 and removed the pin (exact revert of pytorch#202385). TestJunitXml: 9.1.1 venv and 9.2.0 overlay pass (goldens skip); the 7.3.2 golden diffs I saw were my overlay run missing the repo-root pytest.ini; the PR-review agent confirmed old and new conftest match the goldens on 7.3.2 run in place. Not pushed. - 2026-10-30: Edward approved; pushed 1d11f93 via git (ssh). PR body not yet regenerated: gh on devgpu013 is logged out (~/.config/gh/hosts.yml emptied 2026-10-30 06:20). Body regenerated with ptq pr on 2026-10-30 after gh re-auth. - 2026-10-30: automated review of 1d11f93: without pytest installed, TestJunitXml errored (subprocess ran, no XML) instead of skipping. setUpClass now raises SkipTest first when pytest is missing. Verified with _junit_provenance patched to "pytest missing" (class skipped, subprocess not run); 9.1.1 venv and 9.2.0 overlay unchanged; lint clean. - 2026-10-30: per Edward, replaced the provenance-string checks: setUpClass skips when the child stderr has "No module named pytest" (no XML), and the counts test probes `import pytest_rerunfailures` with the child env. Note: earlier PYTHONPATH-overlay runs of TestJunitXml only changed the parent; the child drops PYTHONPATH and used the venv pytest 9.1.1. Re-verified with symlinked throwaway venvs: 9.2.0 counts pass (fail with origin/main conftest), 7.3.2 all pass, no pytest -> class skipped, no rerunfailures -> all skip. - 2026-10-30: replaced the stderr match and import probe with one child probe (_junit_child_versions, JSON of pytest/rerunfailures versions under _JUNIT_CHILD_ENV): it drives the class skip, has_rerunfailures, and the golden provenance (fixes parent/child mismatch under PYTHONPATH). Child env also drops PYTEST_PLUGINS/PYTEST_DISABLE_PLUGIN_AUTOLOAD. Verified: no pytest -> class skip; no rerunfailures -> 3 skips; 9.2.0 counts pass (fail with origin/main conftest); 7.3.2 all pass; job venv with PYTHONPATH=ov-7.3.2 or PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 -> OK. </details> --- *This PR was generated by [ptq](https://github.com/drisspg/pt_job_queue) with human review.* Pull Request resolved: pytorch#202390 Approved by: https://github.com/nWEIdia, https://github.com/atalman, https://github.com/huydhn, https://github.com/jeanschmidt

github-actions

Automated review of b6237d8: nothing blocking; ready for human review.

The PR builds --subprocess --use-pytest child test ids from os.path.relpath(item.path, config.invocation_params.dir) plus the node id's Class::name part, replacing config.cwd_relative_nodeid, which pytest 9.2 broke. If the file and cwd are on different Windows drives, the id falls back to the absolute path. Recommendation: Approve. The remaining points are all non-blocking test improvements, though the review recommends adding the os.path.isabs assertion before landing.

⚪ 4 non-blocking findings

pytorch/test/torchci/test_testing.py

Line 616 in b6237d8

self.assertTrue(os.path.samefile(ids[0].split("::", 1)[0], __file__), ids)

Non-blocking (continuing from the earlier review): nothing checks the relative-path behaviour. The only path assertion is os.path.samefile(ids\[0\].split("::", 1)\[0\], __file__), which also passes for an absolute path. If common_utils.py:1439-1442 were reverted to path = str(item.path), both new tests would still pass. The relative path is the whole reason for this design: pytest splits arguments at the first \[, so ids under a directory like ws\[1\] break when absolute. Fix: path = ids\[0\].split("::", 1)\[0\], then self.assertFalse(os.path.isabs(path), ids). I recommend adding this one-line assertion before landing.

pytorch/test/torchci/test_testing.py

Line 610 in b6237d8

ids = get_pytest_test_cases([__file__, "-k", "test_ids_resolve_from_cwd"])

Non-blocking: the tests don't use the same input as production (here and at :627). Both tests pass __file__ as the collection argument, and for a script run as __main__ that is absolute on Python 3.9+. Production passes the relative argv\[0\] (test/run_test.py:687 common_utils.py:1242 :1568). In a checkout whose path contains \[, the tests' own inner collection would hit the splitting problem this PR describes, even though production works there. Passing sys.argv\[0\] (or os.path.relpath(__file__)) would match production.

pytorch/test/torchci/test_testing.py

Line 621 in b6237d8

def cross_drive_relpath(path, start=os.curdir):

Non-blocking: the fake relpath affects the whole process. unittest.mock.patch("os.path.relpath", ...) replaces relpath for every caller while the nested pytest.main runs, not just for get_pytest_test_cases. The fake calls os.path.samefile(path, __file__) on every call. Real relpath handles a nonexistent path fine, but samefile raises FileNotFoundError on one. No caller in the repo hits this today; pytest's internals weren't checked. Comparing os.path.abspath(path) == os.path.abspath(__file__) would avoid the stat. So would moving the id construction into a small helper and testing that directly, which would also remove the nested pytest.main.

pytorch/test/torchci/test_testing.py

Line 604 in b6237d8

    
           @unittest.skipIf(__name__ != "__main__", "--use-main-module collects from __main__")

Non-blocking: the new class skips none of the configs that the file's other pytest-driving classes skip. The classes at lines 166-172, 440-444, 522-528 and 659-665 carry @ skipIfTorchDynamo, skipIf(TEST_WITH_CROSSREF) and skipIf(TEST_CUDA or TEST_WITH_ROCM), but TestGetPytestTestCases has only the __name__ skip. So it runs on the Dynamo-wrapped, crossref and GPU shards (.ci/pytorch/test.sh:768, :276-277). That costs a little time but causes no known failure. Leaving out the Windows skip makes sense, since the class tests the Windows fallback; the other skips are worth matching for consistency.

_{This review is AI-generated and advisory. Each new automated review replaces the previous one. Report a bad review}

Withdrawn: the automated review of b6237d8 found nothing blocking.

ezyang added the topic: not user facing topic category label Oct 28, 2026

ezyang force-pushed the ptq/20261028-pytorch-adhoc-9e02fa branch from 48ef4ba to 957f776 Compare October 28, 2026 16:39

ezyang changed the title ~~Use item.path for --subprocess pytest test ids~~ Build --subprocess pytest test ids from public fields Oct 28, 2026

ezyang marked this pull request as ready for review October 28, 2026 16:48

ezyang requested a review from a team as a code owner October 28, 2026 16:48

ezyang added autoland mergedog mergedog is watching this PR labels Oct 28, 2026

github-actions Bot requested review from SherlockNoMad, albanD, antoniojkim, bdhirsh and miladm October 28, 2026 16:48

ezyang added this pull request to stack #202415 October 28, 2026 16:51

pytorchgreenlight Bot added the GL/human GreenLight marked as needing human review label Oct 28, 2026

pytorchbot added the in progress PR is being worked on by the author, until the automated review accepts it label Oct 28, 2026

This comment was marked as outdated.

Sign in to view

ezyang added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 28, 2026

This comment was marked as outdated.

Sign in to view

izaitsevfb mentioned this pull request Oct 29, 2026

Automated PR review blocks more than maintainers would pytorch/test-infra#9245

Open

ezyang mentioned this pull request Oct 30, 2026

Make test/conftest.py LogXMLReruns compatible with pytest 9.2 #202390

Closed

ezyang added 2 commits October 30, 2026 08:44

ezyang force-pushed the ptq/20261028-pytorch-adhoc-9e02fa branch from 12cfd87 to b6237d8 Compare October 30, 2026 15:53

This comment was marked as outdated.

Sign in to view

izaitsevfb mentioned this pull request Oct 30, 2026

Automated PR review asks for tests that add little value pytorch/test-infra#9242

Open

ezyang mentioned this pull request Oct 30, 2026

[pr-review] Block in proportion to stakes, not on everything #202820

Draft

ezyang added in progress PR is being worked on by the author, until the automated review accepts it and removed in progress PR is being worked on by the author, until the automated review accepts it labels Nov 10, 2026

github-actions Bot reviewed Nov 10, 2026

View reviewed changes

pytorch-bot Bot added ready for review Automated review passed, ready for a final review by a maintainer. and removed in progress PR is being worked on by the author, until the automated review accepts it labels Nov 10, 2026

Navigation Menu

Repository navigation

Build --subprocess pytest test ids from public fields - #202406

Build --subprocess pytest test ids from public fields#202406
ezyang wants to merge 2 commits into
mainfrom
ptq/20261028-pytorch-adhoc-9e02fa

ezyang commented Oct 28, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

ezyang commented Oct 28, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions Bot left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ezyang commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Human Note

Agent Report

Uh oh!

pytorch-bot Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Status: ready for maintainer review

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/202406

✅ No Failures

Uh oh!

This comment was marked as outdated.

Uh oh!

ezyang commented Oct 28, 2026

mergedog handoff

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ezyang commented Oct 28, 2026 •

edited

Loading

pytorch-bot Bot commented Oct 28, 2026 •

edited

Loading