Add env PYTORCH_TEST_DO_NOT_USE_PYTEST as an option to not use pytest in unit testing by xwang233 · Pull Request #96444 · pytorch/pytorch

xwang233 · 2023-03-09T20:49:45Z

Set environment variable

PYTORCH_TEST_DO_NOT_USE_PYTEST=1

to not use pytest in pytorch unit testing.

This change is related to some recent changes, e.g. #96210, #96016, #95844, #95659, that enabled the use of pytest in many test modules. Those test modules were testing normally before, but failed immediately after pytest is used. Sample stacktraces are:

root@8e3168a83ee2:/opt/pytorch/pytorch# python test/run_test.py -v -i test_optim -- -v --save-xml
Ignoring disabled issues:  []
/opt/pytorch/pytorch/test/run_test.py:1225: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
Selected tests:
 test_optim
parallel (file granularity) tests:
 test_optim
serial (file granularity) tests:

Ignoring disabled issues:  []
Ignoring disabled issues:  []
Running test_optim ... [2023-03-09 12:51:59.358110]
Executing ['/usr/local/bin/python', '-bb', 'test_optim.py', '-v', '--save-xml', '-v', '--use-pytest', '-vv', '-rfEX', '-x', '--reruns=2'] ... [2023-03-09 12:51:59.358810]

Test results will be stored in test-reports/python-pytest/test_optim/test_optim-5e41643c8bac8ace.xml
Traceback (most recent call last):
  File "/opt/pytorch/pytorch/test/test_optim.py", line 4581, in <module>
    run_tests()
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 796, in run_tests
    exit_code = pytest.main(args=pytest_args)
  File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 329, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.10/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/usr/local/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1060, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1348, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1231, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/usr/local/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/usr/local/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/usr/local/lib/python3.10/site-packages/xdist/looponfail.py", line 16, in <module>
    import execnet
  File "/usr/local/lib/python3.10/site-packages/execnet/__init__.py", line 14, in <module>
    from .gateway_base import DataFormatError
  File "/usr/local/lib/python3.10/site-packages/execnet/gateway_base.py", line 1138, in <module>
    FLOAT_FORMAT_SIZE = struct.calcsize(FLOAT_FORMAT)
BytesWarning: Comparison between bytes and string
FINISHED PRINTING LOG FILE of test_optim (/opt/pytorch/pytorch/test/test-reports/test_optim_1pnlesrz.log)

test_optim failed!
Traceback (most recent call last):
  File "/opt/pytorch/pytorch/test/run_test.py", line 1428, in <module>
    main()
  File "/opt/pytorch/pytorch/test/run_test.py", line 1386, in main
    raise RuntimeError(
RuntimeError: test_optim failed!

Tip: You can keep running tests even on failure by passing --keep-going to run_test.py.
If running on CI, add the 'keep-going' label to your PR and rerun your jobs.

I'd like to propose this option that allows users to use the good old python unit test way instead of pytest to run their testing in CI.

cc @ezyang @malfet @clee2000 @huydhn @ptrblck

pytorch-bot · 2023-03-09T20:49:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/96444

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 2 Pending

As of commit 5cfaeaa:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

malfet

I think you have some sort of a setup problem on your system, but yeah, it would not hurt to have a predicate (that should also return false if pytest is not installed on the system)

malfet · 2023-03-09T20:57:21Z

test/run_test.py

        for test in selected_tests_parallel:
            options_clone = copy.deepcopy(options)
-            if test not in PYTEST_BLOCKLIST:
+            if test not in PYTEST_BLOCKLIST and os.getenv('PYTORCH_TEST_DO_NOT_USE_PYTEST', '0') == '0':


To avoid code duplication, do you mind adding can_run_in_pytest predicate that would be use on lines 1375 and 1393

Suggested change

if test not in PYTEST_BLOCKLIST and os.getenv('PYTORCH_TEST_DO_NOT_USE_PYTEST', '0') == '0':

if can_run_in_pytes(test):

xwang233 · 2023-03-09T21:06:47Z

Thanks for the comment @malfet !

Though not very related to this issue/PR. The reason that we don't prefer pytest is that, when no test is selected with -k flag, pytest returns failed but normal python unit test returns successful, e.g.

root@8e3168a83ee2:/opt/pytorch/pytorch# pytest test/test_optim.py -v -k aaaaaaa
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.10.9, pytest-7.2.2, pluggy-1.0.0 -- /usr/local/bin/python3.10
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/opt/pytorch/pytorch/.hypothesis/examples')
rootdir: /opt/pytorch/pytorch, configfile: pytest.ini
plugins: hypothesis-4.50.8, xdist-3.2.0, rerunfailures-11.1.2, shard-0.1.2, xdoctest-1.0.2
collected 170 items / 170 deselected / 0 selected
Running 0 items in this shard:

============================================================================================ 170 deselected in 2.78s ============================================================================================
root@8e3168a83ee2:/opt/pytorch/pytorch# echo $?
5
root@8e3168a83ee2:/opt/pytorch/pytorch# python test/test_optim.py -v -k aaaaaaa

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
root@8e3168a83ee2:/opt/pytorch/pytorch# echo $?
0

I've probably raised this issue somewhere 2 years ago when pytorch team started to switch to pytest. We currently rely on this -k pattern matching to only run CUDA related tests, though string pattern matching is not the best solution. Related #44159

huydhn · 2023-03-09T21:34:24Z

I found a thread on pytest about this issue pytest-dev/pytest-xdist#596. Two things stand out:

You could try to upgrade hypothesis package to a newer version BytesWarning in execnet when deserializing float pytest-dev/pytest-xdist#596 (comment)
python -bb is the one that raise this as error @clee2000 Do you know the historical context of why this flag is needed in the python command?

clee2000 · 2023-03-09T22:08:14Z

-bb was added in https://github.com/pytorch/pytorch/pull/79868/files
We also run into the ran 0 tests but get around it by directly checking for the exit code in common_utils so you shouldn't run into it when calling the test file unless you invoke pytest directly

xwang233 · 2023-03-09T22:16:26Z

@pytorchmergebot merge -g

pytorchmergebot · 2023-03-09T22:19:26Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

xwang233 · 2023-03-09T22:19:28Z

Upgrading hypothesis to the latest version 6.68.2 resolves the issue. I'll try to check the common_utils with pytest return codes. Thanks for the reference.

… in unit testing (#96444) Set environment variable ``` PYTORCH_TEST_DO_NOT_USE_PYTEST=1 ``` to not use pytest in pytorch unit testing. This change is related to some recent changes, e.g. #96210, #96016, #95844, #95659, that enabled the use of pytest in many test modules. Those test modules were testing normally before, but failed immediately after pytest is used. Sample stacktraces are: ```python root@8e3168a83ee2:/opt/pytorch/pytorch# python test/run_test.py -v -i test_optim -- -v --save-xml Ignoring disabled issues: [] /opt/pytorch/pytorch/test/run_test.py:1225: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6": Selected tests: test_optim parallel (file granularity) tests: test_optim serial (file granularity) tests: Ignoring disabled issues: [] Ignoring disabled issues: [] Running test_optim ... [2023-03-09 12:51:59.358110] Executing ['/usr/local/bin/python', '-bb', 'test_optim.py', '-v', '--save-xml', '-v', '--use-pytest', '-vv', '-rfEX', '-x', '--reruns=2'] ... [2023-03-09 12:51:59.358810] Test results will be stored in test-reports/python-pytest/test_optim/test_optim-5e41643c8bac8ace.xml Traceback (most recent call last): File "/opt/pytorch/pytorch/test/test_optim.py", line 4581, in <module> run_tests() File "/opt/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 796, in run_tests exit_code = pytest.main(args=pytest_args) File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 148, in main config = _prepareconfig(args, plugins) File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 329, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/usr/local/lib/python3.10/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse config: Config = outcome.get_result() File "/usr/local/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1060, in pytest_cmdline_parse self.parse(args) File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1348, in parse self._preparse(args, addopts=addopts) File "/usr/local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1231, in _preparse self.pluginmanager.load_setuptools_entrypoints("pytest11") File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints plugin = ep.load() File "/usr/local/lib/python3.10/importlib/metadata/__init__.py", line 171, in load module = import_module(match.group('module')) File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "/usr/local/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/usr/local/lib/python3.10/site-packages/xdist/looponfail.py", line 16, in <module> import execnet File "/usr/local/lib/python3.10/site-packages/execnet/__init__.py", line 14, in <module> from .gateway_base import DataFormatError File "/usr/local/lib/python3.10/site-packages/execnet/gateway_base.py", line 1138, in <module> FLOAT_FORMAT_SIZE = struct.calcsize(FLOAT_FORMAT) BytesWarning: Comparison between bytes and string FINISHED PRINTING LOG FILE of test_optim (/opt/pytorch/pytorch/test/test-reports/test_optim_1pnlesrz.log) test_optim failed! Traceback (most recent call last): File "/opt/pytorch/pytorch/test/run_test.py", line 1428, in <module> main() File "/opt/pytorch/pytorch/test/run_test.py", line 1386, in main raise RuntimeError( RuntimeError: test_optim failed! Tip: You can keep running tests even on failure by passing --keep-going to run_test.py. If running on CI, add the 'keep-going' label to your PR and rerun your jobs. ``` I'd like to propose this option that allows users to use the good old python unit test way instead of pytest to run their testing in CI. Pull Request resolved: pytorch/pytorch#96444 Approved by: https://github.com/malfet

add env PYTORCH_TEST_DO_NOT_USE_PYTEST

32c0db2

xwang233 requested a review from a team as a code owner March 9, 2023 20:49

pytorch-bot bot added the topic: not user facing topic category label Mar 9, 2023

xwang233 requested a review from malfet March 9, 2023 20:53

malfet approved these changes Mar 9, 2023

View reviewed changes

pytorchbot added the open source label Mar 9, 2023

can_run_in_pytest

5cfaeaa

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 9, 2023

pytorchmergebot added the Merged label Mar 10, 2023

pytorchmergebot closed this in cf3d3a5 Mar 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add env PYTORCH_TEST_DO_NOT_USE_PYTEST as an option to not use pytest in unit testing#96444

Add env PYTORCH_TEST_DO_NOT_USE_PYTEST as an option to not use pytest in unit testing#96444
xwang233 wants to merge 2 commits intopytorch:masterfrom
xwang233:options-to-not-use-pytest-in-unit-testing

xwang233 commented Mar 9, 2023 •

edited

Loading

Uh oh!

pytorch-bot bot commented Mar 9, 2023 •

edited

Loading

Uh oh!

malfet left a comment

Uh oh!

malfet Mar 9, 2023

Uh oh!

xwang233 commented Mar 9, 2023 •

edited

Loading

Uh oh!

huydhn commented Mar 9, 2023 •

edited

Loading

Uh oh!

clee2000 commented Mar 9, 2023

Uh oh!

xwang233 commented Mar 9, 2023

Uh oh!

pytorchmergebot commented Mar 9, 2023

Uh oh!

xwang233 commented Mar 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	if test not in PYTEST_BLOCKLIST and os.getenv('PYTORCH_TEST_DO_NOT_USE_PYTEST', '0') == '0':
	if can_run_in_pytes(test):

Conversation

xwang233 commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/96444

⏳ No Failures, 2 Pending

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet Mar 9, 2023

Choose a reason for hiding this comment

Uh oh!

xwang233 commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huydhn commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clee2000 commented Mar 9, 2023

Uh oh!

xwang233 commented Mar 9, 2023

Uh oh!

pytorchmergebot commented Mar 9, 2023

Merge started

Uh oh!

xwang233 commented Mar 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xwang233 commented Mar 9, 2023 •

edited

Loading

pytorch-bot bot commented Mar 9, 2023 •

edited

Loading

xwang233 commented Mar 9, 2023 •

edited

Loading

huydhn commented Mar 9, 2023 •

edited

Loading