Python: default test runner fails often because it imports all modules #14849
Description
Since we upgraded setuptools it changed behavior as to how python setup.py test functions.
Before, it would often not do anything, silently passing the checkPhase.
After the upgrade, it became more aggressive, and it now appears it tries to find tests by importing all modules. When importing a module exceptions can occur, but these are not catched by setuptools, resulting straight away in a failing checkPhase.
Example:
Traceback (most recent call last):
File "nix_run_setup.py", line 6, in <module>
exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
File "setup.py", line 173, in <module>
**kwargs
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 159, in run
File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 140, in with_project_on_sys_path
File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 180, in run_tests
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/main.py", line 93, in __init__
self.parseArgs(argv)
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/main.py", line 123, in parseArgs
self._do_discovery([])
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/main.py", line 228, in _do_discovery
self.test = loader.discover(self.start, self.pattern, self.top)
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 475, in _find_test_path
tests = self.loadTestsFromModule(package, pattern=pattern)
File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 38, in loadTestsFromModule
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 190, in loadTestsFromName
return self.loadTestsFromModule(obj)
File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 38, in loadTestsFromModule
File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 153, in loadTestsFromName
module = __import__(module_name)
File "/tmp/nix-build-python3.5-tornado-4.2.1.drv-0/tornado-4.2.1/tornado/platform/kqueue.py", line 23, in <module>
assert hasattr(select, 'kqueue'), 'kqueue not supported'
AssertionError: kqueue not supported
builder for ‘/nix/store/s32ylw2jql4fbphb1fyzlgrldmn5bmig-python3.5-tornado-4.2.1.drv’ failed with exit code 1
error: build of ‘/nix/store/s32ylw2jql4fbphb1fyzlgrldmn5bmig-python3.5-tornado-4.2.1.drv’ failed