Skip to content

Use standard mechanism for stdlib names#81520

Closed
Erotemic wants to merge 3 commits intopytorch:masterfrom
Erotemic:dev/std-stdlib
Closed

Use standard mechanism for stdlib names#81520
Erotemic wants to merge 3 commits intopytorch:masterfrom
Erotemic:dev/std-stdlib

Conversation

@Erotemic
Copy link
Copy Markdown
Contributor

@Erotemic Erotemic commented Jul 14, 2022

I noticed that in #81261 all of the stdlib module names were explicitly listed, however as of Python 3.10 the stdlib now has a mechanism for this. python/cpython#87121

I figured it was better to use sys.stdlib_module_names going forward for 3.10+ instead of having to maintain this file for every new Python release. For docs see:
https://docs.python.org/3/library/sys.html#sys.stdlib_module_names

I did a symmetric difference to determine what the effective change would be. I verified that everything listed in this file ins included in sys.stdlib_module_names. However, there are files in sys.stdlib_module_names that are not included in the previous hard coded definition. Namely these are:

frozenset({'__future__',
           '_abc',
           '_aix_support',
           '_asyncio',
           '_bisect',
           '_blake2',
           '_bootsubprocess',
           '_bz2',
           '_codecs',
           '_codecs_cn',
           '_codecs_hk',
           '_codecs_iso2022',
           '_codecs_jp',
           '_codecs_kr',
           '_codecs_tw',
           '_collections',
           '_collections_abc',
           '_compat_pickle',
           '_compression',
           '_contextvars',
           '_crypt',
           '_csv',
           '_ctypes',
           '_curses',
           '_curses_panel',
           '_datetime',
           '_dbm',
           '_decimal',
           '_elementtree',
           '_frozen_importlib',
           '_frozen_importlib_external',
           '_functools',
           '_gdbm',
           '_hashlib',
           '_heapq',
           '_imp',
           '_io',
           '_json',
           '_locale',
           '_lsprof',
           '_lzma',
           '_markupbase',
           '_md5',
           '_msi',
           '_multibytecodec',
           '_multiprocessing',
           '_opcode',
           '_operator',
           '_osx_support',
           '_overlapped',
           '_pickle',
           '_posixshmem',
           '_posixsubprocess',
           '_py_abc',
           '_pydecimal',
           '_pyio',
           '_queue',
           '_random',
           '_scproxy',
           '_sha1',
           '_sha256',
           '_sha3',
           '_sha512',
           '_signal',
           '_sitebuiltins',
           '_socket',
           '_sqlite3',
           '_sre',
           '_ssl',
           '_stat',
           '_statistics',
           '_string',
           '_strptime',
           '_struct',
           '_symtable',
           '_threading_local',
           '_tkinter',
           '_tracemalloc',
           '_uuid',
           '_warnings',
           '_weakref',
           '_weakrefset',
           '_winapi',
           '_zoneinfo',
           'antigravity',
           'genericpath',
           'idlelib',
           'nt',
           'nturl2path',
           'opcode',
           'pydoc_data',
           'pyexpat',
           'this'})

I'm not sure if excluding these matters. I wouldn't think it would, but if it does and it is better to explicitly update this file each time, then feel free to close this.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jul 14, 2022

🔗 Helpful links

❌ 1 New Failures, 32 Pending

As of commit 500b434 (more details on the Dr. CI page):

Expand to see more
  • 1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

See GitHub Actions build pull / linux-focal-py3.7-gcc7-mobile-lightweight-dispatch-build / build (1/1)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

2022-07-15T23:16:16.5843646Z ##[error]Process completed with exit code 137.
2022-07-15T23:11:13.9915847Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/TensorIterator.cpp.o�[0m
2022-07-15T23:11:14.4289836Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/nnapi/nnapi_register.cpp.o�[0m
2022-07-15T23:11:14.5013262Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_0.cpp.o�[0m
2022-07-15T23:11:16.4377683Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_1.cpp.o�[0m
2022-07-15T23:11:17.2085293Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_2.cpp.o�[0m
2022-07-15T23:11:17.6635749Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_3.cpp.o�[0m
2022-07-15T23:11:24.9696521Z [ 96%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_4.cpp.o�[0m
2022-07-15T23:11:25.8021450Z [ 97%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_5.cpp.o�[0m
2022-07-15T23:13:15.8728658Z [ 97%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_6.cpp.o�[0m
2022-07-15T23:14:58.0156934Z [ 97%] �[32mBuilding CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCodegenUnboxedKernels_7.cpp.o�[0m
2022-07-15T23:16:16.5843646Z ##[error]Process completed with exit code 137.
2022-07-15T23:16:16.6187132Z Prepare all required actions
2022-07-15T23:16:16.6250372Z ##[group]Run ./.github/actions/teardown-linux
2022-07-15T23:16:16.6250664Z with:
2022-07-15T23:16:16.6250889Z ##[endgroup]
2022-07-15T23:16:16.6297380Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh
2022-07-15T23:16:16.6297762Z �[36;1m.github/scripts/wait_for_ssh_to_drain.sh�[0m
2022-07-15T23:16:16.7254549Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-07-15T23:16:16.7254866Z ##[endgroup]
2022-07-15T23:16:16.7704353Z Holding runner for 2 hours until all ssh sessions have logged out
2022-07-15T23:16:16.7845000Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@Erotemic
Copy link
Copy Markdown
Contributor Author

@malfet @suo thoughts?

Copy link
Copy Markdown
Contributor

@malfet malfet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but please fix lint (apply the suggested edits)

@Erotemic
Copy link
Copy Markdown
Contributor Author

Looks like the linter is still upset? Not sure what to do with this:

You can run `lintrunner -a` to apply this patch.
26  26 |        if sys.version_info.minor == 9:
27  27 |            return stdlib3_9
28  28 |        if sys.version_info.minor >= 10:
28     |-            return sys.stdlib_module_names# type: ignore[attr-defined]
    29 |+            return sys.stdlib_module_names# type: ignore[attr-defined]
30  30 |    elif sys.version_info.major > 3:
30     |-        return sys.stdlib_module_names# type: ignore[attr-defined]
    31 |+        return sys.stdlib_module_names# type: ignore[attr-defined]
32  32 |
33  33 |    raise RuntimeError(f"Unsupported Python version: {sys.version_info}")
34  34 |

@kit1980
Copy link
Copy Markdown
Contributor

kit1980 commented Jul 15, 2022

Looks like the linter is still upset? Not sure what to do with this:

@Erotemic it wants 2 spaces before #, not 3.

@malfet
Copy link
Copy Markdown
Contributor

malfet commented Jul 15, 2022

@pytorchbot merge -f

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link
Copy Markdown
Contributor

Hey @Erotemic.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

@Erotemic
Copy link
Copy Markdown
Contributor Author

I don't see a way I can add the topic: not user facing label

@kit1980 kit1980 added the topic: not user facing topic category label Jul 16, 2022
@kit1980
Copy link
Copy Markdown
Contributor

kit1980 commented Jul 16, 2022

I've just added it, but it's not too important.

facebook-github-bot pushed a commit that referenced this pull request Jul 18, 2022
Summary:
I noticed that in #81261 all of the stdlib module names were explicitly listed, however as of Python 3.10 the stdlib now has a mechanism for this. python/cpython#87121

I figured it was better to use `sys.stdlib_module_names` going forward for 3.10+ instead of having to maintain this file for every new Python release. For docs see:
https://docs.python.org/3/library/sys.html#sys.stdlib_module_names

I did a symmetric difference to determine what the effective change would be. I verified that everything listed in this file ins included in sys.stdlib_module_names. However, there are files in sys.stdlib_module_names that are not included in the previous hard coded definition. Namely these are:

```
frozenset({'__future__',
           '_abc',
           '_aix_support',
           '_asyncio',
           '_bisect',
           '_blake2',
           '_bootsubprocess',
           '_bz2',
           '_codecs',
           '_codecs_cn',
           '_codecs_hk',
           '_codecs_iso2022',
           '_codecs_jp',
           '_codecs_kr',
           '_codecs_tw',
           '_collections',
           '_collections_abc',
           '_compat_pickle',
           '_compression',
           '_contextvars',
           '_crypt',
           '_csv',
           '_ctypes',
           '_curses',
           '_curses_panel',
           '_datetime',
           '_dbm',
           '_decimal',
           '_elementtree',
           '_frozen_importlib',
           '_frozen_importlib_external',
           '_functools',
           '_gdbm',
           '_hashlib',
           '_heapq',
           '_imp',
           '_io',
           '_json',
           '_locale',
           '_lsprof',
           '_lzma',
           '_markupbase',
           '_md5',
           '_msi',
           '_multibytecodec',
           '_multiprocessing',
           '_opcode',
           '_operator',
           '_osx_support',
           '_overlapped',
           '_pickle',
           '_posixshmem',
           '_posixsubprocess',
           '_py_abc',
           '_pydecimal',
           '_pyio',
           '_queue',
           '_random',
           '_scproxy',
           '_sha1',
           '_sha256',
           '_sha3',
           '_sha512',
           '_signal',
           '_sitebuiltins',
           '_socket',
           '_sqlite3',
           '_sre',
           '_ssl',
           '_stat',
           '_statistics',
           '_string',
           '_strptime',
           '_struct',
           '_symtable',
           '_threading_local',
           '_tkinter',
           '_tracemalloc',
           '_uuid',
           '_warnings',
           '_weakref',
           '_weakrefset',
           '_winapi',
           '_zoneinfo',
           'antigravity',
           'genericpath',
           'idlelib',
           'nt',
           'nturl2path',
           'opcode',
           'pydoc_data',
           'pyexpat',
           'this'})
```

I'm not sure if excluding these matters. I wouldn't think it would, but if it does and it is better to explicitly update this file each time, then feel free to close this.

Pull Request resolved: #81520
Approved by: https://github.com/malfet

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/4c9eae331b2695be011d96534a5ece44ba629601

Reviewed By: DanilBaibak

Differential Revision: D37919540

Pulled By: DanilBaibak

fbshipit-source-id: 6765556778ecad6edf254ad2ffc0d50d57ee1bbf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants