-
Notifications
You must be signed in to change notification settings - Fork 84
Closed
Labels
A-pythonArea: Python (everything related to the Python interpreter we're using)Area: Python (everything related to the Python interpreter we're using)C-goalCategory: Long term goal. May be code-related or a meta goal.Category: Long term goal. May be code-related or a meta goal.C-internalCategory: Keep track of internal refactoring. Implies the C-todo label except if it is a full C-goalCategory: Keep track of internal refactoring. Implies the C-todo label except if it is a full C-goalM-relnotesMisc: Issue should be listed in the version history for its milestoneMisc: Issue should be listed in the version history for its milestone
Milestone
Description
Goold old Python 2 is dead in about 3 months as of the time of writing.
What follows is a list of roadblocks:
-
chardet. We bundle version 1.0.1, from 2008. May just happen to work on py3, but I wouldn't hold my breath - done in 8e201c4. -
wxPython. Neither 2.8 nor 3.0 have Python 3 versions available. However, the work put into the wx3 upgrade was not wasted; it will make upgrading to wxPython 4.0, which does have Python 3 support, much easier - see 050391c, done in 22de7ff. -
loot-api-python. Python 3 versions are available starting from version 4.0, but see Replace libloot-python with handwritten parser #431 for the obvious problem there - done in 934f352. - Records Refactoring #480 and Patchers Refactoring #312 are blockers, for the most part - will be much easier to proceed to Python 3 once they've progressed further
- Sub-issues:
- Well, the entire codebase. A decent chunk can probably be handled by automated tools, but we need to consider the output carefully - plus automated tools won't be nearly enough to catch everything. In particular:
- Unicode strings and byte strings
-
ur''is gone - done in 660ecbb and d43ad24 - We should go through all our unprefixed string usages and add either
b''oru''- done in way too many commits to list - StringIO needs to be changed to BytesIO where applicable - done in 7c95d05
-
strneeds to be replaced withbytesorunicodedepending on how it's used - done in 04835ec -
basestringis problematic - it's gone in py3, but 2to3 will change it tostr. Most usages should be replaced withunicodeorbytes, but a couple really do need to handle both - done in 42b879f - Revisit all
openuses, check if opening in bytes mode is needed - done in c120e18
-
- We need to get rid of
getattrcalls that pass in bytes, most notably due toModFile.__getattr__- Patchers Refactoring #312/Records Refactoring #480, (hopefully) done in 0848872 - Tuple parameter unpacking is gone in py3. The only place we use this is in lambdas, which is handled by 2to3 - done in 7a6b7c2
-
zip/map/filter/rangeneed to get replaced with either theitertools/xrangevariant or an equivalent comprehension - plus addlist()calls if we modify the iterables in the loop - worked on in 709e146 and e8c4d43, done in a75dbc6 - All
keysanditerkeyscalls can go -> done in 567f153 and 09e1633 -
itemsandvalues: Change toiteritems/itervalues, then run 2to3'sdictfixer. If it changes them toiter(...), change them toviewitems/viewvaluesinstead - done in d22469b - Old-style classes need to be switched to new-style classes ahead of time to catch any issues that arise - see the commit message in fa74a1b, for example - worked on in cae844b, done in 6db5b8b
- True division updating - for each usage, we need to consider if we want floats or integers - done in 505d26f
- Relative imports - are we using the explicit ones everywhere already? Should mostly be caught by automated tools, but problems can arise here - done in c692662
- Longs are gone, replaced with regular integers - a bunch of unnecessary usages in records code, which will be dropped in the course of Records Refactoring #480, any other usages should be investigated - worked on in d8d03ca, done in 197b6a2
- ints no longer have a size limit, so
sys.maxintis gone - depending on the use case, could be replaced bysys.maxsize- done in 659e5b6 -
cmpand__cmp__are gone; usage is rare - five in ScriptParser and belt and one each in cint, balt and mods_metadata - done in 5a98eb4'
- Unicode strings and byte strings
Things that are not roadblocks:
-
comtypes- dropped during wxPython 3.0.2.0 upgrade -
gitpython- dropped in build script rework, see pygit2 below -
py2exe- readily supports Python 3 - we should however consider our options once we get there, see Replace py2exe with PyInstaller #491 -
pyfiglet- readily supports Python 3 -
pygit2- readily supports Python 3 -
pymupdf- readily supports Python 3 -
pytest- readily supports Python 3 -
python-lz4- readily supports Python 3 -
pywin32- readily supports Python 3 -
pyyaml- readily supports Python 3 -
scandir- built into Python 3, we'll just drop this once we upgrade -
toml- readily supports Python 3
Useful tracking regexes (make sure to enable *.py mask):
urprefix:ur('|")[^,)]- Old-style classes:
class [^:(]+: - Old-style super initializer calls:
(?<!self\))\.__init__ - WIP regex for all old-style super calls, has tons of false positives:
[A-Z]\w+\.\w+\(self(,|\)) - Absurd regex for tracking non-library imports that need updating:
^ *import (?!os|sys|re|errno|time|subprocess|io|threading|wx|wx\.adv|cPickle|argparse|atexit|codecs|ctypes|platform|shutil|traceback|codecs|chardet|lz4|win32api|win32com|yaml|tempfile|Tkinter|collections|copy|csv|copy|datetime|stat|string|struct|textwrap|scandir|inspect|pkgutil|math|gettext|locale|msgfmt|pygettext|operator|webbrowser|array|zlib|binascii|_winreg|win32gui|importlib|random|toml|pytest|glob|logging|zipfile|pygit2|py2exe) openusages without false positives:(?<!webbrowser\.)(?<!io\.)(?<!def )\bopen\b
Some resources:
- Supporting Python 3: An in-depth guide - and by 'in-depth' this does mean in-depth
- Porting Python 2 Code to Python 3 - a general overview from python.org
- 2to3 - automated code porting tool
Added to 308, may be addressed sooner - there's lots of factors at play here. There is also no way that I caught every issue we'll encounter on the road to py3.
Supersedes: #194 and #404
Follow-ups: #55
Reactions are currently unavailable
Metadata
Metadata
Labels
A-pythonArea: Python (everything related to the Python interpreter we're using)Area: Python (everything related to the Python interpreter we're using)C-goalCategory: Long term goal. May be code-related or a meta goal.Category: Long term goal. May be code-related or a meta goal.C-internalCategory: Keep track of internal refactoring. Implies the C-todo label except if it is a full C-goalCategory: Keep track of internal refactoring. Implies the C-todo label except if it is a full C-goalM-relnotesMisc: Issue should be listed in the version history for its milestoneMisc: Issue should be listed in the version history for its milestone