Skip to content

Python 3 Upgrade #460

@Infernio

Description

@Infernio

Goold old Python 2 is dead in about 3 months as of the time of writing.

What follows is a list of roadblocks:

  • chardet. We bundle version 1.0.1, from 2008. May just happen to work on py3, but I wouldn't hold my breath - done in 8e201c4.
  • wxPython. Neither 2.8 nor 3.0 have Python 3 versions available. However, the work put into the wx3 upgrade was not wasted; it will make upgrading to wxPython 4.0, which does have Python 3 support, much easier - see 050391c, done in 22de7ff.
  • loot-api-python. Python 3 versions are available starting from version 4.0, but see Replace libloot-python with handwritten parser #431 for the obvious problem there - done in 934f352.
  • Records Refactoring #480 and Patchers Refactoring #312 are blockers, for the most part - will be much easier to proceed to Python 3 once they've progressed further
  • Sub-issues:
  • Well, the entire codebase. A decent chunk can probably be handled by automated tools, but we need to consider the output carefully - plus automated tools won't be nearly enough to catch everything. In particular:
    • Unicode strings and byte strings
      • ur'' is gone - done in 660ecbb and d43ad24
      • We should go through all our unprefixed string usages and add either b'' or u'' - done in way too many commits to list
      • StringIO needs to be changed to BytesIO where applicable - done in 7c95d05
      • str needs to be replaced with bytes or unicode depending on how it's used - done in 04835ec
      • basestring is problematic - it's gone in py3, but 2to3 will change it to str. Most usages should be replaced with unicode or bytes, but a couple really do need to handle both - done in 42b879f
      • Revisit all open uses, check if opening in bytes mode is needed - done in c120e18
    • We need to get rid of getattr calls that pass in bytes, most notably due to ModFile.__getattr__ - Patchers Refactoring #312/Records Refactoring #480, (hopefully) done in 0848872
    • Tuple parameter unpacking is gone in py3. The only place we use this is in lambdas, which is handled by 2to3 - done in 7a6b7c2
    • zip/map/filter/range need to get replaced with either the itertools/xrange variant or an equivalent comprehension - plus add list() calls if we modify the iterables in the loop - worked on in 709e146 and e8c4d43, done in a75dbc6
    • All keys and iterkeys calls can go -> done in 567f153 and 09e1633
    • items and values: Change to iteritems/itervalues, then run 2to3's dict fixer. If it changes them to iter(...), change them to viewitems/viewvalues instead - done in d22469b
    • Old-style classes need to be switched to new-style classes ahead of time to catch any issues that arise - see the commit message in fa74a1b, for example - worked on in cae844b, done in 6db5b8b
    • True division updating - for each usage, we need to consider if we want floats or integers - done in 505d26f
    • Relative imports - are we using the explicit ones everywhere already? Should mostly be caught by automated tools, but problems can arise here - done in c692662
    • Longs are gone, replaced with regular integers - a bunch of unnecessary usages in records code, which will be dropped in the course of Records Refactoring #480, any other usages should be investigated - worked on in d8d03ca, done in 197b6a2
    • ints no longer have a size limit, so sys.maxint is gone - depending on the use case, could be replaced by sys.maxsize - done in 659e5b6
    • cmp and __cmp__ are gone; usage is rare - five in ScriptParser and belt and one each in cint, balt and mods_metadata - done in 5a98eb4'

Things that are not roadblocks:

  • comtypes - dropped during wxPython 3.0.2.0 upgrade
  • gitpython - dropped in build script rework, see pygit2 below
  • py2exe - readily supports Python 3 - we should however consider our options once we get there, see Replace py2exe with PyInstaller #491
  • pyfiglet - readily supports Python 3
  • pygit2 - readily supports Python 3
  • pymupdf - readily supports Python 3
  • pytest - readily supports Python 3
  • python-lz4 - readily supports Python 3
  • pywin32 - readily supports Python 3
  • pyyaml - readily supports Python 3
  • scandir - built into Python 3, we'll just drop this once we upgrade
  • toml - readily supports Python 3

Useful tracking regexes (make sure to enable *.py mask):

  • ur prefix: ur('|")[^,)]
  • Old-style classes: class [^:(]+:
  • Old-style super initializer calls: (?<!self\))\.__init__
  • WIP regex for all old-style super calls, has tons of false positives: [A-Z]\w+\.\w+\(self(,|\))
  • Absurd regex for tracking non-library imports that need updating: ^ *import (?!os|sys|re|errno|time|subprocess|io|threading|wx|wx\.adv|cPickle|argparse|atexit|codecs|ctypes|platform|shutil|traceback|codecs|chardet|lz4|win32api|win32com|yaml|tempfile|Tkinter|collections|copy|csv|copy|datetime|stat|string|struct|textwrap|scandir|inspect|pkgutil|math|gettext|locale|msgfmt|pygettext|operator|webbrowser|array|zlib|binascii|_winreg|win32gui|importlib|random|toml|pytest|glob|logging|zipfile|pygit2|py2exe)
  • open usages without false positives: (?<!webbrowser\.)(?<!io\.)(?<!def )\bopen\b

Some resources:

Added to 308, may be addressed sooner - there's lots of factors at play here. There is also no way that I caught every issue we'll encounter on the road to py3.
Supersedes: #194 and #404
Follow-ups: #55

Metadata

Metadata

Labels

A-pythonArea: Python (everything related to the Python interpreter we're using)C-goalCategory: Long term goal. May be code-related or a meta goal.C-internalCategory: Keep track of internal refactoring. Implies the C-todo label except if it is a full C-goalM-relnotesMisc: Issue should be listed in the version history for its milestone

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions