Tools

Several tools exist to automate as much of the porting as possible, and to check for common errors. Here is a survey of tools we recommend.

Compatibility library: six

When porting a large piece of software, it is desirable to support both Python 2 and Python 3 in the same codebase. Many projects will need this dual support for a long time, but even those that can drop Python 2 support as soon as the port is done, will typically go through a period of adding Python 3 support, in which the software should continue to work on Python 2.

Benjamin Peterson’s six module makes it practical to write such version-straddling code by offering compatibility wrappers over the differences.

For example, the Python 3 syntax for specifying metaclasses is not valid Python 2, and the Python 2 way does nothing in Python 3, so six provides an add_metaclass decorator for this purpose. It also provides stable names for standard library modules that were moved or reorganized in Python 3.

Six is a run-time dependency, albeit a very small one. If your project is unfriendly to third-party dependencies, push for this one as hard as possible. If you do not use six, you will most likely end up reimplementing it or outright copying relevant pieces of it into your code.

Automated fixer: python-modernize

Some steps of the porting process are quite mechanical, and can be automated. These are best handled by the python-modernize tool – a code-to-code translator that takes a Python 2 codebase and updates it to be compatible with both Python 2 and 3.

Note

python-modernize was built on top of 2to3 from of Python’s standard library. 2to3 was once intended as the main porting tool. It turned out inadequate for that task, but python-modernize (among others) successfully reuses its general infrastructure. Because 2to3 itself is built into Python and thus missing improvements newer than the the Python that runs it, python-modernize now uses a fork of 2to3 called fissix.

Assuming code is in version control, you’ll generally want to run python-modernize with the -wn flags: -w flag causes the tool to actually change the affected files, and -n suppresses creating backups.

The tool operates by applying individual fixers – one for each type of change needed. You can select individual fixers to run using the -f option. We’ve found that running a single fixer at a time results in changes that are easier to review and more likely to be accepted, so that is what this guide will recommend. The order of fixers matters sometimes. This guide will present them in order, but if you skip around, you will need to pay a bit more attention.

The tool always needs a directory (or individual files) to operate on; usually you’ll use the current directory (.).

Combining all that, the recommended invocation is:

python-modernize -wnf <fixer-name> .

While python-modernize is useful, it is not perfect. Some changes it makes might not make sense at all times, and in many cases. It is necessary to know what and why is changed, and to review the result as closely as if a human wrote it. This guide will provide the necessary background for each fixer as we go along.

Compatibility headers and guide for C extensions: py3c

Some projects involve extension modules written in C/C++, or embed Python in a C/C++-based application. An easy way to find these is to search your codebase for PyObject. For these, we have two pieces of advice:

• Even though this is a conservative guide, we encourage you to try porting C extensions away from the Python C API. For wrappers to external libraries we recommend CFFI; for code that needs to be fast there’s Cython.

  While this is relatively disruptive, the result will very likely be more maintainable and less buggy, as well as more portable to alternative Python implementations.

• If you decide to keep your C extension, follow a dedicated porting guide similar to this one, which also comes with a six-like library for C extensions: py3c.

Automated checker: pylint --py3k

Pylint is a static code analyzer that can catch mistakes such as initialized variables, unused imports, and duplicated code. It also has a mode that flags code incompatible with Python 3.

If you are already using Pylint, you can run the tool with the --py3k option on any code that is already ported. This will prevent most regressions.

You can also run pylint --py3k on unported code to get an idea of what will need to change, though python-modernize is usually a better choice here.