Other Changes

This page documents a few miscellaneous topics at the edges of this guide’s scope: low-level buffers, doctests, and bytecode cache files.

Raw buffer protocol: buffer and memoryview

  • Fixer: None
  • Prevalence: Very rare

Python 2 used a buffer interface for sharing blocks of memory between Python objects and other libraries.

The buffer object and the corresponding C API proved inaequate, and over Python 2.6 and 2.7, a new mechanism was implemented: the Py_buffer structure and the memoryview object.

In Python 3, the buffer object and the related C API is removed.

Unfortunately, the specifics of low-level interfaces between Python and non-Python libraries are too different across projects for us to offer universal advice on porting to the new API. If your code uses buffer (or the PyBuffer C API), you will need to refer to the Python documentation for details, and combine that with knowledge about your particular interface.

Doctests

  • Fixer: None
  • Prevalence: Common

Doctests are a common practice for quick tests and testing documentation. They work by extracting snippets example code and its output in documentation, running the code, and verifying that its result matches, textually, the example output.

This relies on minute details of textual representation of Python objects, which is generally not under any backwards-compatibility guarantee and may change at any time – even across minor versions of Python (e.g. 2.6 to 2.7 or 3.5 to 3.6).

Note

Some examples of what changed between Python 2 and 3 are:

  • String have different u and b prefixes depending on if they’re bytes or text.
  • Large integers lost the L suffix.
  • The order of items in dictionaries may be different (and unpredictable).

Doctests are a good way to ensure that the documentation is correct (i.e. it doesn’t contain broken examples), but they are not a good way to actually test the code.

If your code uses doctests as the main means of testing, rewrite them as tests that do not rely on exact textual output. You can use the built-in unittest, or the third-party pytest library, among others.

Once your doctests are only testing documentation, we recommend the following strategy:

  • Keep running doctests under Python 2
  • Port all code to be compatible with (and tested on) both Python 2 and 3
  • At one moment, update examples in the docs, and start only using Python 3 to run the doctests.

Since the code is tested elsewhere, it generally does not matter that code examples are tested under only one of the supported Python versions.

Reorganization of .pyc files

Since compiling Python code is a relatively expensive operation, and many modules do not change often, Python caches compiled bytecode in .pyc files.

In Python 2, .pyc files were written in the same directory as the corresponding .py source files, with only a c added to the filename. The exact mechanism had two major drawbacks:

  • Bytecode is not compatible across Python versions. If the same module was being imported by different versions of Python, each would overwrite the .pyc file with its own flavor of bytecode on import. This would invalidate the cache for all other versions.
  • The .pyc cache could be used even without a corresponding .py file, which allowed some space saving (by distributing only the compiled file). However, if one deleted a .py file but forgot to also remove the .pyc, Python would act as if the module was still present. This was quite confusing, especially for beginners.

Python 3 puts .pyc files in a separate directory called __pycache__, and adds version information to the filename. The new mechanism avoids the above two problems: per-version caches are separated, and if the .py source is missing, the .pyc file is not considered.

If your code relies on the location of .pyc files (for example, if your build/packaging system doesn’t handle Python 3), you will need to update to the new location.

If you rely on importing .pyc files without corresponding source, you will need to move the .pyc to the old location, and remove the version tag. For example, move:

__pycache__/foo.cpython-36.pyc

to:

foo.pyc

Under this name, the .pyc will be recognized by Python 3’s import machinery.