Iterators¶
Python 3 changes return values of several basic functions from list to iterator. The main reason for this change is that iterators usually cause better memory consumption than lists.
If you need to keep Python2-compatible behavior, you can wrap the affected
functions with a call to list
. However, in most cases it is better
to apply a more specific fix.
New behavior of map()
and filter()
¶
- Fixers (See caveat below):
python-modernize -wnf libmodernize.fixes.fix_map
python-modernize -wnf libmodernize.fixes.fix_filter
- Prevalence: Common
In Python 3, the map()
and filter()
functions return
iterators (map
or filter
objects, respectively).
In Python 2, they returned lists.
In Python 2, the iterator behavior is available as itertools.imap()
and itertools.ifilter()
.
The Compatibility library: six library provides the iterator behavior under names common to
both Python versions: from six.moves import map
and
from six.moves import filter
.
Higher-order functions vs. List Comprehensions¶
The map
and filter
functions are often used with lambda
functions
to change or filter iterables. For example:
numbers = [1, 2, 3, 4, 5, 6, 7]
powers_of_two = map(lambda x: 2**x, numbers)
for number in filter(lambda x: x < 20, powers_of_two):
print(number)
In these cases, the call can be rewritten using a list comprehension, making the code faster and more readable:
numbers = [1, 2, 3, 4, 5, 6, 7]
powers_of_two = [2**x for x in numbers]
for number in [x for x in powers_of_two if x < 20]:
print(number)
If named functions, rather than lambda
, are used, we also recommend
rewriting the code to use a list comprehension.
For example, this code:
def power_function(x):
return(2**x)
powered = map(power_function, numbers)
should be changed to:
def power_function(x):
return(2**x)
powered = [power_function(num) for num in numbers]
Alternatively, you can keep the higher-order function call, and wrap the
result in list
.
However, many people will find the resulting code less readable:
def power_function(x):
return(2**x)
powered = list(map(power_function, numbers))
Iterators vs. Lists¶
In cases where the result of map
or filter
is only iterated over,
and only once, it makes sense to use a generator expression rather than
a list. For example, this code:
numbers = [1, 2, 3, 4, 5, 6, 7]
powers_of_two = map(lambda x: 2**x, numbers)
for number in filter(lambda x: x < 20, powers_of_two):
print(number)
can be rewritten as:
numbers = [1, 2, 3, 4, 5, 6, 7]
powers_of_two = (2**x for x in numbers)
for number in (x**2 for x in powers_of_two if x < 20):
print(number)
This keeps memory requirements to a minimum. However, the resulting generator object is much less powerful than a list: it cannot be mutated, indexed or sliced, or iterated more than once.
Fixer Considerations¶
When the recommended fixers detect calls to map()
or filter()
, they add
the imports from six.moves import filter
or from six.moves import map
to the top of the file.
In many cases, the fixers do a good job discerning the different usages of
map()
and filter()
and, if necessary, adding a call to list()
.
But they are not perfect.
Always review the fixers’ result with the above advice in mind.
The fixers do not work properly if the names map
or filter
are rebound to something else than the built-in functions.
If your code does this, you’ll need to do appropriate changes manually.
New behavior of zip()
¶
- Fixer:
python-modernize -wnf libmodernize.fixes.fix_zip
(See caveat below) - Prevalence: Common
Similarly to map
and filter
above, in Python 3, the zip()
function returns an iterator (specifically, a zip
object).
In Python 2, it returned a list.
The Compatibility library: six library provides the iterator behavior under a name common to
both Python versions, using the from six.moves import zip
statement.
With this import in place, the call zip(...)
can be rewritten to
list(zip(...))
.
Note, however, that the list
is unnecessary when the result is only
iterated over, and only iterated once, as in for items in zip(...)
.
The recommended fixer adds the mentioned import, and changes calls to
list(zip(...)
if necessary.
If you review the result, you might find additional places where conversion
to list
is not necessary.
The fixer does not work properly if the name zip
is rebound to something else than the built-in function.
If your code does this, you’ll need to do appropriate changes manually.
New behavior of range()
¶
- Fixer:
python-modernize -wnf libmodernize.fixes.fix_xrange_six
(See caveat below) - Prevalence: Common
In Python 3, the range
function returns an iterable range
object, like the xrange()
function did in Python 2.
The xrange
function was removed in Python 3.
Note that Python 3’s range
object, like xrange
in Python 2,
supports many list-like operations: for example indexing, slicing, length
queries using len()
, or membership testing using in
.
Also, unlike map
, filter
and zip
objects, the range
object
can be iterated multiple times.
The Compatibility library: six library provides the “xrange
” behavior in
both Python versions, using the from six.moves import range
statement.
Using this import, the calls:
a_list = range(9)
a_range_object = xrange(9)
can be replaced with:
from six.moves import range
a_list = list(range(9))
a_range_object = range(9)
The fixer does the change automatically.
Note that in many cases, code will work the same under both versions
with just the built-in range
function.
If the result is not mutated, and the number of elements doesn’t exceed
several thousands, the list and the range behave very similarly.
In this case, just change xrange
to range
; no import is needed.
If the name range
is rebound to something else than the built-in
function, the fixer will not work properly.
In this case you’ll need to do appropriate changes manually.
New iteration protocol: next()
¶
- Fixer:
python-modernize -wnf libmodernize.fixes.fix_next
(See caveat below) - Prevalence: Common
In Python 3, the built-in function next()
is used to get the next
result from an iterator.
It works by calling the __next__()
special method,
similarly to how len()
calls iterator.__len__
.
In Python 2, iterators had the next
method.
The next()
built-in was backported to Python 2.6+, where it calls the
next
method.
When getting items from an iterator, the next
built-in function should be
used instead of the next
method. For example, the code:
iterator = iter([1, 2, 3])
one = iterator.next()
two = iterator.next()
three = iterator.next()
should be rewritten as:
iterator = iter([1, 2, 3])
one = next(iterator)
two = next(iterator)
three = next(iterator)
Another change concerns custom iterator classes.
These should provide both methods, next
and __next__
.
An easy way to do this is to define __next__
, and assign that function
to next
as well:
class IteratorOfZeroes(object):
def __next__(self):
return 0
next = __next__ # for Python 2
The recommended fixer will only do the first change – rewriting next
calls.
Additionally, it will rewrite calls to any method called next
, whether
it is used for iterating or not.
If you use a class that uses next
for an unrelated purpose, check the
fixer’s output and revert the changes for objects of this class.
The fixer will not add a __next__
method to your classes.
You will need to do this manually.
Generators cannot raise StopIteration
¶
- Fixer: None
- Prevalence: Rare
Since Python 3.7, generators cannot raise StopIteration
directly,
but must stop with return
(or at the end of the function).
This change was done to prevent subtle errors when a StopIteration
exception “leaks” between unrelated generators.
For example, the following generator is considered a programming error,
and in Python 3.7+ it raises RuntimeError
:
def count_to(maximum):
i = 0
while True:
yield i
i += 1
if i >= maximum:
raise StopIteration()
Convert the raise StopIteration()
to return
.
If your code uses a helper function that can raise StopIteration
to
end the generator that calls it, you will need to move the returning logic
to the generator itself.