Introduction
Python list comprehensions provide a concise way of creating lists. For example, if you want to write code that squares all the numbers from 1 to 10, you would write as below (the traditional way). First, you need to create an empty list, then loop through all the elements, square each element and then append to the list.
# Traditional way
squares = []
for num in range(1,11):
squares.append(num**2)
You can achieve the same result using Python list comprehensions as shown below. As you can see, list comprehensions are readable, concise, and more Pythonic.
# List comprehension
squares = [num**2 for num in range(1,11)]
Understanding list comprehensions don’t stop here. But there is much to learn about list comprehensions. In this comprehensive guide, we’ll dig deeper and understand more about the list comprehensions. Once you go through this guide, you would have mastered list comprehensions in Python. I know you are excited!! Let’s get started.
Lists are one of the sequence types in Python. Refer to our in-depth article here for introduction to sequence types in Python.
List Comprehensions
As mentioned earlier, list comprehensions are concise ways of creating lists. The general syntax of list comprehension looks as below. It consists of three parts: iteration, transformation, and filtering.
squares = [num**2 for num in range(1,11) if num%2==0]
- Iteration: This is the first part that gets executed in a list comprehension that iterates through all the elements in the iterable.
- Transformation: As the name says transformation will be applied to each element in the iterable. In the above example, we are squaring each element from the iterable.
- Filter (optional): This is optional and it filters out some of the elements based on the conditions. The above example loops through all the elements from 1 to 10, filters out odd numbers and considers only positive numbers before applying the transformation.
Let’s see couple more examples of list comprehensions. The below example converts all the strings in the list to upper case.
>>> my_list = ["python", "simplified", "dot", "com"]
>>> [s.upper() for s in my_str]
['PYTHON', 'SIMPLIFIED', 'DOT', 'COM']
Here is another example that displays all the numbers less than 50 that are even and divisible by 7.
>>> my_list = [ num for num in range(50) if i%2==0 and i%7==0]
>>> my_list
[0, 14, 28, 42]
What happens under the hood
Now that you have understood the syntax and how to write list comprehensions, let’s try to understand what goes on under the hood.
Whenever Python encounters a list comprehension, it creates a temporary function that gets executed during the runtime. The result is stored at some memory location and then gets assigned the variable on the LHS. Refer to the diagram below.
You can confirm this by running the list comprehension through the dis module. In the below example, the built-in compile() function generates the bytecode of the list comprehension. Then when you run it through the dis module, it generates byte code instructions as below.
>>> import dis
>>> bytecode = compile("[i**2 for i in [1, 2, 3, 4, 5]]", filename="", mode="eval")
>>> dis.dis(bytecode, depth=0)
Refer to the output of dis below. The bytecode instruction has MAKE_FUNCTION, CALL_FUNCTION. This confirms that Python converts list comprehensions to temporary functions internally.
1 0 LOAD_CONST 0 (<code object <listcomp> at 0x000001D22C82A5B0, file "", line 1>)
2 LOAD_CONST 1 ('<listcomp>')
4 MAKE_FUNCTION 0
6 LOAD_CONST 2 ((1, 2, 3, 4, 5))
8 GET_ITER
10 CALL_FUNCTION 1
12 RETURN_VALUE
Since list comprehensions are internally treated as functions, they exhibit some of the properties of functions. List comprehensions also have local and global scopes. In the below example, num is a local variable and factor is a global variable. But you can still access the global variable factor inside the list comprehension.
>>> factor = 0.25
>>> result = [ num*factor for num in range(5)]
>>> result
[0.0, 0.25, 0.5, 0.75, 1.0]
Nested list comprehensions
It’s possible to write nested list comprehensions meaning you can include a list comprehension within another. One of the uses of nested list comprehensions is that they can be used to create n-dimensional matrices.
In the example below, we are able to create a 5×5 matrix using nested list comprehensions.
>>> my_list = [[i*j for j in range(5)] for i in range(5)]
>>> my_list
[[0, 0, 0, 0, 0],
[0, 1, 2, 3, 4],
[0, 2, 4, 6, 8],
[0, 3, 6, 9, 12],
[0, 4, 8, 12, 16]]
Nested loops in list comprehensions
It’s also possible to include multiple loops in list comprehensions. You can use as many loops as you want. But, as a best practice, you should limit the depth to two so as not to lose the readability. The below example uses two for loops.
>>> my_list = [i+j for i in range(2) for j in range(2)]
>>> my_list
[0, 1, 1, 2]
One of the uses of nested loops is to vectorize the matrix (flattening the matrix). In the below example, my_matrix is a 2×2 matrix. Using nested loops, we are able to create a vectorized version.
>>> my_matrix = [[1, 2, 3],
>>> [4, 5, 6]]
>>> print( [value for row in my_matrix for value in row] )
[1, 2, 3, 4, 5, 6]
Multi-line list comprehensions
Most of the examples you have seen are one-line list comprehensions. But note that you can also write multi-line list comprehensions and are preferred when you can’t fit all the instructions in one line. They just increase the readability and won’t offer any added benefit. See an example below.
>>> my_list = [(i, j, k) for i in range(2)
for j in range(3,5)
for k in range(6,8)]
>>>> my_list
[(0, 3, 6),
(0, 3, 7),
(0, 4, 6),
(0, 4, 7),
(1, 3, 6),
(1, 3, 7),
(1, 4, 6),
(1, 4, 7)]
The same multi-line list comprehension is written in the single line as below. As you can see this is not readable.
my_list = [(i, j, k) for i in range(2) for j in range(2) for k in range(2)]
List comprehensions with conditionals (if-else)
You can also use if-else conditions with list comprehensions. An example of using the if condition was covered in the syntax section. However, let me provide two examples here one with if and another with if-else.
Using if conditional:
The below example uses the if statement to filter out the languages that start with the letter “P”. Notice that if statement comes after the loop. In general, if condition is used to filtering.
>>> prog_langs = ["C", "Java", "Python", "Rust", "Perl", "R", "PHP", "Ruby"]
>>> [lang for lang in prog_langs if lang.startswith("P")]
['Python', 'Perl', 'PHP']
Using if-else conditional:
When the if-else is used with list comprehension, the syntax is a little different compared to comprehension with only if statement. The below example calculates the square even numbers and cube of odd numbers between 1 to 10 using an if-else statement.
>>> [i**2 if i%2==0 else i**3 for i in range(1,11)]
[1, 4, 27, 16, 125, 36, 343, 64, 729, 100]
Do you want to master Python for else and while else? Please read our detailed article here.
List comprehensions with walrus operator
You can create list comprehensions using the walrus operator := and is also called as “assignment expression”. The walrus operator was first introduced in Python 3.8. As mentioned in PEP 572, it was introduced to speedup the coding by writing short code.
The below example creates a list of even numbers between 0 to 10 using the walrus operator.
>>> [s for num in range(10) if (s := num) % 2 == 0]
[0, 2, 4, 6, 8]
List comprehensions Vs map and filter
The map and filter function can also be used as alternatives to list comprehensions as both the functions help us in creating the list object.
map vs. list comprehension
The below example shows that the list comprehension function can be an alternative to the map function.
# Example using map function
>>> list( map(lambda x: x**2, range(1,11)) )
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
# Example using list comprehension
>>> [num**2 for num in range(1,11)]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
The below code also confirms that list comprehensions are faster than the map function.
>>> import timeit
# Time taken for map function
>>> timeit.timeit("list( map(lambda x: x**2, range(1,11)) )", number=10_000_000)
43.036773899999844
# Time taken for list comprehension
>>> timeit.timeit("[num**2 for num in range(1,11)]", number=10_000_000)
35.61648379999997
filter vs. list comprehension
From the below example, it is evident that list comprehensions can be an alternative to the filter function.
# Example using filter function
>>> list( filter(lambda x: x%2==0, range(1,11)) )
[2, 4, 6, 8, 10]
# Example using list comprehension
>>> [num for num in range(1,11) if num%2==0]
[2, 4, 6, 8, 10]
Even in this case, list comprehensions are faster than filter functions. Refer to the below example.
import timeit
# Time taken for filter function
>>> timeit.timeit("list( filter(lambda x: x%2==0, range(1,11)) )", number=10_000_000)
26.987109699999564
# Time taken for list comprehension
>>> timeit.timeit("[num for num in range(1,11) if num%2==0]", number=10_000_000)
17.020929400000114
Set Comprehensions
You can also create set comprehensions but the only difference is they are enclosed within curly brackets { }. The below example creates set comprehension from a list. It converts all the strings in the given list to upper case and creates a set (of course, by removing the duplicates)
>>> names = ["python", "PYTHON", "simplified", "SIMPLIFIED", ".", "com", "COM"]
>>> { name.upper() for name in names }
{'.', 'COM', 'PYTHON', 'SIMPLIFIED'}
Dictionary Comprehensions
Dictionary comprehensions are also enclosed with { } brackets but its contents follow key: value format as you expect. The below example creates a dictionary comprehension from a list. It creates a dictionary with string as key and length of the string as value.
>>> name = ["Python", "Simplified", ".", "Com"]
>>> { s: len(s) for s in name}
{'Python': 6, 'Simplified': 10, '.': 1, 'Com': 3}
Generator Expressions
The generator expressions look very similar to list comprehension with the difference is that generator expressions are enclosed within parenthesis ( ). You could say generator comprehension a lazy version of list comprehension.
In the below example, we created a list of squares of numbers between 1 to 10 that are even-numbered using generator expression. Note that the output of generator expression is a generator object. To get the result, you need to pass the generator object to the list constructor (or to a tuple or set the constructor based on your need).
>>> gen_compr = (num**2 for num in range(1, 11) if num % 2 == 0)
>>> gen_compr
<generator object <genexpr> at 0x000002485C1ED748>
>>> list(gen_compr)
[4, 16, 36, 64, 100]
Generator Expressions Vs. List Comprehensions
a) Storage efficiency: A list comprehension stores the entire list in the memory whereas a generator yields one item at a time on-demand. Hence, generator expressions are more memory efficient than lists. As seen from the below example, the same code generator expression takes far less memory.
# Size with list comprehension
>>> print(sys.getsizeof([i for i in range(10000)]))
87624
# Size with generator expression
>>> print( sys.getsizeof((i for i in range(10000))))
120
b) Speed efficiency: List comprehensions are faster than generator expressions. The same code takes 7 seconds for list comprehensions but 9 seconds with generator expressions.
# Time taken for list comprehensions
>>> timeit.timeit("sys.getsizeof(sum([i for i in range(10000)]))", number=10000)
7.4047205000024405
# Time taken for generator expression
>>> timeit.timeit("sys.getsizeof( sum(i for i in range(10000)) )", number=10000)
9.064736799999082
c) Use list comprehensions if you need to iterate over the list multiple times. If not, consider using generator expressions. Because you won’t be able to iterate generator expression again once you already iterated over. See the example below. When you iterate over a second time, it returns an empty list.
>>> gen_compr = (num**2 for num in range(1, 11) if num % 2 == 0)
>>> list(gen_compr)
[4, 16, 36, 64, 100]
>>> list(gen_compr)
[]
So, which one should you consider? list comprehensions or generator expressions. There is always a trade-off between memory and speed efficiency so you need to choose based on the requirements.
Do we have tuple comprehensions?
No, there are no tuple comprehensions! As we have gone through earlier, list comprehensions loop through each item in the iterable and append to the list under the hood. You are mutating the list by appending elements to it. List, set and dictionary comprehensions exist because they are mutable. But tuple is an immutable object. Hence, we don’t have tuple comprehensions.
Refer to this detailed discussion on the same subject on Stack Overflow.
Summary
- List comprehensions provide a concise way of creating lists.
- Apart from list comprehensions, Python also has set comprehensions, dictionary comprehensions, and generator expressions.
- Under the hood, list comprehensions converted to equivalent functions.
- List comprehensions come in different formats — nested list comprehensions, with nested loop, with multi-lines, with walrus operator, with if and if-else conditionals, etc.
- List comprehensions with if conditional is used for filtering while if-else is used when you want to perform different transformations based on the conditions.
- List comprehensions can be a replacement for map, filter, reduce, and for loop. In most cases list comprehensions is faster than all these methods.
- If you need to use more than two for loops in the comprehension then it’s better to use the traditional way of creating a list. Because using more than two for loops is not easy to readable.
- Consider using generator expressions instead of list comprehension if you are dealing with large data and not iterating over the list multiple times.
- Comprehensions are supposed to be concise, clear, and Pythonic. Don’t write complicated lengthy comprehensions that will defeat the very purpose of using them.
