Introduction
As a Python developer, you hear these two terms iterables and iterators a lot. You are actually using them in your day-to-day work but do you know the difference between iterables and iterators? What happens under the hood? Whether you are a beginner or intermediate in Python, this article is for you.
In this article, we’ll explore both iterables and iterators in great detail.
Iterables
Iterables are objects that can be iterated over and are capable of returning one item at a time. For example, lists, tuples, sets, dictionaries, strings, range, etc are iterables. Because, as we just read in the definition, we can iterate over each item in these objects.
In the below code sample, we are able to iterate over a list and a string using for loop.
# list iterable
>>> for num in [1,2,3]:
... print(num)
1
2
3
# String iterable
>>> for num in "Python":
... print(num)
P
y
t
h
o
n
We can also define iterable as an object that implements the __iter__() dunder method. You can verify if the given object is iterable or not by checking the output of dir(iterable_obj). For example, if you can check the output of dir(list), you will notice that list implements the __iter__() method.
Iterators
Iterators are the objects that implement iterator protocol which consists of __iter__() and __next__() dunder methods. When you pass iterable to __iter__(), it returns the iterator object, and then using __next__() method you can iterate over all the elements. We will discuss this in detail shortly.
As mentioned earlier, you can check the output of dir(object), to see if the given object is iterable or iterator.
The above definition of iterable, iterator and the dunder methods will be a little daunting at first. But don’t panic. We have got you covered. As you go through the rest of the article, you will gain a better understanding of them. Now, let’s understand the dunder methods first.
__iter__()
Consider a my_list example as shown below. We know that list is a iterable and is capable of returning one element at a time. But a list or any iterable by itself won’t be able to iterate over each item. To iterate over each element in my_list, we need an iterator object. So, we use the iter() function or __iter__() method on my_list to generate an iterator object.
The __iter__() method takes an iterable object such as a list and returns an iterator object. Below is the syntax of using __iter__() method or iter() function.
To recap, List, tuples, dictionaries, sets, strings, etc. are all examples of iterables.
iter_object = iter(iterable)
OR
iter_object = iterable.__iter__()
Now, refer to the output of dir(my_iter). As you can see my_iter object has both __iter__() and __next__() and now we can say that it is an iterator object.
__next__()
The __next__() method is used to access each element in the iterator.
In the previous section, we created an iterator object. Now, by repeatedly calling next() on the iterator object, we can access all the elements in the iterable. After all the elements are exhausted, next() will raise the StopIteration exception. Note that you can also use next(my_iter) which internally calls my_iter.__next__().
>>> next(my_iter)
1
>>> next(my_iter)
2
>>> next(my_iter)
3
>>> next(my_iter)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Don’t worry. You don’t have to implement __iter__() and __next__() dunder methods unless you are creating a custom iterator of your own. For example, when using iterables with for loop, you will not implement __iter__() and __next__() as it will be handled by Python under the hood.
Uses of iterables/iterators
Now you know the difference between iterables and iterators and how it works, let’s look into some of the real world use cases.
For loop
The for loop is the most commonly used with iterables such as lists, tuples, dictionaries, range, etc.
The below code iterate through all the elements in the list which is iterable and prints each element. But do you know how is it working without creating an iterator? Well, Python does the heavy lifting for you under the hood.
my_list = [1, 2, 3]
for num in my_list:
print(num)
When Python encounters a for loop, it does the following:
- Calls iter() to create an iterator object for my_list
- Calls next() repeatedly to obtain each item from the iterator and
- Terminates the loop when next() raises the StopIteration exception
Built-in functions that return an iterator
The built-in functions map, zip, filter, etc. return iterator objects. You can confirm this by checking the output of dir(object). By passing the output of the map, zip, and filter function to the list you can get the expected result as a list. Or you can even use for loop to get all the elements.
>>> map_obj = map(lambda x: x**2, [1,2,3])
>>> map_obj
<map object at 0x00000203D8C142B0>
>>> dir(map_obj)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__',
'__ge__', '__getattribute__', '__gt__', '__hash__','__init__','__init_subclass__',
'__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> list(map_obj)
[1, 4, 9]
Built-in functions that support the iterator
There are some built-in functions such as min, max, sum, sorted, etc that take iterator as an argument and return the value. Refer to the below examples.
>>> iter_obj = map(lambda x: x**2, [1,2,3])
>>> min(iter_obj)
1
>>> iter_obj = map(lambda x: x**2, [1,2,3])
>>> max(iter_obj)
9
>>> iter_obj = map(lambda x: x**2, [1,2,3])
>>> sum(iter_obj)
14
Membership operators
The membership operators in and not in also support iterators. The below code is self-explanatory. If an element is found in the iterable, the in operator returns True otherwise False. The not in operator works in the opposite way.
>>> "t" in "Python"
True
>>> 30 in [10, 20, 30]
True
Data types that support iterators
The data types such as string, list, tuple, set, dictionary, and file objects support iterators.
# tuple contructor supports iterator
>>> mylist = [1, 2, 3]
>>> tuple(iter(mylist))
(1, 2, 3)
# list constructor supports iterator
>>> mytuple = (1, 2, 3)
>>> list(iter(mytuple))
[1, 2, 3]
# set constructor supports iterator
>>> myset = {1, 2, 3}
>>> list(iter(myset))
[1, 2, 3]
# dictionary constructor supports iterator
>>> mydict = {"Python": 1, "Javascript": 2, "Java":3}
>>> list(iter(mydict))
['Python', 'Javascript', 'Java']
Generators
Generators are just another way of creating iterators. Generators are implemented using a function and yield keyword. In the below example, my_gen is a generator function and gen is a generator object as you can see. Then you can iterate over each element using the next() method or for loop etc.
>>> def my_gen(num):
... for i in range(num):
... yield i
...
>>> gen = my_gen(3)
>>> type(gen)
<class 'generator'>
>>> gen.__next__()
0
>>> gen.__next__()
1
>>> gen.__next__()
2
>>> gen.__next__()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
You might have a question what is the difference between an iterator and a generator. We’ll be covering this topic in a future article. Stay tuned.
Sequence unpacking
Iterators are also useful in unpacking. Refer to the below code for an example. As you know map returns an iterator object map_iter and is used to unpack the elements into a, b, and c.
>>> map_iter = map(lambda x: x**2, [1,2,3])
>>> a, b, c = map_iter
>>> a
1
>>> b
4
>>> c
9
These are only some of the examples where iterables and iterators find their usefulness. We won’t be able to cover everything in one article. Nevertheless now you know a lot more about iterables and iterator than before.
Build custom iterator
By now, I am sure you have got a better understanding of iterable, iterator, and uses of __iter__() & __next__() methods. How about we build one custom iterator to strengthen our understanding.
In the below code, we have created a class named Square that takes a number num and returns the square of all the numbers ≤ num. As you can see we have implemented __iter_() and __next__() methods which confirm that it’s an iterator.
class Square:
def __init__(self, num):
self.num = num
self.index = 1
def __iter__(self):
return self
def __next__(self):
if self.index <= self.num:
result = self.index ** 2
self.index += 1
return result
else:
raise StopIteration
Next, using the for loop you can iterate over all the elements as shown below.
>>> sq = Square(5)
>>> for i in sq:
... print(i)
1
4
9
16
25
Infinite Iterator
So far we have seen iterator examples that are finite. But it’s also possible to create infinite iterators. This can be done with the help of itertools module. To keep this article short, we’ll cover this in a separate article on the itertools module.
Summary
- Iterables are the objects that can be iterated over. And iterables are the objects are used to iterate over the iterables.
- Iterable objects implement only __iter__ dunder method where as iterators implement both __iter__ and __next__ methods
- The iter() function or __iter__() method takes an iterable and returns an iterator.
- The next() function or __next__() method can be used to iterate over all the elements in the iterator.
- Iterables and iterators find its application with for loop, functions that return iterators, functions that support iterators, membership operators, generators, sequence unpacking, etc.
- One can build custom iterator by implementing __iter__() and __next__() method in the class.
