How Iterables actually work in Python

One of the more impressive features of Python language is the use of “for-each” looping construct that has awed me ever since I started out with Python. For the uninitiated, here is a simple loop which prints the first 10 natural numbers: for for num in range(1, 11):print(num) We can also loop over the primitive types such as and in similar ways: list, tuples, dictionaries strings numbers = [1, 2, 3, 4, 5]record = ('Kshitij', 21, 'Loves Python')details = {'name': 'Kshitij','age': 21} for num in numbers:print(num) # 1 2 3 4 5 for data in record:print(data) # Kshitij 21 Loves Python for key, value in details.items():print(key, value) # age 21 name Kshitij As one implements few data structures in Python using , he feels the desire to loop over the data stored in it’s instances. This is where the Iterator Protocol comes into picture. class Sample Implementation Let us suppose we are tasked with implementing a . A sample implementation might look something like this: standard 52-card deck This works fine with regards to creating new instances of and representing it. However, a major pain point in this implementation is the lack of ability to iterate over the object. Deck Deck >>> from cards import Deck>>> new_deck = Deck() # New deck instantiated>>> print(new_deck)... # Works great>>> for card in new_deck:... print(card) TypeError: 'Deck' object is not iterable One can be smart enough to explore the instance and conclude that the attributes holds the data required for iterations and it, in fact is a . With this knowledge, he can hack the above loop as follows: new_deck cards list >>> for card in new_deck.cards:... print(card)Card(...).... This code works great. However, the end user must attain the internal information about the implementation to perform the iteration. This makes our code lose the advantages of data abstraction and leads much to be desired of the implementation. There must be a better way! Urged by the enthusiasm from Raymond Hettinger, I searched for ways to improve my implementation to couple with the Python’s loop. for And soon I found the answer — The Iterator Protocol. The Iterator Protocol In order to learn what the Protocol is and how to implement it in Python, we need to understand some basic terms. Iterable It is any object that . you can loop over with a for loop are not always indexable, they don’t always have lengths, and they’re not always finite. Iterables An iterable can be passed to built-in function to get an iterator for them. iter() Iterator Iterators have exactly one job: return the “next” item in our iterable. can be passed to the built-in function to get the next item from them and if there is no next item (because we reached the end), a exception will be raised. Iterators next StopIteration Iterators return themselves when passed to the built-in. iter() The Protocol Step 01: How the built-in works? iter() Whenever the interpreter needs to iterate over an object , it automatically calls . The built-in function: x iter(x) iter Checks whether the object implements, method and calls that to obtain an __iter__ iterator. If method is not implemented, but method is implemented, Python creates an that attempts to fetch items in order, starting from index . __iter__ __getitem__ iterator 0 If that fails, Python raises exception saying . TypeError object is not iterable Step 02: How to implement the Protocol? I will present two approaches to implementing the Iterator Protocol: Approach 1: Traditional Way Create a new class representing the (say ) iterator DeckIterator . Implement the following two methods in DeckIterator: : returns the next item in the iterable. __next__ : returns itself i.e . __iter__ self 3. Define an method in the class over whose instances you want to iterate i.e. class The method should return an instance of . __iter__ Deck. DeckIterator Approach 2: The Pragmatic Way Implement the method in the class as a . __iter__ Deck generator function Features This is the list of all the features that our object magically seem to support as soon as we implement the protocol. Iteration via for loop Unpacking similar to tuples Can be used in list comprehensions Can be used with built-in functions (such as , ) which consume an iterable. min max >>> new_deck = Deck() >>> # 1. Looping through a for loop>>> for card in new_deck:... print(card) # Works great! >>> # 2. Unpacking similarly to tuples>>> first_card, *rest, last_card = new_deck >>> # 3. List Comprehensions>>> spades = [card for card in new_deck if card.suit == 'Spades'] >>> # 4. Built-in functions>>> max_card, min_card = max(new_deck), min(new_deck) Lessons Learnt: Iterators in Python aren’t a matter of type but of the protocol i.e. any class that implements this protocol can be iterated over. Python iteration. groks I hope that the knowledge of the Iterator Protocol will help you out when writing Python. In order to raise awareness about this seemingly under appreciated feature of Python, I have proposed a on this topic. talk at PyCon India 2017