PYTHON / ITERATORS, GENERATORS, AND COMPREHENSIONS
The iterator protocol
Use iter() and next() directly to drive any Python iteration by hand, and explain the iterable/iterator split that for loops rely on.
What you will learn
- Desugar a for loop into iter(), repeated next(), and StopIteration
- Tell an iterable apart from an iterator by whether iter(x) is x
- Use next(it, default) to avoid try/except around a single step
- Recognise iter()'s two argument form and its __getitem__ fallback
Understanding The iterator protocol
Python's iteration is built on two small contracts. An iterable is any object whose __iter__ returns a fresh iterator; an iterator is any object with __next__, which hands back the next item or raises StopIteration when there is nothing left. An iterator must also implement __iter__ returning itself, which is exactly why you can pass a half-consumed iterator straight to a for loop.
A for loop is not magic: it calls iter() on the object once, then calls next() on the result over and over, and treats StopIteration as the signal to stop rather than as an error. That is why StopIteration inherits from Exception but never reaches your code in normal loops, and why leaking one from inside a callback can silently truncate a loop. Understanding this desugaring lets you read builtins like zip, min, and sum as thin wrappers around next().
The consequence that trips people up is that iterators are one-shot and stateful. A list can be looped over any number of times because each iter(list) call builds a brand new list_iterator holding its own position; an iterator has only one position, so once exhausted it stays exhausted and every later next() raises StopIteration again. When a function accepts "any iterable", it must not iterate the argument twice unless it materialises it first.
nums = [10, 20, 30]
it = iter(nums)
print(type(it).__name__)
print(iter(it) is it)
print(iter(nums) is iter(nums))
while True:
try:
value = next(it)
except StopIteration:
print('exhausted')
break
print(value)
print(next(it, 'default'))for loops are shorthand for iter() plus repeated next() with StopIteration as the stop signal, and iterators carry their own single position.
Worked examples
One-shot iterators versus reusable iterables
Shows that a shared iterator is empty on the second pass while the underlying list is not.
data = [1, 2, 3]
shared = iter(data)
print([x for x in shared], [x for x in shared])
print([x for x in data], [x for x in data])Example explained
Line 1The first comprehension drives shared to exhaustion, so its position sits past the end.
Line 2The second comprehension calls iter(shared), gets shared back, and immediately sees StopIteration.
Line 3Each comprehension over data calls iter(data), which builds a new list_iterator starting at index 0.
iter() with a sentinel
Demonstrates the two argument form, which turns a repeatedly called function into an iterator.
import io
stream = io.StringIO('ab cd ef')
for chunk in iter(lambda: stream.read(3), ''):
print(repr(chunk))Example explained
Line 1iter(callable, sentinel) returns an iterator that calls the callable with no arguments on each next().
Line 2Iteration stops as soon as a returned value equals '' , the sentinel, so the empty read at end of stream ends the loop.
Line 3The final chunk is only two characters because read(3) returns what is left rather than padding.
The __getitem__ fallback
Shows that iter() can build an iterator for an object that has no __iter__ at all.
class Countdown:
def __getitem__(self, index):
if index > 2:
raise IndexError(index)
return 3 - index
c = Countdown()
print(list(c))
print(next(iter(c)))
print(hasattr(c, '__iter__'))Example explained
Line 1iter() finds no __iter__, so it wraps the object in a legacy sequence iterator that probes indices 0, 1, 2, ...
Line 2IndexError from __getitem__ is translated into StopIteration, which is what stops list().
Line 3hasattr(c, '__iter__') is False even though the object iterates fine, so that check is not a reliable iterability test.
Important notes
next(it, default) suppresses only StopIteration; a TypeError or ValueError raised inside __next__ still propagates.
isinstance(x, collections.abc.Iterable) checks for __iter__ only, so it returns False for __getitem__-style objects that iterate perfectly well; try: iter(x) is the honest test.
Common mistakes
Calling next(my_list) directly, which raises TypeError: 'list' object is not an iterator, because lists have __iter__ but no __next__.
Passing the same iterator to two loops or to zip(it, it) and being surprised by empty or interleaved results, since both consumers share one position.
Wrapping a whole block in except StopIteration, so an unrelated exhausted iterator inside the block quietly ends iteration instead of reporting a bug.
Try it yourself
Change, predict, then run
Write the while-loop equivalent of for ch in 'abc': print(ch) using only iter, next, and try/except StopIteration, then call next() once more with a default of 'END' and print it to confirm the iterator stays exhausted.
Open the Python workspaceCheck your understanding
With it = iter([1, 2, 3, 4]), why does list(zip(it, it)) give [(1, 2), (3, 4)]?
- zip calls next() on each of its arguments in turn, and both arguments are the same iterator, so each tuple consumes two items from one position.
- zip calls iter() on each argument, which restarts the underlying list from the beginning for the second argument.
- zip pairs neighbouring items automatically when it is given fewer than two distinct sequences.
- The list iterator caches the item it just returned and replays it to the second argument.
Show answer
zip builds each tuple by calling next() once per argument; with the same iterator twice, the first call yields 1 and the second yields 2, then 3 and 4. Option two is tempting because zip really does call iter() on its arguments, but iter() on an iterator returns that same object, so nothing restarts.