PYTHON / ITERATORS, GENERATORS, AND COMPREHENSIONS
Generator functions and yield
Write functions that pause at yield and resume where they left off, and know exactly when the body runs and when iteration stops.
What you will learn
- Recognise that a function containing yield returns a generator without running its body
- Drive a generator with next() and for, knowing precisely where execution resumes
- Carry state in ordinary local variables that survive across every yield
- Stop a generator early with return and read the value off StopIteration
Understanding Generator functions and yield
The presence of the word yield anywhere in a function body changes what calling that function means. Python compiles it into a generator function: calling it allocates a generator object wrapping a paused stack frame and returns immediately, executing not one line of the body. That is why print statements, argument validation, or file opening at the top of a generator function appear to be skipped — they are only waiting for the first next().
Each next(gen) resumes that frame and lets it run until it hits a yield; the yielded expression becomes the return value of next(), and the frame freezes again with its local variables and its position in the code intact. This is the whole mental model: a generator is a function you can stop in the middle and continue later. Loop counters, accumulators and half-built strings just stay in locals, so you never write the manual state bookkeeping a hand-rolled iterator class needs.
The generator ends when control falls off the end of the body or hits a return, at which point Python raises StopIteration, which for loops catch and treat as the end of the sequence. A bare return ends it with value None; return x attaches x to the exception as StopIteration.value, so the value is invisible to a for loop and must be read explicitly. Once the frame has finished it cannot be rewound: a generator object is single-use, and getting a fresh pass means calling the generator function again.
def countdown(n):
print(f"body starts, n={n}")
while n > 0:
yield n
n -= 1
print("body ends")
gen = countdown(3)
print("created:", type(gen).__name__)
print("first:", next(gen))
print("second:", next(gen))
for value in gen:
print("loop:", value)
print("exhausted:", list(gen))yield turns a function into a resumable frame, so calling it only builds a generator and each next() runs one slice of the body while keeping the locals alive.
Worked examples
Locals survive between yields
Shows that an accumulator declared once keeps its value across suspensions, and that exhaustion is signalled by StopIteration.
def running_total(numbers):
total = 0
for n in numbers:
total += n
yield total
print(list(running_total([3, 1, 4, 1, 5])))
gen = running_total([10, 20])
print(next(gen))
print(next(gen))
try:
next(gen)
except StopIteration:
print("StopIteration raised")Example explained
Line 1total = 0 runs once, on the first next(), not once per yielded value.
Line 2After yield total the frame freezes with total still bound, so the next resume adds to 30 rather than restarting at 20.
Line 3list() drives the generator to completion by calling next() until StopIteration, then returns the collected values.
Line 4The third next() finds the for loop over numbers finished, falls off the end of the body, and raises StopIteration.
Ending early with return
Demonstrates that return inside a generator stops iteration and hides its value inside StopIteration.
def take_until_blank(lines):
count = 0
for line in lines:
if not line.strip():
return count
count += 1
yield line.upper()
for text in take_until_blank(["alpha", "beta", "", "gamma"]):
print(text)
gen = take_until_blank(["x", "", "y"])
print(next(gen))
try:
next(gen)
except StopIteration as stop:
print("returned:", stop.value)Example explained
Line 1return count is legal in a generator, but it does not hand count back to the caller of next(); it terminates the generator.
Line 2The for loop never sees the 1: it catches StopIteration and ends silently, so "gamma" is never reached.
Line 3Catching StopIteration manually exposes stop.value, which is exactly the returned count of non-blank lines.
Line 4Because the return happened on the second resume, only one value ('X') was ever yielded from gen.
A generator object is single-use
Shows that re-iterating the same generator yields nothing while calling the function again gives a fresh run.
def evens(limit):
for n in range(limit):
if n % 2 == 0:
yield n
nums = evens(6)
print("first pass:", list(nums))
print("second pass:", list(nums))
print("fresh call:", list(evens(6)))Example explained
Line 1The first list() consumes the frame until the for loop over range(6) finishes.
Line 2A finished generator keeps raising StopIteration, so the second list() immediately gets an empty result instead of an error.
Line 3evens(6) builds a brand new generator with a brand new frame, which is the only way to repeat the sequence.
Cleanup with try/finally
Shows that a partially consumed generator still runs its finally block when it is closed.
def numbered(items):
try:
for i, item in enumerate(items, start=1):
yield i, item
finally:
print("cleanup ran")
gen = numbered(["a", "b", "c"])
print(next(gen))
gen.close()
print("after close")Example explained
Line 1The generator is suspended at yield inside the try block after producing (1, 'a').
Line 2gen.close() throws GeneratorExit in at that yield, which unwinds the frame and triggers the finally block.
Line 3Without close(), the same cleanup would only run whenever the generator object is garbage collected, which is far less predictable.
Important notes
One yield is enough to change the whole function, even if it sits in a branch that never executes; the function can then never return a plain value to its caller.
Explicitly raising StopIteration inside a generator body is an error in Python 3.7+: it is converted into a RuntimeError, so use return to finish instead.
Common mistakes
Putting argument checks at the top of a generator function and wrapping the call in try/except: the body has not run yet, so the exception escapes later from the first next() or for loop, far from the call you guarded.
Writing return total at the end of a generator and expecting the caller to receive total: the caller receives the stream of yielded values, and total quietly disappears into StopIteration.value.
Testing a generator with something like `if value in gen:` and then looping over gen: the membership test already consumed part or all of the frame, so the loop sees fewer items or none at all.
Calling len(gen) or gen[0]: a generator implements only __iter__ and __next__, so both raise TypeError instead of giving a size or index.
Try it yourself
Change, predict, then run
Write a generator function chunks(text, size) that yields consecutive slices of text of length size, with a shorter final chunk if needed, then print list(chunks("abcdefgh", 3)) and confirm you get ['abc', 'def', 'gh'].
Open the Python workspaceCheck your understanding
A generator function begins with `if n < 0: raise ValueError("n must be non-negative")`. You write `gen = g(-1)` inside a try/except ValueError, and iterate gen later outside that try. What happens?
- Nothing is raised at the call; the ValueError surfaces from the first next()/for and is not caught by the try block
- The ValueError is raised by g(-1) and caught by the surrounding except
- The generator is created but yields nothing, and no exception is ever raised
- The ValueError is automatically converted into StopIteration, so the loop just ends
Show answer
Calling a generator function only builds a generator object; no line of the body executes, so the guard runs on the first resume and raises there, outside the protected block. Option 2 is tempting because that is exactly how a normal function behaves, but a normal function has no yield and therefore runs its body at call time.