Lesson 2 of 10
A function that builds a list has to finish it before returning. A generator produces values one at a time, only as they're asked for.
Swap return for yield:
def count_to(n):
for i in range(1, n + 1):
yield i
for number in count_to(5):
print(number)
Calling count_to(5) runs no code at all. It returns a generator object; the body only runs
as you iterate.
def count_to(n):
for i in range(1, n + 1):
yield i
gen = count_to(3)
print(type(gen).__name__)
print(next(gen))
print(next(gen))
print(next(gen))
Each next() runs until the next yield, hands back that value, and freezes — local
variables and position intact. The following next() resumes right there. A fourth next()
here would raise StopIteration, which is exactly what a for loop catches to know it's done.
Why bother? Memory
This is the point. Compare:
def squares_list(n):
return [i ** 2 for i in range(n)]
def squares_gen(n):
for i in range(n):
yield i ** 2
import sys
print(sys.getsizeof(squares_list(10000)) > 10000)
print(sys.getsizeof(squares_gen(10000)) < 500)
The list holds 10,000 numbers in memory. The generator holds a position. That difference is what lets you process a 10GB file on a laptop — the pattern from lesson 6.1, where looping over a file never loads it all:
def read_lines(path):
"""Yield each line, one at a time."""
with open(path) as f:
for line in f:
yield line.strip()
with open("data.txt", "w") as f:
f.write("alpha\nbeta\ngamma\n")
for line in read_lines("data.txt"):
print(line)
Generator expressions
Like a list comprehension with parentheses:
squares = (n ** 2 for n in range(5))
print(type(squares).__name__)
print(list(squares))
When a generator expression is the only argument, you can drop the extra parentheses — this is the idiomatic way to sum a large sequence without building it:
print(sum(n ** 2 for n in range(1, 6)))
print(max(len(w) for w in ["a", "bbb", "cc"]))
print(any(n > 100 for n in [1, 2, 500, 3]))
any() and all() short-circuit — they stop at the first decisive value, so pairing them
with a generator means you may not evaluate the rest at all.
One-shot only
A generator is exhausted after one pass:
gen = (n for n in range(3))
print(list(gen))
print(list(gen))
The second list() is empty. If you need the values twice, store them in a list.
yield in a pipeline
Generators chain neatly, each stage passing values along without building intermediates:
def numbers():
for i in range(1, 11):
yield i
def evens(source):
for n in source:
if n % 2 == 0:
yield n
def squared(source):
for n in source:
yield n ** 2
print(list(squared(evens(numbers()))))
Nothing is computed until list() pulls on the chain.
Try it yourself
Checkpoint
yieldmakes a generator: values are produced lazily, one at a time.- Calling a generator function runs nothing; iteration drives it.
- Each
yieldfreezes state and resumes on the nextnext(). - Generators use constant memory regardless of how many values they produce.
(x for x in ...)is a generator expression; generators are single-use.
Questions about this lesson
Log in to ask a question.