Mastering Python Generators
A generator is a function that produces a sequence of values one at a time instead of building
the whole sequence in memory at once. That single idea makes generators the right tool for reading huge
files line by line, generating infinite sequences, and keeping memory use flat no matter how much data
you process. This guide explains what generators are, how yield and next()
work, when they save you real memory, and the mistakes that trip people up.
What Is a Generator?
A generator is a special kind of iterator created by any function that contains the
yield keyword. Instead of returning one value and ending, a generator pauses at
each yield and resumes from that exact point the next time you ask for a value.
Normal Function vs Generator
Compare these two versions of the same idea — returning the numbers 1, 2, 3:
# Normal function: builds the whole list up front
def get_numbers():
return [1, 2, 3]
# Generator: produces values lazily, one at a time
def gen_numbers():
yield 1
yield 2
yield 3
The visible difference is return vs yield, but the behavior is very different:
calling get_numbers() immediately creates a three-element list, while calling
gen_numbers() creates a generator object that has computed nothing yet.
yield and next(): The Heart of Generators
When you call a generator function, no code runs. You get a generator object you can hand to
next():
>>> g = gen_numbers()
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Each next() runs the function up to the next yield, hands you the value, and
pauses. When there is no yield left, the generator raises StopIteration — this
is the signal that the sequence is finished, and it is exactly how for loops know when to
stop.
Lazy Evaluation
"Lazy" means values are computed only when requested, never earlier. Consider a function that would be expensive to call:
def values():
print("computing...")
yield 10
g = values() # nothing printed yet
next(g) # "computing..." prints now, value 10 returned
For range(10**9), nothing actually stores a billion integers — each number is produced on
demand. That is lazy evaluation, and it is why generators scale.
Memory Usage: The Real Win
Reading a large file is the classic example. The list version loads everything:
# Loads the ENTIRE file into memory as a list of lines
lines = open("app.log").readlines()
The generator version reads one line at a time:
# One line in memory at a time, no matter the file size
with open("app.log") as f:
for line in f: # file objects are iterators too
process(line)
The memory difference is dramatic: a 5 GB log file would consume roughly 5 GB as a list, but a few KB as a stream. If you process data and discard it (parsing logs, summing values, filtering lines), a generator keeps memory flat.
Iterating a Generator
You rarely call next() manually. for and list() do it for you:
for n in gen_numbers():
print(n)
list(gen_numbers()) # [1, 2, 3]
Beware: converting a generator to a list destroys the memory benefit — use it only when you genuinely need all values at once.
Generator Expressions
A generator expression is the lazy cousin of a list comprehension — same syntax, parentheses instead of brackets:
squares = (x * x for x in range(10))
# vs the list version, which builds all 10 now
squares_list = [x * x for x in range(10)]
The generator version computes each square only when consumed. Pass it straight into
sum(), any() or max() for one-pass, low-memory calculations:
sum(x * x for x in range(1_000_000))
Infinite Sequences
Because generators are lazy, they can represent infinite series — something a list can never do:
def count_from(n):
while True:
yield n
n += 1
for i in count_from(1):
if i > 5:
break
print(i)
Real-World Use Cases
- Reading and processing large log files line by line
- Streaming API responses or database cursors row by row
- Pipelines that chain transforms without intermediate lists:
clean = (line.strip() for line in raw) parsed = (parse(line) for line in clean if line) - Generating IDs, timestamps, or test data on demand
Common Mistakes
Mistake 1: Generators can only be iterated once
A generator is exhausted after one pass — it does not rewind. If you need the values twice, create the generator again or store the result.
Mistake 2: Forgetting that the function body runs only when consumed
Debugging "my generator function prints nothing" is usually this: the code inside never ran because
nobody called next() or iterated the object yet.
Mistake 3: Using return value inside a generator
return inside a generator function ends the iteration; the value in return
value is attached to the StopIteration exception and is not yielded. Use
yield for values.
Mistake 4: Wrapping everything in list()
Converting a generator to a list is fine for small sequences but cancels the memory advantage that made you choose a generator in the first place.
When Generators Are Useful — and When They Are Not
Use them when: data is large or streaming, you want one-pass pipelines, or you need an infinite/unbounded sequence.
Skip them when: you need random access by index, you need the data multiple times, or the sequence is tiny and a list is clearer. Generators add a level of indirection; simplicity wins for small cases.
Conclusion
Generators turn "build everything, then process" into "produce one value, process it, move on." The
yield keyword creates the pause/resume behavior, next() drives it, and lazy
evaluation keeps memory flat no matter the input size. Reach for a generator whenever you process data
that could grow beyond what you want in RAM.