Mastering Python Generators

A generator is a function that produces a sequence of values one at a time instead of building the whole sequence in memory at once. That single idea makes generators the right tool for reading huge files line by line, generating infinite sequences, and keeping memory use flat no matter how much data you process. This guide explains what generators are, how yield and next() work, when they save you real memory, and the mistakes that trip people up.

What Is a Generator?

A generator is a special kind of iterator created by any function that contains the yield keyword. Instead of returning one value and ending, a generator pauses at each yield and resumes from that exact point the next time you ask for a value.

Normal Function vs Generator

Compare these two versions of the same idea — returning the numbers 1, 2, 3:

# Normal function: builds the whole list up front
def get_numbers():
    return [1, 2, 3]

# Generator: produces values lazily, one at a time
def gen_numbers():
    yield 1
    yield 2
    yield 3

The visible difference is return vs yield, but the behavior is very different: calling get_numbers() immediately creates a three-element list, while calling gen_numbers() creates a generator object that has computed nothing yet.

yield and next(): The Heart of Generators

When you call a generator function, no code runs. You get a generator object you can hand to next():

>>> g = gen_numbers()
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Each next() runs the function up to the next yield, hands you the value, and pauses. When there is no yield left, the generator raises StopIteration — this is the signal that the sequence is finished, and it is exactly how for loops know when to stop.

Lazy Evaluation

"Lazy" means values are computed only when requested, never earlier. Consider a function that would be expensive to call:

def values():
    print("computing...")
    yield 10

g = values()        # nothing printed yet
next(g)             # "computing..." prints now, value 10 returned

For range(10**9), nothing actually stores a billion integers — each number is produced on demand. That is lazy evaluation, and it is why generators scale.

Memory Usage: The Real Win

Reading a large file is the classic example. The list version loads everything:

# Loads the ENTIRE file into memory as a list of lines
lines = open("app.log").readlines()

The generator version reads one line at a time:

# One line in memory at a time, no matter the file size
with open("app.log") as f:
    for line in f:        # file objects are iterators too
        process(line)

The memory difference is dramatic: a 5 GB log file would consume roughly 5 GB as a list, but a few KB as a stream. If you process data and discard it (parsing logs, summing values, filtering lines), a generator keeps memory flat.

Iterating a Generator

You rarely call next() manually. for and list() do it for you:

for n in gen_numbers():
    print(n)

list(gen_numbers())   # [1, 2, 3]

Beware: converting a generator to a list destroys the memory benefit — use it only when you genuinely need all values at once.

Generator Expressions

A generator expression is the lazy cousin of a list comprehension — same syntax, parentheses instead of brackets:

squares = (x * x for x in range(10))

# vs the list version, which builds all 10 now
squares_list = [x * x for x in range(10)]

The generator version computes each square only when consumed. Pass it straight into sum(), any() or max() for one-pass, low-memory calculations:

sum(x * x for x in range(1_000_000))

Infinite Sequences

Because generators are lazy, they can represent infinite series — something a list can never do:

def count_from(n):
    while True:
        yield n
        n += 1

for i in count_from(1):
    if i > 5:
        break
    print(i)

Real-World Use Cases

Common Mistakes

Mistake 1: Generators can only be iterated once

A generator is exhausted after one pass — it does not rewind. If you need the values twice, create the generator again or store the result.

Mistake 2: Forgetting that the function body runs only when consumed

Debugging "my generator function prints nothing" is usually this: the code inside never ran because nobody called next() or iterated the object yet.

Mistake 3: Using return value inside a generator

return inside a generator function ends the iteration; the value in return value is attached to the StopIteration exception and is not yielded. Use yield for values.

Mistake 4: Wrapping everything in list()

Converting a generator to a list is fine for small sequences but cancels the memory advantage that made you choose a generator in the first place.

When Generators Are Useful — and When They Are Not

Use them when: data is large or streaming, you want one-pass pipelines, or you need an infinite/unbounded sequence.

Skip them when: you need random access by index, you need the data multiple times, or the sequence is tiny and a list is clearer. Generators add a level of indirection; simplicity wins for small cases.

Conclusion

Generators turn "build everything, then process" into "produce one value, process it, move on." The yield keyword creates the pause/resume behavior, next() drives it, and lazy evaluation keeps memory flat no matter the input size. Reach for a generator whenever you process data that could grow beyond what you want in RAM.

Related Articles

A.M. Rinas

Written by A.M. Rinas

Full-Stack Software Engineer working on web applications, DevOps automation, and databases. More about me on the about page, or say hi through the contact page and LinkedIn.

← Back to all articles