Python Generators and Iterators: Yield Like a Pro

Generators are Python’s most elegant solution to working with sequences of data without loading everything into memory at once. They produce values lazily — one at a time, on demand — making them ideal for large datasets, infinite sequences, and pipelining data processing.

Iterators vs Iterables

An iterable is anything you can loop over: lists, tuples, strings, dictionaries, sets, and files. An iterator is an object that produces values one at a time using __next__().

my_list = [1, 2, 3]
iterator = iter(my_list)

print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
print(next(iterator))  # StopIteration

Every iterator is iterable, but not every iterable is an iterator. A list is iterable but not an iterator — calling iter() on it returns a new iterator.

Creating Generators

A generator is a function that uses yield instead of return:

def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1

counter = count_up_to(5)
print(next(counter))  # 1
print(next(counter))  # 2

for num in count_up_to(3):
    print(num)  # 1, 2, 3

When Python encounters yield, it saves the function’s state — all local variables, the instruction pointer, and the call stack. On the next call to next(), execution resumes from the line after yield. This state suspension is what makes generators memory-efficient.

How Yield Works Internally

def simple_generator():
    print("First yield")
    yield 1
    print("Between yields")
    yield 2
    print("Last yield")
    yield 3

gen = simple_generator()
print(next(gen))  # First yield / 1
print(next(gen))  # Between yields / 2
print(next(gen))  # Last yield / 3

Each next() call resumes execution until the next yield.

Generator Expressions

Generator expressions are like list comprehensions but with parentheses:

squares = (x ** 2 for x in range(10))

print(next(squares))  # 0
print(next(squares))  # 1
print(next(squares))  # 4

They are memory-efficient — the list comprehension [x ** 2 for x in range(10_000_000)] would use hundreds of megabytes. The generator expression uses almost nothing.

When to Use Generator vs List Comprehension

# Bad: loading entire file into memory
lines = [line.strip() for line in open("huge_file.txt")]

# Good: processing one line at a time
lines = (line.strip() for line in open("huge_file.txt"))
for line in lines:
    process(line)

The send() Method

Generators can receive values through send():

def echo():
    while True:
        received = yield
        print(f"Received: {received}")

gen = echo()
next(gen)           # advance to first yield
gen.send("Hello")   # Received: Hello
gen.send("World")   # Received: World

The first call must be next(gen) (or gen.send(None)) to advance to the first yield. After that, each send() sends a value to the generator and resumes execution.

Practical send() Example

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value
    return total

acc = accumulator()
next(acc)              # prime the generator
print(acc.send(10))    # 10
print(acc.send(20))    # 30
print(acc.send(30))    # 60

try:
    acc.send(None)     # stop the generator
except StopIteration as e:
    print(f"Final total: {e.value}")  # Final total: 60

yield from — Delegating to Subgenerators

yield from delegates iteration to another generator:

def sub_generator():
    yield 1
    yield 2
    yield 3

def main_generator():
    yield "start"
    yield from sub_generator()
    yield "end"

list(main_generator())  # ['start', 1, 2, 3, 'end']

Chaining Generators

def read_lines(filename):
    with open(filename) as f:
        for line in f:
            yield line.strip()

def filter_comments(lines):
    for line in lines:
        if not line.startswith("#"):
            yield line

def split_words(lines):
    for line in lines:
        yield from line.split()

# Pipeline
pipeline = split_words(filter_comments(read_lines("config.txt")))
for word in pipeline:
    print(word)

Each generator in the pipeline processes one item at a time. The entire pipeline uses constant memory regardless of file size.

Generator Performance Characteristics

Generators trade index-based access for memory efficiency. A generator yielding one million items uses a fixed amount of memory — just the generator object’s state. The equivalent list would consume approximately 8MB for integers alone (8 bytes each × overhead). For string data, the savings are even larger since strings are reference types with additional object overhead.

Async Generators

Python 3.6+ supports asynchronous generators using async for and yield inside async def functions. These are useful for streaming data from async sources like databases, HTTP responses, or file I/O: async def read_stream(): async for chunk in db.fetch_large_query(): yield chunk. Each value is produced asynchronously, allowing the event loop to handle other tasks while waiting for I/O.

Infinite Sequences

Generators can represent infinite sequences:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
print([next(fib) for _ in range(10)])  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Common Infinite Generators

def counter(start=0, step=1):
    while True:
        yield start
        start += step

def repeat(value):
    while True:
        yield value

def cycle(iterable):
    while True:
        for item in iterable:
            yield item

Comparing Generator Memory Usage

Processing Large Files

def process_log(file_path):
    with open(file_path) as f:
        for line in f:
            yield parse_log_line(line)

errors = (entry for entry in process_log("server.log")
          if entry["level"] == "ERROR")
for error in errors:
    send_alert(error)

Paginating API Calls

def fetch_all_pages(api_url):
    page = 1
    while True:
        response = requests.get(f"{api_url}?page={page}")
        data = response.json()
        if not data["results"]:
            break
        yield from data["results"]
        page += 1

for item in fetch_all_pages("https://api.example.com/items"):
    process(item)

Generator-Based Pipelines

def parse_logs(file_path):
    with open(file_path) as f:
        for line in f:
            yield parse_line(line)

def filter_errors(entries):
    for entry in entries:
        if entry.level == "ERROR":
            yield entry

def format_alerts(errors):
    for error in errors:
        yield f"[{error.timestamp}] {error.message}"

pipeline = format_alerts(filter_errors(parse_logs("app.log")))
for alert in pipeline:
    print(alert)

Generator Cleanup with try/finally

Generators can define cleanup logic using try/finally blocks. The finally block executes when the generator is garbage collected, explicitly closed with .close(), or raises an exception. This pattern is essential for managing resources in generator functions: open files, database connections, and network sockets should be released in a finally block. The contextlib.contextmanager decorator wraps this pattern into a reusable context manager, combining generator flexibility with resource safety.

Related: Learn about Python decorators and list comprehensions.

Using itertools with Generators

Python’s itertools module provides combinatoric generators that compose perfectly with custom generators and produce efficient lazy evaluation chains for stream processing. itertools.chain concatenates multiple generators into one sequence. itertools.islice takes slices from infinite generators without materializing them. itertools.takewhile and itertools.dropwhile filter based on predicates. itertools.product produces Cartesian products lazily. itertools.groupby groups consecutive elements by a key function. Combining itertools functions with generator pipelines creates expressive, memory-efficient data processing chains that handle arbitrarily large datasets.

Generators vs Async Generators

Synchronous generators work with regular for loops and produce values immediately when next() is called. Async generators (Python 3.6+) use async for and await within async def functions, producing values asynchronously. Use async generators when each element requires an I/O operation — fetching from a database, reading from a network stream, or processing files. The decision between sync and async generators is driven by whether the work is CPU-bound (use sync) or I/O-bound (use async).

FAQ

When should I use a generator vs a list?

Use a generator when processing large datasets that don’t fit in memory, when you only need to iterate once, or when building processing pipelines. Use a list when you need random access, need to iterate multiple times, or the dataset is small enough to fit in memory comfortably.

How do I convert a generator to a list?

Call list(generator) to materialize all values. Be careful — if the generator is infinite or produces millions of items, this will exhaust memory. Use list(gen) only when you know the output size is bounded and manageable.

What is the difference between return and yield in a generator?

A return in a generator terminates it and can optionally provide a value accessible through StopIteration.value. yield suspends execution, produces a value, and can be resumed. A generator with only return and no yield is not a generator — it is a regular function.

Generator Memory Profiling

Profile generator memory usage with memory_profiler or tracemalloc. A generator yielding one million integers uses only ~200 bytes regardless of item count — just the generator frame object. The equivalent list uses ~36 MB (8 bytes per int × overhead). For string data, savings are even larger. Use sys.getsizeof() on lists vs generators to quantify memory savings. Always measure before optimizing — for datasets under 10,000 items, list overhead is usually negligible compared to development clarity.

Can generators be pickled?

No — generators are not serializable because they contain live execution state (stack frames, instruction pointers). If you need to persist or distribute iteration state, use iterables backed by data structures (lists, queues) or use itertools.tee for independent iterator copies.