Topic 10 of 64
Generators & Iterators
Overview
Generators produce values lazily (one at a time) using yield, saving massive amounts of memory compared to returning full lists. They are essential for processing large files, infinite sequences, and streaming data pipelines.
Syntax
python
# Generator function
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Usage
fib = fibonacci()
print(next(fib)) # 0
print(next(fib)) # 1
print(next(fib)) # 1
# Generator expression (lazy list comprehension)
squares_gen = (x**2 for x in range(1_000_000)) # no list created!
# yield from (delegating to sub-generator)
def chain(*iterables):
for iterable in iterables:
yield from iterable
# Custom iterator class
class Range:
def __init__(self, stop):
self.current = 0
self.stop = stop
def __iter__(self): return self
def __next__(self):
if self.current >= self.stop: raise StopIteration
val = self.current; self.current += 1; return valCommon Pitfalls
- Generators can only be iterated ONCE — after exhaustion, they return empty. Create a new one to re-iterate.
- next() raises StopIteration at the end — use for loops to handle this automatically.
- Interview tip: Use generators for infinite sequences, streaming data, or any dataset too large to fit in memory.
Real-World Example
Processing a huge CSV file line by line without loading it into memory:
example
python
import csv
from pathlib import Path
def read_large_csv(file_path: str, batch_size: int = 1000):
"""Generator that yields batches of rows — memory efficient."""
with open(file_path, "r", encoding="utf-8") as f:
reader = csv.DictReader(f)
batch = []
for row in reader:
batch.append(row)
if len(batch) >= batch_size:
yield batch
batch = []
if batch: # yield remaining rows
yield batch
# Process 10 million rows without loading into RAM
for batch in read_large_csv("transactions_2025.csv", batch_size=5000):
process_transactions(batch)
save_to_db(batch)
print(f"Processed {len(batch)} transactions")