Generators & yield
Overview
Generators are a simpler, vastly more powerful way to create Iterators. Instead of building a complex Class with __next__ logic, you write a standard function and use the yield keyword instead of return. When Python hits yield, it pauses the function, returns the value, and freezes the local state. The next time it's called, it resumes exactly where it left off. Generators are 'lazy', computing one value at a time, making them capable of processing infinitely large datasets with near-zero memory footprint.
Syntax
def generate_numbers(limit):
count = 1
while count <= limit:
yield count # PAUSES execution and outputs the value
count += 1 # RESUMES here on the next call
# The function does not run yet! It returns a Generator Object
gen = generate_numbers(3)
print(next(gen)) # 1
print(next(gen)) # 2
# --- Generator Expressions ---
# Like list comprehensions, but with parentheses.
# This does NOT create 1 million items in memory; it evaluates lazily.
massive_gen = (x**2 for x in range(1_000_000))
print(next(massive_gen)) # 0
print(next(massive_gen)) # 1Common Pitfalls
- Attempting to get the
len()of a generator. Because they evaluate lazily on-the-fly, they have no concept of their total length. - Trying to iterate over a generator twice. They are one-time use. Once exhausted, they yield nothing. You must invoke the function again to get a fresh generator.
Interview Questions
return and yield?return completely destroys the function's local state and exits permanently. yield preserves all local variables and simply suspends execution, allowing the function to be resumed later.
Real-World Example
Processing a massive streaming API payload or Log File without crashing the server's RAM.
def stream_large_file(file_path):
with open(file_path, "r") as file:
for line in file:
yield line.strip()
# We can process a 100GB file effortlessly because we only ever hold
# ONE line in memory at any given nanosecond.
for line in stream_large_file("massive_log.txt"):
if "FATAL" in line:
print("Alert triggered!")Check Your Knowledge
Test your understanding of Generators & yield with these quick questions.