Topic 22 of 64
Itertools & functools
Overview
The itertools module provides memory-efficient iterators for common patterns. functools provides higher-order function tools like lru_cache (memoization), partial (partial application), and reduce. Both are performance essentials.
Syntax
python
import itertools
import functools
# itertools — lazy, memory-efficient
itertools.chain([1,2], [3,4]) # [1,2,3,4]
itertools.product("AB", repeat=2) # AA, AB, BA, BB
itertools.combinations("ABC", 2) # AB, AC, BC
itertools.permutations("AB", 2) # AB, BA
itertools.groupby(sorted_data, key=lambda x: x["dept"])
itertools.islice(counter, 0, 10) # first 10 from iterator
itertools.cycle([1,2,3]) # 1,2,3,1,2,3,...(infinite)
itertools.takewhile(lambda x: x < 5, [1,2,3,7,8]) # [1,2,3]
# functools — higher-order functions
@functools.lru_cache(maxsize=128)
def expensive_fibonacci(n: int) -> int:
if n < 2: return n
return expensive_fibonacci(n-1) + expensive_fibonacci(n-2)
# partial — fix some arguments
from functools import partial
power_of_2 = partial(pow, 2) # fix base=2
power_of_2(10) # 1024
# reduce
from functools import reduce
product = reduce(lambda acc, x: acc * x, [1,2,3,4,5]) # 120Common Pitfalls
- itertools functions return iterators, not lists — wrap with list() to materialize, or use in for loops.
- lru_cache requires hashable arguments — lists and dicts can't be used as arguments. Convert to tuples first.
- Interview tip: functools.reduce() is the equivalent of JavaScript's Array.reduce() — works left-to-right by default.
Real-World Example
Analytics processing pipeline using itertools and functools:
example
python
import itertools
import functools
from collections import defaultdict
def analyze_sales_combinations(products: list, orders: list[dict]) -> dict:
# Find most common product pairs bought together
pair_counts = defaultdict(int)
for order in orders:
items = sorted(order["product_ids"])
for pair in itertools.combinations(items, 2):
pair_counts[pair] += 1
top_pairs = sorted(
pair_counts.items(),
key=lambda x: x[1],
reverse=True
)[:10]
return {"top_pairs": top_pairs}
# Memoized route calculator (avoids recomputing distances)
@functools.lru_cache(maxsize=500)
def calculate_distance(city_a: str, city_b: str) -> float:
"""Expensive database/API call — cached after first call."""
return expensive_geo_lookup(city_a, city_b)
# Generate paginated API calls lazily
def paginated_api_fetcher(endpoint: str, page_size: int = 100):
for page in itertools.count(1):
batch = api.get(endpoint, page=page, size=page_size)
if not batch: break
yield from batch