itertools, Functional Composition & Lazy Pipelines

The standard library itertools module provides C-accelerated iterator primitives for constructing memory-efficient, functional data pipelines. Understanding infinite generators (count, cycle), combinatorics (product, permutations), grouping (groupby), and sequence slicing (islice) is essential for writing high-performance Python code without custom loop boilerplate.

This chapter details itertools C-extension performance, functional iterator composition, the groupby pre-sorting invariant, and memory-safe infinite streams.

1. C-Accelerated Iterator Primitives

All functions inside itertools are implemented in C as native PyTypeObject iterators. They process data streams in $O(1)$ RAM without creating intermediate list objects:

itertools Functional Pipeline:

[ Raw Stream (e.g. infinite count()) ]
                 |
                 v
[ itertools.filterfalse(is_even) ]   <-- C-level evaluation (no Python loop overhead!)
                 |
                 v
[ itertools.islice(0, 100) ]         <-- Zero-copy memory slicing
                 |
                 v
[ Final Output Stream (100 items in <1KB RAM) ]

2. Key itertools Categories & Functions

1. Infinite Iterators:

• count(start=0, step=1): Generates an infinite sequence of incrementing numbers.
• cycle(iterable): Cycles through an iterable infinitely.
• repeat(elem, n=None): Repeats an element n times (or infinitely).

2. Stream Manipulation & Slicing:

• islice(iterable, stop) or islice(iterable, start, stop, step): Performs zero-copy slicing on generators (which do not support standard gen[start:stop] indexing).
• chain(*iterables): Concatenates multiple iterators into a single seamless stream.
• zip_longest(*iterables, fillvalue=None): Zips iterables of unequal length, filling missing values.

3. Combinatorics:

• product(*iterables, repeat=1): Computes the Cartesian product (equivalent to nested for loops).
• permutations(iterable, r): Generates length-r tuples of unique element orderings.
• combinations(iterable, r): Generates length-r tuples of unique element combinations without repetition.

3. The itertools.groupby Pre-sorting Invariant

itertools.groupby(iterable, key=None) groups consecutive matching elements from an iterable based on a key function.

CRITICAL INVARIANT: groupby ONLY groups CONSECUTIVE matching keys! Inputs MUST BE PRE-SORTED by the key function prior to grouping!

import itertools

data = [{"role": "admin", "name": "Alice"},
        {"role": "user",  "name": "Bob"},
        {"role": "admin", "name": "Charlie"}]  # NOT PRE-SORTED BY ROLE!

# ❌ TRAP: Grouping unsorted data creates DUPLICATE 'admin' groups!
for key, group in itertools.groupby(data, key=lambda x: x["role"]):
    print(key, list(group))
# Output: admin -> [Alice], user -> [Bob], admin -> [Charlie] (ADMIN GROUP SPLIT!)

# ✅ PRODUCTION PATTERN: Pre-sort dataset by key function first!
data_sorted = sorted(data, key=lambda x: x["role"])
for key, group in itertools.groupby(data_sorted, key=lambda x: x["role"]):
    print(key, list(group))
# Output: admin -> [Alice, Charlie], user -> [Bob] (CORRECT COMBINED GROUPS!)

4. Production Trade-offs & Memory Mechanics

• islice Consumption: islice consumes elements from the underlying iterator in-place. Once sliced, consumed items cannot be retrieved from the source iterator.
• groupby Inner Iterator Consumption: The inner group iterator returned by groupby shares the underlying data stream. You must consume or convert each group iterator to a list before advancing to the next groupby iteration, or the group data will be lost!