Iterables, Iterators & Generator Functions

Iteration is Python’s fundamental mechanism for processing sequences and data streams lazily. Powered by the Iteration Protocol (__iter__ and __next__) and Generator Functions (yield), Python processes datasets in $O(1)$ memory without materializing full collections in RAM.

This chapter details the Iteration Protocol, PyGenObject frame suspension, YIELD_VALUE bytecode mechanics, and generator execution state machines.

1. The Iteration Protocol (__iter__ & __next__)

Python separates data containers (Iterables) from iteration cursors (Iterators):

• Iterable: An object implementing __iter__() that returns a new Iterator instance (e.g. list, dict, set).
• Iterator: An object implementing __next__() (which returns the next item or raises StopIteration) AND __iter__() (which returns self).

The CPython for-Loop Protocol Execution:

[ for item in container: ]
          |
          v
[ Call: iterator = iter(container) (container.__iter__()) ]
          |
          v
[ Loop Head: Call item = next(iterator) (iterator.__next__()) ]
          |
          +---> Next item returned?
          |         ├── YES: Execute loop body, then repeat Loop Head!
          |         └── NO (Raises StopIteration): Catch exception & exit loop cleanly!

2. Generator Functions & PyGenObject Struct

A function containing the yield keyword is compiled by CPython into a generator function. When called, a generator function does not execute its code immediately. Instead, it returns a heap-allocated Generator Object (PyGenObject):

PyGenObject Execution State Machine:

[ GEN_CREATED ] ---> (Call next()) ---> [ GEN_RUNNING ]
                                             |
                                       (Hits yield)
                                             v
[ GEN_CLOSED ]  <--- (Raises StopIteration) <-- [ GEN_SUSPENDED ]

• GEN_CREATED: Generator object instantiated, execution frame allocated, instruction pointer at 0.
• GEN_RUNNING: Active bytecode execution on the frame stack.
• GEN_SUSPENDED: Executed YIELD_VALUE opcode. Frame state (local variables, value stack pointer) is saved on the heap, and control yields back to the caller.
• GEN_CLOSED: Generator exhausted or closed; execution frame deallocated.

3. Bytecode Mechanics (YIELD_VALUE)

When CPython encounters yield x:

1. Evaluates x onto the frame stack.
2. Emits the YIELD_VALUE opcode.
3. Suspends the execution frame (f_state = FRAME_SUSPENDED).
4. Returns x to the caller of next().
5. When next() is called again, CPython resumes the frame at the instruction pointer directly after YIELD_VALUE!

4. Production Trade-offs: Generators vs. Lists

• Memory Efficiency: Streaming 1,000,000 records via a generator consumes ~128 bytes of RAM; materializing a 1,000,000-item list consumes ~8MB of RAM.
• Single-Pass Limitation: Generators are single-pass cursors. Once exhausted, a generator cannot be reused or reset; calling next() repeatedly raises StopIteration. Re-run the generator function to create a fresh iterator.