Generators & Iterators

Reviewed & published by Brayan K

Generators and iterators are two of Python's most powerful features — allowing you to process massive datasets, stream data, write memory-efficient code, build pipelines, and design systems that behave like professional-grade libraries.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• • What iterators are and how the iterator protocol works
• • How to create generators with yield
• • The difference between return and yield
• • Generator expressions and memory-efficient data pipelines
• • Real-world use cases: streaming large files, infinite sequences, lazy evaluation

What You'll Learn

This lesson will take you from advanced fundamentals → deep internal mechanics → real-world patterns used in production.

🔥 1. What Exactly Are Iterators?

An iterator is any object that can give you items one at a time. It follows a simple contract:

# Manual Iterator Example
class CountUpTo:
    def __init__(self, max_value):
        self.max = max_value
        self.current = 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.max:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

# Usage
for x in CountUpTo(5):
    print(x)

# ✅ Expected output:
# 1
# 2
# 3
# 4
# 5

# Generators Practice

# Simple generator function
def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1

print("Counting to 5:")
for num in count_up_to(5):
    print(num)

# Generator expression
squares = (x**2 for x in range(1, 6))
print("\nSquares:")
print(list(squares))

# Fibonacci generator
def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

print("\nFirst 10 Fibonacci numbers:")
print(list(fibonacci(10)))

# Memory efficient - generators don't store all values
def infinite_counter():
    n = 0
    while True:
        yield n
        n += 1

counter = infinite_counter()
print("\nFirst 5 from infinite counter:")
for _ in range(5):
    print(next(counter))

⚙️ 2. Why Iterators Matter

Iterators solve the "Big Data" problem:

• Streaming: Process data as it arrives (like Netflix streaming)
• Pipelines: Chain operations together (like assembly lines)
• Lazy evaluation: Only compute when needed (saves CPU & RAM)
• Infinite sequences: Generate data forever without memory limits

⚡ 3. Enter Generators — The Shortcut to Iterators

A generator is Python's elegant shortcut for creating iterators using the yield keyword.

Instead of writing an entire class, just use yield:

def count_up_to(max_value):
    current = 1
    while current <= max_value:
        yield current
        current += 1

# Usage
for num in count_up_to(5):
    print(num)

# ✅ Expected output:
# 1
# 2
# 3
# 4
# 5

This produces the same behavior as the iterator class — but with 90% less code.

🧠 4. How yield Works Internally

When Python sees yield, something magical happens:

Step 1: Function becomes a generator object (not executed yet!)

Step 2: First next() runs until first yield

Step 3: Execution pauses, value is returned

Step 4: Next next() resumes from exact pause point

Step 5: Repeat until function ends → StopIteration

🧪 4b. Worked Example — Watch a Generator Freeze and Resume

The single idea behind generators is suspension: yield hands a value out and freezes the function exactly where it stands, keeping every local variable alive until someone asks for the next value. The prints inside the generator below are there so you can see precisely when its body runs.

Read the comments, predict the order of the output, then run it and see if you were right.

# A generator function looks ordinary, but because its body contains "yield",
# calling it does NOT run the body — it hands back a generator object instead.

def countdown(start):
    print("  (generator body starts now)")
    current = start
    while current > 0:
        yield current          # hand a value out, then FREEZE on this line
        current -= 1
    print("  (generator body finished)")


gen = countdown(3)
print(type(gen))          # a generator object — and still no output from the body

# next() thaws the body, runs it up to the next yield, then freezes it again.
print(next(gen))          # the "(body starts now)" line appears now, then 3
print(next(gen))          # resumes after the yield: current becomes 2

# A for loop is just next() called for you until the generator is exhausted.
for value in gen:
    print("loop got", value)   # picks up exactly where next() left off

# Once exhausted, a generator stays exhausted. It is single-use, unlike a list.
print(list(gen))          # [] — there is nothing left to hand out

# The same laziness in one line: a generator EXPRESSION (round brackets).
squares = (n * n for n in range(1, 6))
print(type(squares))      # nothing has been computed yet
print(sum(squares))       # 55, computed one square at a time
print(sum(squares))       # 0 — the second sum finds an empty generator

# ✅ Expected output:
# <class 'generator'>
#   (generator body starts now)
# 3
# 2
# loop got 1
#   (generator body finished)
# []
# <class 'generator'>
# 55
# 0

Two results catch nearly everyone out. (generator body starts now) prints after print(type(gen)), because nothing runs until the first next(). And the second sum(squares) is 0, not 55 — a generator you have already walked through is empty. If you need the values twice, keep a list.

🎯 4c. Your Turn — Make It Lazy

Three blanks, all of them generator syntax: the keyword that yields, the built-in that pulls one value, and the body of a generator expression. Everything else is written for you.

# 🎯 YOUR TURN — build a lazy stream of even numbers
# Fill in the three blanks marked ___

def even_numbers(limit):
    for n in range(limit):
        if n % 2 == 0:
            # 👉 blank 1: the keyword that hands one value out and freezes here
            #    (use return instead and you get a function that stops at 0)
            ___ n


gen = even_numbers(10)
print(type(gen))        # proves you built a generator, not a list

# 👉 blank 2: the built-in that pulls exactly one value out of a generator
print(___(gen))         # the first even number

print(list(gen))        # everything still waiting behind it

# 👉 blank 3: the value to produce for each n — the cube of n
total = sum(___ for n in range(1, 5))     # 1 + 8 + 27 + 64
print(total)

# ✅ Expected output:
# <class 'generator'>
# 0
# [2, 4, 6, 8]
# 100

If list(gen) gives you [0, 2, 4, 6, 8] including the zero, blank 2 has not consumed anything yet — check you used a call, next(gen), not just the name.

🧵 5. Real-World Example — Log File Streaming

Imagine parsing a 5GB log file. Without generators, you'd crash. With generators:

def read_logs(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

# Simulated usage (without actual file)
def simulate_logs():
    logs = ["INFO: Started", "ERROR: Failed", "INFO: Done"]
    for log in logs:
        yield log

for log in simulate_logs():
    print(log)

# ✅ Expected output:
# INFO: Started
# ERROR: Failed
# INFO: Done

• ✔ Doesn't load the entire file
• ✔ Works line by line
• ✔ Streams infinitely large files

Python itself uses this pattern in its own IO libraries.

💧 6. Infinite Generators (Perfect for Simulations & AI)

def infinite_counter(start=0):
    while True:
        yield start
        start += 1

# Take first 10 values
counter = infinite_counter()
for _ in range(10):
    print(next(counter))

• Reinforcement learning training loops
• Randomized datasets
• Unique ID generation

📦 7. Generator Pipelines (Functional Programming Style)

Chain generators together like Unix shell pipes (command1 | command2 | command3):

def read_lines(text):
    for line in text.split("\n"):
        yield line.strip()

def filter_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

def extract_messages(lines):
    for l in lines:
        yield l.split(":")[-1].strip()

# Sample log data
logs = """INFO: System started
ERROR: Connection timeout
INFO: User logged in
ERROR: File not found"""

pipeline = extract_messages(filter_errors(read_lines(logs)))
for msg in pipeline:
    print(msg)

# ✅ Expected output:
# Connection timeout
# File not found

Each step processes data lazily. Nothing loads into memory at once.

🧩 8. yield from — Delegating to Sub-Generators

yield from lets you delegate iteration to another generator:

def flatten(list_of_lists):
    for sub in list_of_lists:
        yield from sub

nested = [[1, 2], [3, 4], [5, 6]]
print(list(flatten(nested)))

# ✅ Expected output:
# [1, 2, 3, 4, 5, 6]

This is cleaner and faster than nested loops.

🌀 9. Two-Way Generators (send() Method)

You can send values INTO generators, making them interactive:

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is not None:
            total += value

g = accumulator()
next(g)        # start generator  
print(g.send(10))     # -> 10  
print(g.send(5))      # -> 15
print(g.send(3))      # -> 18

This technique is used in:

• Async frameworks
• Stream processors
• Complex event simulation

⚙️ 10. Generator-Based Coroutines (Pre-asyncio Style)

Generators can act as coroutines — functions that can pause and receive data:

def coroutine():
    while True:
        message = yield
        print("Received:", message)

c = coroutine()
next(c)  # Start coroutine
c.send("Hello")
c.send("World")

Now superseded by async def, but still heavily used in internal libraries and advanced scheduling systems.

📚 11. Generators as Context Managers

from contextlib import contextmanager

@contextmanager
def managed_resource(name):
    print(f"Opening {name}")
    try:
        yield name
    finally:
        print(f"Closing {name}")

with managed_resource("database") as db:
    print(f"Using {db}")

# ✅ Expected output:
# Opening database
# Using database
# Closing database

This pattern merges generators + cleanup logic.

🚀 12. Generator Expressions (Faster, Cleaner, Memory Efficient)

# Generator expression - uses no extra memory
squares = (x*x for x in range(10))
print("Generator:", squares)
print("Values:", list(squares))

# Compare with list comprehension
squares_list = [x*x for x in range(10)]
print("\nList:", squares_list)

This uses 0 RAM regardless of range size.

🔬 13. Building Your Own Iterable Class (Advanced)

Here's a complete example with detailed comments:

class Fibonacci:
    def __init__(self, limit):
        self.limit = limit
        self.a, self.b = 0, 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.a > self.limit:
            raise StopIteration
        value = self.a
        self.a, self.b = self.b, self.a + self.b
        return value

for num in Fibonacci(100):
    print(num)

# ✅ Expected output:
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# 21
# 34
# 55
# 89

🕹 14. How Python's For-Loops Use Iterators Internally

data = [1, 2, 3]

# This for loop:
for x in data:
    print(x)

# Actually does this internally:
print("\nManual iteration:")
it = data.__iter__()
while True:
    try:
        x = it.__next__()
        print(x)
    except StopIteration:
        break

Understanding this gives you total control over how objects behave.

📊 15. Real-World Use Cases (Professional Level)

Streaming large CSVs with chunksize parameter

Uses iterators to stream mini-batches.

Iterators control crawling pipelines.

Rows are lazily streamed.

Generators still power internal scheduling logic.

🧪 16. Mini Project — Build a Streaming Data Pipeline

• A data reader
• A filter stage
• A transformer

Each one is a generator.

This simulates how Airflow, Spark, and Pandas internally process data.

Here is the brief. Only the outline is given — no logic, no filled-in bodies. Each stage is a generator that takes the previous stage as its argument, so a single row travels the whole pipeline before the next row is even read.

# 🎯 MINI-CHALLENGE: a five-stage streaming pipeline
#
# Your data (copy this line exactly):
#   RAW = ["ada,40", "grace,65", "bad line", "alan,52", "linus,17"]
#
# Write five generators. Each one loops over what it is given and yields:
# 1. read(rows)       -> yield every row unchanged (the source stage)
# 2. parse(rows)      -> row.split(",") ; skip any row that does not split into
#                        exactly 2 parts ; yield {"name": name, "score": int(score)}
# 3. adults(records)  -> yield only the records whose score is 40 or more
# 4. shout(records)   -> yield the record again with the name uppercased
# 5. export(records)  -> for each record, print   NAME: score
#
# Then wire the stages together and run the whole thing in one line:
#   export(shout(adults(parse(read(RAW)))))
#
# ✅ Expected output:
# ADA: 40
# GRACE: 65
# ALAN: 52

# your code here

If nothing prints at all, the last stage is probably yielding instead of printing. export is the one stage that consumes rather than produces — without something pulling on the chain, a pipeline of generators does no work whatsoever.

🎉 Conclusion

You now fully understand:

✔ How iterators work internally

✔ How to build custom iterable objects

✔ How to use generators for memory-efficient processing

✔ How Python implements lazy evaluation

✔ How to build generator pipelines

✔ How advanced frameworks use iterators under the hood

📋 Quick Reference — Generators

🏆 Lesson Complete!

You understand lazy evaluation and how to build memory-efficient pipelines with generators — the same technique used inside Pandas, Airflow, and Spark.

Practice quiz

Continue this course

• Previous: Context Managers & the with Statement in Depth
• Next: Advanced Async & Await Patterns — Write clean asynchronous code with async/await and coroutines
• Quick reference: Python cheat sheet