Generators & Iterators
Published by Brayan K
Generators and iterators are two of Python's most powerful features — allowing you to process massive datasets, stream data, write memory-efficient code, build pipelines, and design systems that behave like professional-grade libraries.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
A generator is an iterator you write yourself
You've used iterators already: map() and filter() in Lambda Functions, generator expressions in Comprehensions, zip() and enumerate() in enumerate & zip, everything in itertools, and a file object in File Handling. Each one hands over one value at a time, when a loop or list() asks for it, and is used up after one pass. This lesson is about the ones you write yourself, with yield. Writing one as a class is in Iterators: __iter__ & __next__, and files too big for memory are in Working with Files, Streams & Large Datasets.
A function with yield in its body is a generator function. Calling it doesn't run the body: it hands back a generator. next(gen) asks a generator for its next value. Each next() runs the body up to the next yield, hands out that value and pauses there, keeping its variables, until the next next():
def countdown(start):
print("starting")
n = start
while n > 0:
yield n
n -= 1
print("finished")
gen = countdown(3)
print("made the generator")
print(next(gen))
print(next(gen))
print(list(gen))
# ✅ Expected output:
# made the generator
# starting
# 3
# 2
# finished
# [1]made the generator prints first, because the call ran none of the body. The first next() starts it, so starting prints, and it pauses at the first yield with 3. The second resumes after that yield and pauses at the next one. list() asks until the body ends: finished prints before [1] appears, and only 1 was left.
The function stays a function. What a call gives back is the generator:
def countdown(start):
n = start
while n > 0:
yield n
n -= 1
gen = countdown(3)
print(type(countdown).__name__)
print(type(gen).__name__)
print(list(gen))
# ✅ Expected output:
# function
# generator
# [3, 2, 1]Printing the generator itself doesn't show its values: you get something like <generator object countdown at 0x…>, with a memory address. Use list() or a loop.
One yield anywhere in the body is enough, even one that never runs:
def maybe_values(flag):
print("running")
if flag:
yield 1
result = maybe_values(False)
print(type(result).__name__)
print(list(result))
# ✅ Expected output:
# generator
# running
# []The call printed nothing. running only appears when list() starts the body, and since the if is false, there's nothing to collect.
next() past the end raises StopIteration, and a for loop stops quietly
When the body has ended, the generator has nothing left, and next() raises StopIteration:
def countdown(start):
n = start
while n > 0:
yield n
n -= 1
gen = countdown(2)
print(next(gen))
print(next(gen))
try:
next(gen)
except StopIteration:
print("StopIteration: nothing left")
# ✅ Expected output:
# 2
# 1
# StopIteration: nothing leftUncaught, it stops the program like any other error, and the last line of the error is StopIteration. To get a value instead of the error, give next() a default, which it returns once the generator is used up:
def countdown(start):
n = start
while n > 0:
yield n
n -= 1
gen = countdown(1)
print(next(gen, "done"))
print(next(gen, "done"))
print(next(gen, "done"))
# ✅ Expected output:
# 1
# done
# doneWith a generator expression from Comprehensions, that finds the first match, or None when nothing matches. Only the words up to the first match are checked:
words = ["fig", "kiwi", "banana", "cherry"]
print(next((w for w in words if len(w) > 5), None))
print(next((w for w in words if len(w) > 9), None))
# ✅ Expected output:
# banana
# NoneA for loop and list() call next() for you, and take StopIteration as the end. That's why the loops and list() calls in this lesson stop without an error.
next() only works on iterators, and a list isn't one:
names = ["Ada", "Ben"]
print(next(names))
# ✅ Expected error:
# TypeError: 'list' object is not an iteratoriter() makes an iterator over the list that remembers its place, as in itertools. The list itself doesn't change:
names = ["Ada", "Ben"]
it = iter(names)
print(next(it))
print(next(it))
print(names)
# ✅ Expected output:
# Ada
# Ben
# ['Ada', 'Ben']A generator is an iterator already, so next() works on it directly. A for loop calls iter() first, which is why it works on a list too. Iterators: __iter__ & __next__ shows how that works, and how to write an iterator class.
A StopIteration raised inside a generator's body doesn't end it quietly. Python turns it into a RuntimeError, so a bug can't pass for the normal end. Here next(it) finds nothing left to pair with 3:
def pairs(items):
it = iter(items)
for first in it:
second = next(it)
yield first, second
print(list(pairs([1, 2, 3])))
# ✅ Expected error:
# RuntimeError: generator raised StopIterationGive that next() a default, and the generator decides what happens:
def pairs(items):
it = iter(items)
for first in it:
second = next(it, None)
yield first, second
print(list(pairs([1, 2, 3])))
# ✅ Expected output:
# [(1, 2), (3, None)]return ends a generator, and yield hands out values
In an ordinary function, return hands back one value and the function is done. A generator hands out values with yield and carries on:
def first_even(nums):
for n in nums:
if n % 2 == 0:
return n
def evens(nums):
for n in nums:
if n % 2 == 0:
yield n
print(first_even([3, 4, 7, 6]))
print(list(evens([3, 4, 7, 6])))
# ✅ Expected output:
# 4
# [4, 6]first_even returned 4 and never looked at 6. evens handed out both.
Inside a generator, return ends it, just as reaching the end of the body does:
def read_scores(rows):
for row in rows:
if row == "END":
return
yield int(row)
print(list(read_scores(["3", "5", "END", "9"])))
print(sum(read_scores(["3", "5", "END", "9"])))
# ✅ Expected output:
# [3, 5]
# 8A value after return doesn't become one of the values: with return "stopped early", the list is still [3, 5]. The FAQ at the end of this page says where that value goes.
Writing return where you meant yield turns the generator back into an ordinary function. The call runs it at once and returns a number, and next() on a number fails:
def even_numbers(limit):
for n in range(limit):
if n % 2 == 0:
return n
gen = even_numbers(10)
print(gen)
print(next(gen))
# ✅ Expected output:
# 0
# ✅ Expected error:
# TypeError: 'int' object is not an iteratorThe call returned 0, the first even number, and next(0) stops with the TypeError.
A generator is used up after one pass
Lambda Functions, Comprehensions, enumerate & zip, itertools and File Handling all showed iterators that are empty after one pass, and a generator is the same. What's new is the fix: call the generator function again, and the new generator starts from the beginning:
def evens(limit):
for n in range(0, limit, 2):
yield n
gen = evens(7)
print(sum(gen))
print(sum(gen))
print(sum(evens(7)))
# ✅ Expected output:
# 12
# 0
# 12Each call makes a separate generator with its own place: a = countdown(3) and b = countdown(3) both start at 3, however far a has got.
in uses values up too. It asks for values until it finds a match, and the ones it read are gone:
def evens(limit):
for n in range(0, limit, 2):
yield n
gen = evens(10)
print(4 in gen)
print(list(gen))
# ✅ Expected output:
# True
# [6, 8]That causes a real bug in nested loops. The inner loop uses the generator up during the first size, so the second size gets nothing:
sizes = ["S", "M"]
colours = (c.upper() for c in ["red", "blue"])
for size in sizes:
for colour in colours:
print(size, colour)
print("done")
# ✅ Expected output:
# S RED
# S BLUE
# doneA list can be read again and again, so build one when you need the values more than once:
sizes = ["S", "M"]
colours = [c.upper() for c in ["red", "blue"]]
for size in sizes:
for colour in colours:
print(size, colour)
# ✅ Expected output:
# S RED
# S BLUE
# M RED
# M BLUEA generator can't be tested with if either. Control Flow - If/Else said nearly everything counts as true, and a generator does even when it has no values:
def evens(limit):
for n in range(0, limit, 2):
yield n
gen = evens(0)
if gen:
print("looks like there are values")
print(list(gen))
print(bool([]))
# ✅ Expected output:
# looks like there are values
# []
# FalseTo find out whether there's a value, ask for one with next() and a default; the FAQ at the end shows how to put it back. Unpacking uses values up as well: after first, second, *rest = gen, the generator is empty.
A generator has no length and no positions
A generator makes each value when it's asked for, so it doesn't know how many it will make, and it can't jump to a position:
def evens(limit):
for n in range(0, limit, 2):
yield n
gen = evens(10)
print(len(gen))
# ✅ Expected error:
# TypeError: object of type 'generator' has no len()def evens(limit):
for n in range(0, limit, 2):
yield n
gen = evens(10)
print(gen[0])
# ✅ Expected error:
# TypeError: 'generator' object is not subscriptableThat holds for every generator, not only the generator expressions in Comprehensions. Anything that reads values one by one works: sorted(), max(), sum(), list() and zip(). Each call below makes a fresh generator:
def evens(limit):
for n in range(0, limit, 2):
yield n
print(sorted(evens(10), reverse=True))
print(max(evens(10)), sum(evens(10)))
print(len(list(evens(10))))
first_three = list(zip(range(3), evens(100)))
print(first_three)
# ✅ Expected output:
# [8, 6, 4, 2, 0]
# 8 20
# 5
# [(0, 0), (1, 2), (2, 4)]zip() stopped after three pairs, so evens(100) made only three of its fifty values. To take the first few yourself, use islice() from itertools. The rest stay in the generator for later:
from itertools import islice
def evens(limit):
for n in range(0, limit, 2):
yield n
gen = evens(100)
print(list(islice(gen, 3)))
print(next(gen))
# ✅ Expected output:
# [0, 2, 4]
# 6A pipeline passes one value at a time, and only when asked
Generators can feed each other: each one loops over the one before it. Each value goes through the whole chain before the next one is read:
def read(rows):
for row in rows:
print("read", row)
yield row
def double(nums):
for n in nums:
print("double", n)
yield n * 2
for result in double(read([1, 2])):
print("got", result)
# ✅ Expected output:
# read 1
# double 1
# got 2
# read 2
# double 2
# got 4read 1 is followed by double 1 and got 2, not by read 2. The loop asks double for one value, and double asks read for one.
Building the chain does nothing. The work starts when something asks for a value:
def read(rows):
for row in rows:
print("read", row)
yield row
def double(nums):
for n in nums:
print("double", n)
yield n * 2
pipeline = double(read([1, 2]))
print("pipeline built")
print(next(pipeline))
# ✅ Expected output:
# pipeline built
# read 1
# double 1
# 2Since nothing is made before it's asked for, a generator can go on forever. This is itertools' count(), written by hand; islice(), zip() or a break decides when to stop:
from itertools import islice
def ticket_numbers(start):
n = start
while True:
yield n
n += 1
tickets = ticket_numbers(1001)
print(list(islice(tickets, 3)))
print(next(tickets))
for ticket, name in zip(tickets, ["Ada", "Ben"]):
print(name, ticket)
# ✅ Expected output:
# [1001, 1002, 1003]
# 1004
# Ada 1005
# Ben 1006list(ticket_numbers(1)) would never finish, and in this site's editor the page would stop responding, as it does for an endless loop.
Stopping early also saves the work for the values after it. 7 and 30 are never checked:
def checked(values):
for v in values:
print("checking", v)
yield v
for v in checked([4, 12, 7, 30]):
if v > 10:
print("first over 10:", v)
break
# ✅ Expected output:
# checking 4
# checking 12
# first over 10: 12A generator expression over range(10 ** 12) hands out its first values at once, because nothing is worked out ahead. A list of the same trillion squares would never fit in memory:
squares = (n * n for n in range(10 ** 12))
print(next(squares), next(squares), next(squares))
# ✅ Expected output:
# 0 1 4The same shape works on a file. A for loop over a file reads one line at a time, as File Handling showed. Put the with inside the generator: the file stays open while values are read, and closes when the generator finishes:
with open("app.log", "w", encoding="utf-8") as f:
f.write("INFO start\nERROR disk full\nINFO retry\nERROR no network\n")
def read_lines(path):
with open(path, encoding="utf-8") as f:
for line in f:
yield line.rstrip("\n")
def errors(lines):
for line in lines:
if line.startswith("ERROR"):
yield line
def messages(lines):
for line in lines:
yield line.split(" ", 1)[1]
for message in messages(errors(read_lines("app.log"))):
print(message)
# ✅ Expected output:
# disk full
# no networkReturning a generator expression from inside the with doesn't work. The with closes the file as the function returns, before anything has been read:
with open("app.log", "w", encoding="utf-8") as f:
f.write("INFO start\nERROR disk full\n")
def read_lines(path):
with open(path, encoding="utf-8") as f:
return (line.rstrip("\n") for line in f)
print(list(read_lines("app.log")))
# ✅ Expected error:
# ValueError: I/O operation on closed file.yield inside the with, as read_lines does in the log-file pipeline, keeps the file open until the generator finishes.
yield from hands over each item, not the whole list
yield sub hands over each inner list as one value. yield from sub hands over the values inside it, one by one:
def flatten_wrong(lists):
for sub in lists:
yield sub
def flatten(lists):
for sub in lists:
yield from sub
nested = [[1, 2], [3], []]
print(list(flatten_wrong(nested)))
print(list(flatten(nested)))
# ✅ Expected output:
# [[1, 2], [3], []]
# [1, 2, 3]For one level, itertools' chain.from_iterable() does the same. yield from takes anything a for loop takes: a string hands over its characters, and a range its numbers:
def numbers(limit):
yield from range(limit)
print(list(numbers(3)))
# ✅ Expected output:
# [0, 1, 2]Where yield from goes further than chain is depth. A generator can yield from a call to itself, so it walks nested data however deep it goes. isinstance(), from Inheritance and Polymorphism, tells folders from files:
def walk(folder, path=""):
for name, content in folder.items():
if isinstance(content, dict):
yield from walk(content, path + name + "/")
else:
yield path + name
files = {
"notes.txt": 120,
"photos": {"cat.jpg": 2048, "2024": {"beach.jpg": 4096}},
"todo.txt": 64,
}
for path in walk(files):
print(path)
# ✅ Expected output:
# notes.txt
# photos/cat.jpg
# photos/2024/beach.jpg
# todo.txtDictionaries keep the order their keys were added in, so the paths come out in the order they're written.
yield from needs something a for loop could go through. A number gives the TypeError that for i in 5 gave in Loops - For and While:
def numbers(limit):
yield from limit
print(list(numbers(3)))
# ✅ Expected error:
# TypeError: 'int' object is not iterablefinally runs when a generator is closed or finishes
A generator paused at a yield can be ended with close(). Its finally block runs then, and after that it has nothing left:
def numbers():
try:
yield 1
yield 2
finally:
print("cleanup")
gen = numbers()
print(next(gen))
gen.close()
print("after close")
print(next(gen, "nothing left"))
# ✅ Expected output:
# 1
# cleanup
# after close
# nothing leftIt also runs when the generator finishes:
def numbers():
try:
yield 1
yield 2
finally:
print("cleanup")
for n in numbers():
print(n)
print("after the loop")
# ✅ Expected output:
# 1
# 2
# cleanup
# after the loopDon't count on it when you just stop using a generator halfway. Then it runs only when Python frees the generator, and when that happens isn't up to your program. Call close(), or let the generator finish.
This is what @contextmanager is built on. It turns a generator function into something with can use: it runs the generator up to its one yield when the block starts, and resumes it when the block ends. The finally makes the cleanup run even when the block fails:
from contextlib import contextmanager
@contextmanager
def section(title):
print("--", title, "--")
try:
yield title.upper()
finally:
print("-- end --")
with section("report") as heading:
print("inside", heading)
# ✅ Expected output:
# -- report --
# inside REPORT
# -- end --Context Managers & the with Statement in Depth covers with, and contextlib: contextmanager, suppress, ExitStack covers @contextmanager in full.
Practice: orders, one at a time
read_orders turns each "name,qty" line into a pair, and big_orders keeps the pairs with a quantity of 10 or more. The program takes the first order on its own, passes the rest to big_orders, then shows what's left. Fill the three blanks: the keyword that hands out a value, the built-in function that asks a generator for one value, and what big_orders should read. Each hint line under the code gives one blank, in order.
def read_orders(lines):
for line in lines:
name, qty = line.split(",")
___ name, int(qty)
def big_orders(orders, minimum):
for name, qty in orders:
if qty >= minimum:
yield name, qty
lines = ["cake,12", "tea,2", "bread,20", "jam,5"]
orders = read_orders(lines)
print(___(orders))
for name, qty in big_orders(___, 10):
print(name, qty)
print(list(orders))
# 👉 replace ___ with yield
# 👉 replace ___ with next
# 👉 replace ___ with orders
# ✅ Expected output:
# ('cake', 12)
# bread 20
# []cake isn't printed again, because next() already took it, and the last line is empty because big_orders read the rest. With return in the first blank, the call returns the first pair, and next() stops with TypeError: 'tuple' object is not an iterator. With lines in the third, big_orders gets the strings, and after the first order the program stops with ValueError: too many values to unpack (expected 2).
On your own: numbers from a file
Write a file readings.txt holding the lines 12, an empty line, 7 and 30. Then write a generator function numbers_in(path) that opens the file with with and hands out each line that isn't blank as an int. Print the total and the largest, then all the numbers as a list, calling numbers_in() again for each.
# Write readings.txt, then the numbers_in generator function.
# your code herewith open("readings.txt", "w", encoding="utf-8") as f:
f.write("12\n\n7\n30\n")
def numbers_in(path):
with open(path, encoding="utf-8") as f:
for line in f:
if line.strip():
yield int(line)
print(sum(numbers_in("readings.txt")))
print(max(numbers_in("readings.txt")))
print(list(numbers_in("readings.txt")))
# ✅ Expected output:
# 49
# 30
# [12, 7, 30]With one generator for both, as in nums = numbers_in("readings.txt"), sum(nums) prints 49 and max(nums) then stops with a ValueError: the sum used every value. max(nums, default="nothing left") prints the default instead. Without the check for blank lines, int() stops at the empty line with ValueError: invalid literal for int() with base 10: '\n'.
Practice quiz
countdown(start) prints starting, then yields start, start - 1 and so on down to 1. What does gen = countdown(2), then print("made"), then print(next(gen)) print, in order?
- starting, made, 2
- made, 2
- made, starting, 2
- starting, 2, made
Answer: made, starting, 2. Calling countdown runs none of its body, so made comes first. next() starts the body, which prints starting, then hands out 2.
evens(limit) yields 0, 2, 4 and so on, below limit. What does gen = evens(7), then print(sum(gen), sum(gen)) print?
- 12 12
- 12 0
- 0 12
- It stops with StopIteration
Answer: 12 0. The first sum() uses every value. The second finds the generator empty, and the sum of nothing is 0.
evens(limit) yields 0, 2, 4 and so on, below limit. What does gen = evens(10), then print(4 in gen, list(gen)) print?
- True [0, 2, 4, 6, 8]
- True []
- False [6, 8]
- True [6, 8]
Answer: True [6, 8]. in reads 0, 2 and 4 to find the match, and those are gone, so list() gets only what's left.
sizes is ["S", "M"] and colours is the generator expression (c.upper() for c in ["red", "blue"]). A loop over sizes holds a loop over colours that adds 1 to count each time. What does print(count) show?
- 2
- 4
- 0
- It stops with a TypeError
Answer: 2. The inner loop uses colours up while size is S, so there is nothing left for M: 2, not 4.
flatten(lists) loops over lists and does yield sub for each one, not yield from sub. What does print(list(flatten([[1, 2], [3]]))) show?
- [1, 2, 3]
- [[1, 2, 3]]
- [[1, 2], [3]]
- It stops with TypeError: 'list' object is not an iterator
Answer: [[1, 2], [3]]. yield sub hands over each inner list as one value. yield from sub would hand over the numbers in it.
read_scores(rows) loops over rows: at "END" it does return "stopped", and otherwise it yields int(row). What does print(list(read_scores(["3", "5", "END", "9"]))) show?
- [3, 5]
- [3, 5, 9]
- [3, 5, 'stopped']
- It stops with a ValueError
Answer: [3, 5]. return ends the generator, so 9 is never read, and the returned value isn't one of the values.
read(rows) prints read and the row, then yields it. double(nums) prints double and the number, then yields it times 2. A loop over double(read([1, 2])) prints got and each result. What is printed, in order?
- read 1, read 2, double 1, double 2, got 2, got 4
- read 1, double 1, read 2, double 2, got 2, got 4
- got 2, got 4
- read 1, double 1, got 2, read 2, double 2, got 4
Answer: read 1, double 1, got 2, read 2, double 2, got 4. Each value goes through the whole chain before the next one is read: the loop asks double for one value, and double asks read for one.
read_lines(path) opens the file with with and, inside the with, returns (line.strip() for line in f). a.txt holds the lines x and y. What does print(list(read_lines("a.txt"))) do?
- It prints ['x', 'y']
- It stops with ValueError: I/O operation on closed file.
- It prints []
- It prints ['x ', 'y ']
Answer: It stops with ValueError: I/O operation on closed file.. The with closes the file as the function returns, before list() reads anything. A yield inside the with keeps the file open until the generator finishes.
Continue this course
- Previous: Context Managers & the with Statement in Depth
- Next: AsyncIO: Event Loop, Tasks & Futures — Deep dive into the AsyncIO event loop, tasks, and concurrent I/O
- Quick reference: Python cheat sheet
Frequently asked questions
Is range() a generator?
No. A range can be read again and has a length and positions: list(r) twice gives [0, 1, 2, 3, 4] both times, and len(r), r[2] and 3 in r give 5, 2 and True. It isn't an iterator either, so next(range(5)) stops with TypeError: 'range' object is not an iterator. A for loop works on it because the loop calls iter() first.
Why can I loop over prices.items() twice, but not over a zip?
prices.items() isn't an iterator: each loop or list() gets a fresh one, so list(items) twice gives [('tea', 2), ('cake', 3)] both times. A zip is an iterator, so after the first list() the second gives [].
How do I look at the first value without losing it?
Take it with next(gen) inside a try, then put it back in front with chain() from itertools. For evens(7), first = next(gen) gives 0, and list(chain([first], gen)) gives [0, 2, 4, 6]. For an empty generator, such as evens(0), next(gen) raises StopIteration instead, so catch it there: there's nothing to put back. Don't use a default such as next(gen, None) for this. It can't tell an empty generator from one whose first value is None, and putting that None back turns an empty generator into [None].
Where does a generator's return value go?
Into the StopIteration that ends the generator, as its value: after return "all done", catching the error with except StopIteration as e and printing e.value prints all done. yield from hands it to the generator that delegated: in status = yield from read_scores(rows), status is "stopped early". You'll rarely need either.
What are send(), throw() and close()?
close() ends a paused generator and runs its finally, as this lesson shows. send(x) resumes the generator like next() does, and makes x the value of the paused yield: a running total written as amount = yield total prints 0, 5 and 8 for next(), send(5) and send(3). Sending a value before the first next() stops with TypeError: can't send non-None value to a just-started generator. throw() raises an error at the paused yield, where the generator can catch it. Before async def, coroutines were written as generators driven this way; Advanced Async & Await Patterns covers async def.
Can my own class be looped over again and again?
Yes, if its __iter__ is a generator function: each loop calls __iter__ again and gets a fresh generator. A Playlist class whose __iter__ yields each song's title gives ['Intro', 'Outro'] for list(mix) twice. Iterators: __iter__ & __next__ covers iterator classes.
What happens to a generator when its body raises an error?
The error comes out of the next() that was running it, and the generator is finished. For a generator that yields int(row) for each of ["1", "x", "3"], the first next() gives 1, the second stops with ValueError: invalid literal for int() with base 10: 'x', and next(gen, "finished") then gives finished, so 3 is never read.
Is a generator faster than a list, and how much memory does it save?
Both give the same result: sum([n * n for n in nums]) and sum(n * n for n in nums) are both 52 for [3, 1, 4, 1, 5]. Which is faster depends on the Python version and the data, so measure it, as Benchmarking with timeit shows. A generator avoids storing every output at once, but its local variables can retain large inputs or intermediate objects. Memory Management & Garbage Collection Internals measures total memory properly.