Advanced Collections, itertools & functools
Published by Brayan K
Master Python's most powerful standard library modules for building high-performance, memory-efficient code.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Three modules, and the parts of them this lesson covers
collections, itertools and functools come with Python, so they need only an import. Several of their tools have lessons of their own:
- count(), cycle(), islice(), chain(), groupby(), accumulate() and the combinations: itertools.
- Counter and defaultdict, for counting and grouping: Counter & defaultdict.
- deque, a sequence that adds and removes quickly at both ends: deque & Queues.
- namedtuple, a tuple whose fields have names: namedtuple & NamedTuple.
- lru_cache, cache, partial, reduce and cached_property: functools: lru_cache, partial, reduce.
- total_ordering, which fills in the missing comparison methods of a class: Operator Overloading & Custom Behaviours.
This lesson covers parts no other lesson does. From collections: ChainMap, which looks a key up in several dictionaries; what OrderedDict still does now that every dictionary keeps its order; and UserDict and its relatives, for writing a class that works like a dictionary, a list or a string. From itertools: starmap(), repeat(), compress() and tee(). From functools: singledispatch, which picks a function by the type of its argument, and cmp_to_key, which sorts by comparing two items. Near the end, "One job with all three modules" puts them to work in one program.
ChainMap looks a key up in several dictionaries, in order
A program's settings often come in layers: the defaults, what the user saved last time, and what they chose for this run. ChainMap(this_run, saved, defaults) keeps the three dictionaries as they are, and looks a key up in each one in turn, from the first to the last. The first dictionary that has the key wins:
from collections import ChainMap
defaults = {"theme": "light", "font": "serif", "size": 12}
saved = {"theme": "dark"}
this_run = {"size": 16}
settings = ChainMap(this_run, saved, defaults)
print(settings["theme"], settings["font"], settings["size"])
print(len(settings))
print(dict(settings))
# ✅ Expected output:
# dark serif 16
# 3
# {'theme': 'dark', 'font': 'serif', 'size': 16}"theme" came from saved, "font" from defaults and "size" from this_run. That's the opposite of | in Dictionaries, where the right-hand dictionary wins: dict(settings) is the same dictionary as defaults | saved | this_run, with the layers written the other way round. len() counts each key once, however many layers have it, and dict() flattens the chain into one ordinary dictionary.
Looking a key up searches every layer, but changing one doesn't. In the words of the collections documentation: "Lookups search the underlying mappings successively until a key is found. In contrast, writes, updates, and deletions only operate on the first mapping."
from collections import ChainMap
defaults = {"theme": "light", "size": 12}
this_run = {}
settings = ChainMap(this_run, defaults)
settings["size"] = 16
print(this_run, defaults)
del settings["size"]
print(settings["size"])
del settings["size"]
# ✅ Expected output:
# {'size': 16} {'theme': 'light', 'size': 12}
# 12
# ✅ Expected error:
# KeyError: "Key not found in the first mapping: 'size'"settings["size"] = 16 added the key to this_run and left defaults alone. The first del removed it from this_run again, so the lookup found 12 in defaults. The second del found nothing to delete in the first dictionary, and a ChainMap never deletes from the others, so it stopped with a KeyError at that line.
That's what keeps the layers safe: with defaults further down the chain, no assignment or del through the ChainMap changes it. To change a value where it lives, change that dictionary by its own name.
A ChainMap is a live view; | makes a copy
A ChainMap holds the dictionaries themselves, not a copy of what was in them, so a change to one of them shows through at once. A merge with | is a new dictionary, made once:
from collections import ChainMap
defaults = {"theme": "light"}
this_run = {}
merged = defaults | this_run
settings = ChainMap(this_run, defaults)
defaults["theme"] = "blue"
this_run["font"] = "mono"
print(merged)
print(settings["theme"], settings["font"])
# ✅ Expected output:
# {'theme': 'light'}
# blue monomerged kept the old theme, while settings saw the new theme and the new font. Use a ChainMap when the layers may still change and you want the current answer, and | or dict() when you want a snapshot.
new_child() returns a new ChainMap with one more dictionary in front, which suits a change you want to try without keeping it. maps is the list of dictionaries, the first one first, and parents is the chain without its first dictionary:
from collections import ChainMap
settings = ChainMap({"theme": "dark"}, {"theme": "light", "size": 12})
preview = settings.new_child({"size": 20})
preview["theme"] = "high contrast"
print(preview["theme"], preview["size"])
print(settings["theme"], settings["size"])
print(preview.maps[0])
print(preview.parents["theme"])
# ✅ Expected output:
# high contrast 20
# dark 12
# {'size': 20, 'theme': 'high contrast'}
# darkThe preview's write went into its own front dictionary, preview.maps[0], so settings never saw it, and preview.parents looked past that dictionary to find "dark". The practice exercise further down uses new_child() too.
Command-line options in front of saved settings
Building Command-Line Tools with argparse & Typer read options into a Namespace, and vars(args) turned that into a dictionary. It makes a natural front layer, with one catch: an option that wasn't given is in that dictionary too, holding its default=, or None when it has none. To a ChainMap, None is a value like any other:
import argparse
from collections import ChainMap
defaults = {"theme": "light", "size": 12}
saved = {"theme": "dark"}
parser = argparse.ArgumentParser()
parser.add_argument("--theme")
parser.add_argument("--size", type=int)
args = parser.parse_args(["--size", "16"])
print(vars(args))
settings = ChainMap(vars(args), saved, defaults)
print(settings["theme"], settings["size"])
given = {name: value for name, value in vars(args).items() if value is not None}
settings = ChainMap(given, saved, defaults)
print(settings["theme"], settings["size"])
# ✅ Expected output:
# {'theme': None, 'size': 16}
# None 16
# dark 16With vars(args) in front, the lookup for "theme" stopped at its None and never reached "dark". Leaving out the options that are None, as given does, lets the lookup fall through to the saved setting. The collections documentation gives the same recipe for command-line options in front of defaults.
The filter drops only None, so it suits options declared without a default=. An option declared with default=12, or an action="store_true" flag, which holds False when it isn't given, gets through and still hides the saved setting. Declared with default=None, such a flag holds None until it's given.
OrderedDict: order counts in ==, and keys move to either end
Since Python 3.7, every dictionary keeps its keys in the order they were added, as Dictionaries showed, so you no longer need an OrderedDict for that. Its documentation says ordered dictionaries "have become less important" since then, and lists what's still different. Two differences matter most. The first: two OrderedDicts are equal only when their keys are in the same order.
from collections import OrderedDict
a = {"x": 1, "y": 2}
b = {"y": 2, "x": 1}
print(a == b)
print(OrderedDict(a) == OrderedDict(b))
print(OrderedDict(a) == b)
# ✅ Expected output:
# True
# False
# TrueThe two plain dictionaries were equal, and so were an OrderedDict and a plain dictionary. Order counts only when both sides are OrderedDicts.
The second: an OrderedDict can move a key to either end, and remove a pair from either end. move_to_end(key) moves the key to the end, and move_to_end(key, last=False) to the front. popitem() removes the last pair and returns it, as a dictionary's does, and popitem(last=False) removes the first:
from collections import OrderedDict
tabs = OrderedDict(home="Home", docs="Docs", blog="Blog")
tabs.move_to_end("home")
print(list(tabs))
tabs.move_to_end("blog", last=False)
print(list(tabs))
print(tabs.popitem(last=False))
print(tabs.popitem())
print(list(tabs))
# ✅ Expected output:
# ['docs', 'blog', 'home']
# ['blog', 'docs', 'home']
# ('blog', 'Blog')
# ('home', 'Home')
# ['docs']The examples print list(tabs), the keys in order, rather than tabs itself, because an OrderedDict prints differently on different Pythons. print(OrderedDict(b=2, a=1)) shows OrderedDict([('b', 2), ('a', 1)]) on Python 3.11 and in this site's editor, and OrderedDict({'b': 2, 'a': 1}) from Python 3.12 on.
move_to_end() needs a key that's there, and keys are compared exactly, capitals included:
from collections import OrderedDict
tabs = OrderedDict(home="Home", docs="Docs")
tabs.move_to_end("Home")
# ✅ Expected error:
# KeyError: 'Home'A plain dictionary can manage two of these moves, as the documentation notes: tabs[key] = tabs.pop(key) moves a key to the end, and next(iter(tabs)) gives the first key, ready to pop. It has no efficient way to move a key to the front. The documentation adds that OrderedDict "can handle frequent reordering operations better than dict", which "makes it suitable for implementing various kinds of LRU caches": a cache that, when full, drops what was used least recently. The exercise "On your own: a recently opened list", at the end of this lesson, keeps file names that way. To cache a function's results, use the ready-made lru_cache from functools: lru_cache, partial, reduce.
Your own dictionary class: dict's methods skip your __setitem__, UserDict's don't
Inheritance and Polymorphism answered, in its questions at the end, that a class can be built on list, and dict works the same way. Say you want a dictionary that stores every key in lower case. d[key] = value calls the method __setitem__(key, value), the way d[key] calls __getitem__ in Magic Methods & the Python Data Model, so overriding __setitem__ looks like enough:
class LowerKeys(dict):
def __setitem__(self, key, value):
super().__setitem__(key.lower(), value)
stock = LowerKeys({"Tea": 3})
stock["Milk"] = 1
stock.update({"Jam": 2})
print(stock)
# ✅ Expected output:
# {'Tea': 3, 'milk': 1, 'Jam': 2}Only "Milk" was lowered. dict's own methods, the constructor and update() among them, store their pairs directly, without calling your __setitem__, so "Tea" and "Jam" kept their capitals, and nothing reported a problem.
collections.UserDict is written for this job. It keeps a real dictionary in its data attribute, and its constructor and update() store each pair with self[key] = value, which calls your __setitem__:
from collections import UserDict
class LowerKeys(UserDict):
def __setitem__(self, key, value):
super().__setitem__(key.lower(), value)
stock = LowerKeys({"Tea": 3})
stock["Milk"] = 1
stock.update({"Jam": 2})
print(stock)
print(stock.data)
print(isinstance(stock, dict))
# ✅ Expected output:
# {'tea': 3, 'milk': 1, 'jam': 2}
# {'tea': 3, 'milk': 1, 'jam': 2}
# FalseAll three keys were lowered, and stock.data is the plain dictionary inside. The documentation says UserDict "can be easier to work with because the underlying dictionary is accessible as an attribute".
A UserDict isn't a dict, though: isinstance(stock, dict) is False, so code that checks for a dict won't accept one. The type that covers dictionaries, UserDicts and ChainMaps alike is collections.abc.Mapping, from Mixins, Multiple Inheritance & OOP Patterns; the singledispatch part of this lesson uses it.
UserList and UserString wrap a list and a string the same way, in .data. They fix the catch that Inheritance and Polymorphism's answer mentioned: a class built on list gets a plain list back from + or a slice. type(x).__name__ gives the name of x's class:
from collections import UserList, UserString
class Playlist(UserList):
pass
class PlainPlaylist(list):
pass
class Tag(UserString):
pass
class PlainTag(str):
pass
songs = Playlist(["a", "b"])
plain = PlainPlaylist(["a", "b"])
print(type(songs + ["c"]).__name__, type(songs[:1]).__name__)
print(type(plain + ["c"]).__name__, type(plain[:1]).__name__)
print(type(Tag("py").upper()).__name__, type(PlainTag("py").upper()).__name__)
# ✅ Expected output:
# Playlist Playlist
# list list
# Tag strThe classes built on UserList and UserString gave back a Playlist and a Tag. The ones built on list and str gave back a plain list and a plain string.
__missing__ answers d[key] for a missing key, and nothing else
When d[key] doesn't find the key, a dict subclass gets one more chance: if it has a __missing__(key) method, d[key] returns whatever that returns, instead of raising a KeyError:
class Stock(dict):
def __missing__(self, key):
return 0
stock = Stock(tea=3)
print(stock["tea"], stock["jam"])
print(stock.get("jam"), "jam" in stock)
print(stock)
# ✅ Expected output:
# 3 0
# None False
# {'tea': 3}stock["jam"] gave 0, but get() returned None and in said False: neither calls __missing__. And __missing__ stored nothing, so stock still holds one pair. This hook is how defaultdict and Counter answer a missing key; Counter & defaultdict covers both, including the way a defaultdict stores the value it makes.
A UserDict calls __missing__ too, with one difference between versions. With the same __missing__ on a class built on UserDict, stock.get("jam") gave 0 on Python 3.11 and in this site's editor, and None on 3.12 to 3.14: Python 3.12 stopped UserDict's get() calling __missing__, to match dict's.
starmap() unpacks each tuple into arguments
Higher-Order Functions & Function Factories answered, in its questions at the end, that map() can take two lists, one item from each per call. When the arguments are already paired up in tuples, map() passes each tuple whole, as one argument. itertools.starmap() spreads each tuple out into the function's arguments, as line_total(*pair) would, with the star from *args & **kwargs. The documentation describes it as "used instead of map() when argument parameters have already been 'pre-zipped' into tuples":
from itertools import starmap
def line_total(pence, qty):
return pence * qty
orders = [(250, 2), (120, 3), (99, 10)]
print(list(starmap(line_total, orders)))
print(list(map(line_total, [250, 120, 99], [2, 3, 10])))
print(list(map(line_total, orders)))
# ✅ Expected output:
# [500, 360, 990]
# [500, 360, 990]
# ✅ Expected error:
# TypeError: line_total() missing 1 required positional argument: 'qty'starmap() over the pairs and map() over two separate lists gave the same totals. map() over the pairs passed (250, 2) as pence, and stopped because qty was missing. Like map(), starmap() returns an iterator, so list() collects the results.
repeat() supplies one value as often as needed; compress() keeps flagged items
To pass the same second argument to every call, map() needs a second input as long as the first. repeat(x) gives x again and again, for as long as something asks, and repeat(x, n) gives it n times:
from itertools import repeat
def with_vat(pence, rate):
return pence + pence * rate // 100
prices = [250, 120, 99]
print(list(map(with_vat, prices, repeat(20))))
print(list(zip(prices, repeat("GBP"))))
print(list(repeat("-", 3)))
# ✅ Expected output:
# [300, 144, 118]
# [(250, 'GBP'), (120, 'GBP'), (99, 'GBP')]
# ['-', '-', '-']map() and zip() stop at their shortest input, so the endless repeat(20) stopped when prices ran out. Only repeat("-", 3), on its own, needed a count.
compress(items, flags) keeps each item whose flag is true, and stops when either input runs out:
from itertools import compress
names = ["Ada", "Ben", "Cleo", "Dev"]
paid = [True, False, True, False]
print(list(compress(names, paid)))
print(list(compress(names, [0, 1, 1])))
# ✅ Expected output:
# ['Ada', 'Cleo']
# ['Ben', 'Cleo']The second call had only three flags, so "Dev" was never looked at. compress() suits flags you already have in a list of their own, such as who has paid; filter() suits a test you can write as a function.
tee() splits one iterator in two; after that, read only the copies
Generators & Iterators Mastery showed that a generator is used up after one pass. tee(iterator) returns two new iterators that each give every value, so two readers can share one pass:
from itertools import tee
readings = (n * 10 for n in [1, 2, 3, 4])
first, second = tee(readings)
print(sum(first), max(second))
print(list(readings))
# ✅ Expected output:
# 100 40
# []sum() and max() each saw all four readings, and list(readings) was empty: tee had read the original to fill its copies.
tee reads from the original only when a copy asks for a value it hasn't fetched yet. So anything else that reads the original takes values the copies never see:
from itertools import tee
readings = (n * 10 for n in [1, 2, 3, 4])
first, second = tee(readings)
print(next(readings))
print(list(first), list(second))
# ✅ Expected output:
# 10
# [20, 30, 40] [20, 30, 40]next(readings) took the 10 before either copy asked for it, so both copies started at 20. After tee(), use only the copies.
tee keeps every value that one copy has read and the other hasn't yet. Its documentation says: "In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee()." sum() above read everything before max() began, so a list would have done as well. tee is for copies that move forward side by side, where it only holds the values between them.
The itertools documentation for Python 3.12 and earlier shows a recipe that uses tee to split one input in two by a test, with filterfalse() from itertools:
from itertools import tee, filterfalse
def partition(test, items):
for_false, for_true = tee(items)
return filterfalse(test, for_false), filter(test, for_true)
def is_late(minutes):
return minutes > 15
waits = [5, 20, 12, 30, 8]
on_time, late = partition(is_late, waits)
print(list(on_time), list(late))
# ✅ Expected output:
# [5, 12, 8] [20, 30]Each wait went to exactly one of the two results, and the input was read once. Here list(on_time) read everything before list(late) started, so tee held all five waits in the meantime, which is fine for a short list.
singledispatch picks a function by the first argument's type
With isinstance() from Inheritance and Polymorphism, a function can choose what to do by type, and Higher-Order Functions & Function Factories looked functions up in a dictionary. functools.singledispatch joins the two: you write one version of a function per type, and each call runs the version for its first argument's type. The function under @singledispatch is the fallback, for types with no version of their own. @describe.register adds a version, filed under the type in its first parameter's annotation, the kind of hint from Type Hints & Static Typing with mypy:
from functools import singledispatch
@singledispatch
def describe(value):
return "something else"
@describe.register
def _(value: int):
return f"the number {value}"
@describe.register
def _(value: list):
return f"a list of {len(value)}"
print(describe(7))
print(describe([1, 2, 3]))
print(describe(7.5))
# ✅ Expected output:
# the number 7
# a list of 3
# something else7 is an int and [1, 2, 3] a list, so each found its version. 7.5, a float, had none, and got the fallback. The versions are named _ because nothing calls them by name: a call to describe finds them by type.
Like isinstance(), singledispatch counts a subclass as a kind of its parent class. A type with no version of its own gets its parent's, or its parent's parent's, and so on up to the fallback. bool is a subclass of int, so True finds the int version. describe.dispatch(bool) returns the function a bool would get. Print its __name__, because printing the function itself shows a memory address that changes from run to run:
from functools import singledispatch
@singledispatch
def describe(value):
return "something else"
@describe.register
def describe_number(value: int):
return f"the number {value}"
print(describe(True))
print(describe.dispatch(bool).__name__)
@describe.register
def describe_flag(value: bool):
return "yes" if value else "no"
print(describe(True), describe(False), describe(0))
# ✅ Expected output:
# the number True
# describe_number
# yes no the number 0Until a bool version was registered, True got the int version's the number True. After it, True and False had their own, while 0, an int, still got the number.
A class from collections.abc works as well. Registering collections.abc.Mapping, the class from Mixins, Multiple Inheritance & OOP Patterns, catches every kind of mapping, the ones from this lesson included:
from collections import ChainMap, OrderedDict, UserDict
from collections.abc import Mapping
from functools import singledispatch
@singledispatch
def keys_of(value):
return "not a mapping"
@keys_of.register
def _(value: Mapping):
return sorted(value)
print(keys_of({"b": 1, "a": 2}))
print(keys_of(ChainMap({"b": 1}, {"a": 2})))
print(keys_of(UserDict(c=3)))
print(keys_of(OrderedDict(z=0)))
print(keys_of(["a", "b"]))
# ✅ Expected output:
# ['a', 'b']
# ['a', 'b']
# ['c']
# ['z']
# not a mappingA dict, a ChainMap, a UserDict and an OrderedDict are all mappings, so one version served them all, and sorted() of a mapping sorts its keys. A list isn't a mapping, and got the fallback. Abstract Base Classes (abc) covers Mapping and its relatives.
singledispatch reads the first positional argument, and register() needs a class
singledispatch looks only at the type of the first argument passed by position. Called by keyword, it has nothing in that place to look at, so the call stops at once:
from functools import singledispatch
@singledispatch
def describe(value):
return "something else"
print(describe(value=7))
# ✅ Expected error:
# TypeError: describe requires at least 1 positional argumentPass the value by position, as in describe(7). Any other arguments come after it, and their types are never looked at.
The annotation that register reads must be a class. list[int] is a hint about what a list holds, and Type Hints & Static Typing with mypy showed that isinstance() refuses one. register refuses it too, as soon as the decorator runs:
from functools import singledispatch
@singledispatch
def describe(value):
return "something else"
@describe.register
def _(value: list[int]):
return f"{len(value)} numbers"
# ✅ Expected error:
# TypeError: Invalid annotation for 'value'. list[int] is not a class.The fix is to name the class in register() and keep the detailed hint on the parameter, for readers and for type checkers. @describe.register(cls) hands your function back unchanged, so two of them can be stacked, as in Decorators & Advanced Features, to file one function under two types. In the hint, tuple[int, ...] means a tuple of ints of any length:
from functools import singledispatch
@singledispatch
def describe(value):
return "something else"
@describe.register(list)
@describe.register(tuple)
def _(value: list[int] | tuple[int, ...]):
return f"{len(value)} numbers"
print(describe([1, 2]), describe((1, 2, 3)))
# ✅ Expected output:
# 2 numbers 3 numbersThe list and the tuple both found that one version, filed under list and under tuple.
With no annotation at all, a bare @describe.register can't tell which class you mean. It stops with a TypeError that starts Invalid first argument to `register()`:, then names your function with a memory address that changes from run to run, and ends Use either `@register(some_class)` or plain `@register` on an annotated function.
cmp_to_key sorts with a function that compares two items
A key function, the kind sorted() takes as key=, looks at one item at a time. Some orders are a rule about pairs instead. To join the pieces "3", "30", "34", "5" and "9" into the biggest number, "3" must come before "30", because "330" is bigger than "303". A comparison function says that directly: it takes two items and returns a negative number when the first should come first, a positive number when the second should, and zero when either order will do. functools.cmp_to_key() turns one into a key:
from functools import cmp_to_key
def bigger_first(a, b):
if a + b > b + a:
return -1
if a + b < b + a:
return 1
return 0
parts = ["3", "30", "34", "5", "9"]
print("".join(sorted(parts, reverse=True)))
print("".join(sorted(parts, key=cmp_to_key(bigger_first))))
# ✅ Expected output:
# 9534303
# 9534330Sorting the strings in reverse put "30" before "3", which gives 9534303. The comparison put "3" first, for 9534330, the bigger number.
The comparison must return a number below, at or above zero. A function that returns True or False is an easy mistake to make, and nothing reports it:
from functools import cmp_to_key
def shorter_first(a, b):
return len(a) < len(b)
words = ["banana", "fig", "apple", "kiwi"]
print(sorted(words, key=cmp_to_key(shorter_first)))
print(sorted(words, key=len))
# ✅ Expected output:
# ['banana', 'fig', 'apple', 'kiwi']
# ['fig', 'kiwi', 'apple', 'banana']sorted() only ever asks whether one item is less than another, and the key from cmp_to_key answers yes when your function returned a negative number. True and False count as 1 and 0, never negative, so no word counted as smaller than another and the list came back as it was. key=len, one word at a time, was all this order needed.
The opposite mistake is passing a two-item function straight to key=, which calls it with one item:
def shorter_first(a, b):
return len(a) - len(b)
print(sorted(["banana", "fig"], key=shorter_first))
# ✅ Expected error:
# TypeError: shorter_first() missing 1 required positional argument: 'b'Python 2's sorted() took a comparison function as cmp=. Python 3 has no such argument: sorted(words, cmp=...) stops with TypeError: 'cmp' is an invalid keyword argument for sort() on Python 3.11 and 3.12 and in this site's editor, and with TypeError: sort() got an unexpected keyword argument 'cmp' on 3.13 and 3.14. The documentation calls cmp_to_key "primarily used as a transition tool for programs being converted from Python 2", so use a plain key= whenever one will do. Sorting with sorted() & key covers key functions.
One job with all three modules
An order is a list of (item, price in pence, quantity) tuples. The program below reads the currency, and whether to show lines that come to nothing, from a ChainMap of settings. It works out each line's total with starmap(), drops the empty lines with compress(), and formats the totals with a singledispatch function that shows an int as money. Prices are whole pence, so // and % split them into pounds and pence, and {value % 100:02} pads the pence to two digits, as in f-strings & String Formatting:
from collections import ChainMap
from functools import singledispatch
from itertools import compress, starmap
defaults = {"currency": "£", "show_empty": False}
this_run = {"currency": "€"}
settings = ChainMap(this_run, defaults)
orders = [("tea", 250, 2), ("cake", 320, 0), ("jam", 199, 3)]
def line_total(item, pence, qty):
return item, pence * qty
lines = list(starmap(line_total, orders))
wanted = [settings["show_empty"] or total > 0 for item, total in lines]
@singledispatch
def show(value):
return str(value)
@show.register
def _(value: int):
return f"{settings['currency']}{value // 100}.{value % 100:02}"
for item, total in compress(lines, wanted):
print(item, show(total))
print("hidden:", wanted.count(False))
# ✅ Expected output:
# tea €5.00
# jam €5.97
# hidden: 1The cake line has a quantity of 0, so its total was 0 and its flag False: compress() left it out, and wanted.count(False) counted it. The currency came from this_run, in front of the defaults. With "show_empty": True added to this_run, the same program printed all three lines, the cake's as €0.00.
Practice: preview a setting without saving it
Fill in the two blanks so that settings looks a key up in saved and then defaults, and preview puts a layer for this run in front of it. If you get stuck, the hints under the code name the class and the method.
from collections import ChainMap
defaults = {"theme": "light", "font": "serif", "size": 12}
saved = {"theme": "dark"}
settings = ___(saved, defaults)
preview = settings.___({"size": 20})
preview["font"] = "mono"
print(preview["theme"], preview["font"], preview["size"])
print(settings["theme"], settings["font"], settings["size"])
print(saved)
# 👉 replace ___ with ChainMap
# 👉 replace ___ with new_child
# ✅ Expected output:
# dark mono 20
# dark serif 12
# {'theme': 'dark'}On your own: a recently opened list
A list holds file names in the order they were opened, some of them more than once: opens = ["notes.txt", "todo.txt", "notes.txt", "plan.md", "data.csv", "notes.txt"]. Using an OrderedDict, keep the three most recently opened names, oldest first, each with how many times it was opened. Reopening a name moves it to the end and adds 1 to its count. When a new name would make four, forget the oldest and print forget and its name. Print the list of names after each open, and at the end the names and counts as a dictionary.
from collections import OrderedDict
opens = ["notes.txt", "todo.txt", "notes.txt", "plan.md", "data.csv", "notes.txt"]
recent = OrderedDict()
for name in opens:
if name in recent:
recent.move_to_end(name)
recent[name] += 1
else:
recent[name] = 1
if len(recent) > 3:
oldest, count = recent.popitem(last=False)
print("forget", oldest)
print(list(recent))
print(dict(recent))
# ✅ Expected output:
# ['notes.txt']
# ['notes.txt', 'todo.txt']
# ['todo.txt', 'notes.txt']
# ['todo.txt', 'notes.txt', 'plan.md']
# forget todo.txt
# ['notes.txt', 'plan.md', 'data.csv']
# ['plan.md', 'data.csv', 'notes.txt']
# {'plan.md': 1, 'data.csv': 1, 'notes.txt': 3}
Practice quiz
defaults = {"size": 12} and this_run = {}. After settings = ChainMap(this_run, defaults) and settings["size"] = 20, what does print(this_run, defaults) print?
- {'size': 20} {'size': 12}
- {} {'size': 20}
- {'size': 20} {'size': 20}
- {} {'size': 12}
Answer: {'size': 20} {'size': 12}. A ChainMap writes only to its first dictionary, this_run here. defaults still says 12, and settings["size"] now finds 20 first.
What does print(OrderedDict(a=1, b=2) == OrderedDict(b=2, a=1), dict(a=1, b=2) == dict(b=2, a=1)) print?
- True True
- False True
- False False
- True False
Answer: False True. Two OrderedDicts are equal only when their keys are in the same order. Two plain dictionaries ignore the order.
class LowerKeys(dict) overrides __setitem__ to store key.lower(). After stock = LowerKeys(), stock["Tea"] = 3 and stock.update({"Jam": 2}), what does print(stock) print?
- {'tea': 3, 'jam': 2}
- {'Tea': 3, 'Jam': 2}
- {'tea': 3, 'Jam': 2}
- It stops with a KeyError
Answer: {'tea': 3, 'Jam': 2}. stock["Tea"] = 3 calls your __setitem__, but dict's own update() stores its pairs without it. Built on UserDict, the same class gives {'tea': 3, 'jam': 2}.
class Stock(dict) has def __missing__(self, key): return 0. With stock = Stock(tea=3), what does print(stock["jam"], stock.get("jam"), len(stock)) print?
- 0 None 1
- 0 0 1
- 0 0 2
- It stops with a KeyError
Answer: 0 None 1. __missing__ answers stock["jam"] and nothing else: get() doesn't call it, and nothing is stored, so stock still holds one pair.
def line_total(pence, qty): return pence * qty. What does print(list(map(line_total, [(250, 2), (120, 3)]))) do?
- It prints [500, 360]
- It stops with TypeError: line_total() missing 1 required positional argument: 'qty'
- It prints [(250, 2), (120, 3)]
- It prints [250, 120]
Answer: It stops with TypeError: line_total() missing 1 required positional argument: 'qty'. map() passes each tuple whole, as pence, so qty is missing. starmap(line_total, ...) unpacks each tuple and gives [500, 360].
nums = iter([1, 2, 3]), then a, b = tee(nums), then next(nums). What does print(list(a), list(b)) print?
- [1, 2, 3] [1, 2, 3]
- [] []
- [2, 3] [1, 2, 3]
- [2, 3] [2, 3]
Answer: [2, 3] [2, 3]. The copies take their values from nums when they need them. next(nums) took the 1 before either copy asked, so both start at 2.
label() is a singledispatch function whose fallback returns "other", with versions registered for int (returning "int") and str (returning "str"). What does print(label(True), label(2.0), label("2")) print?
- other other str
- bool other str
- int other str
- int int str
Answer: int other str. bool is a subclass of int, so True gets the int version. A float isn't an int and has no version of its own, so 2.0 gets the fallback.
def shorter_first(a, b): return len(a) < len(b). What does print(sorted(["ccc", "a", "bb"], key=cmp_to_key(shorter_first))) print?
- ['a', 'bb', 'ccc']
- ['ccc', 'bb', 'a']
- It stops with a TypeError
- ['ccc', 'a', 'bb']
Answer: ['ccc', 'a', 'bb']. sorted() only asks whether one item is less than another, and cmp_to_key says yes when the function returns a negative number. True and False are 1 and 0, so nothing counts as smaller and nothing moves.
Continue this course
- Previous: Working with Files, Streams & Large Datasets
- Next: Functional Programming Techniques in Python — Write pure, composable code with map, filter, reduce, and comprehensions
- Quick reference: Python cheat sheet › Standard Library
Frequently asked questions
Can a ChainMap write to the dictionary that already has the key?
Not by itself: writes and deletes always go to the first dictionary. The collections documentation gives a recipe called DeepChainMap, a subclass whose __setitem__ looks for the key in each dictionary in turn and changes it there, or adds it to the first dictionary when none has it. With an empty this_run in front of defaults = {"theme": "light", "size": 12}, setting size to 16 and font to "mono" through it left this_run as {'font': 'mono'} and defaults as {'theme': 'light', 'size': 16}.
Is a ChainMap faster than merging the dictionaries?
Making one copies nothing, and the collections documentation says a ChainMap "is often much faster than creating a new dictionary and running multiple update() calls". A lookup, though, searches the dictionaries in turn until one has the key. Choose by what you need: a ChainMap for a live view of the layers, | or dict() for a snapshot.
What does | do with a ChainMap?
It returns a new ChainMap whose first dictionary is a merged copy, and leaves the original alone. ChainMap({"theme": "dark"}, {"theme": "light", "size": 12}) | {"size": 14} gave ChainMap({'theme': 'dark', 'size': 14}, {'theme': 'light', 'size': 12}), and the original's maps were still [{'theme': 'dark'}, {'theme': 'light', 'size': 12}].
If I override append() on a UserList, does extend() use it?
No. A Titles class built on UserList, whose append() title-cased each item, held ['Dune', 'emma'] after t.append("dune") and t.extend(["emma"]): UserList's extend() adds to the list in .data directly. Override every method that adds items, such as extend() and insert(), not just append().
Can a method choose what to do by its argument's type?
Yes, with functools.singledispatchmethod, which looks at the first argument after self. In a Report class whose cell() method had a version registered for int that returned f"{value:,}", r.cell(1234567) gave 1,234,567 and r.cell("tea") gave tea.
Can register() read a union annotation such as int | float?
From Python 3.11, which this site's editor runs, yes: with value: int | float as the annotation, describe(2) and describe(2.5) both found that version, and describe("2") got the fallback. Python 3.10 refuses it with TypeError: Invalid annotation for 'value'. int | float is not a class. On 3.10, name each class in its own register() call and stack them.
Is there a read-only dictionary?
Not in collections. types.MappingProxyType(d) gives a read-only view of a dictionary, which still shows the dictionary's own later changes; assigning through the view stops with TypeError: 'mappingproxy' object does not support item assignment. Functional Programming Techniques in Python covers it.
Related lessons
- itertools
- Counter & defaultdict
- deque & Queues
- namedtuple & NamedTuple
- functools: lru_cache, partial, reduce
- Operator Overloading & Custom Behaviours
- Dictionaries
- Building Command-Line Tools with argparse & Typer
- Inheritance and Polymorphism
- Magic Methods & the Python Data Model
- Mixins, Multiple Inheritance & OOP Patterns
- Higher-Order Functions & Function Factories