Profiling & Optimising Performance
Published by Brayan K
High-performance Python isn't about writing "faster code" — it's about finding bottlenecks and eliminating them with scientific precision. You cannot optimise what you do not measure.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Count the calls before you guess
This lesson finds where a program spends its time, with cProfile, and fixes it by making the program do less work. Three neighbours have lessons of their own: timing a small piece of code in detail is Benchmarking with timeit, measuring memory is Memory Management & Garbage Collection Internals, and caching results is functools: lru_cache, partial, reduce. Every example here runs in the browser too.
A profiler watches a program run and counts every function call. Reading the code, the function below that cleans a word looks cheap. The count shows how often it really runs: the nested loops call it twice for every pair of words, 870 times for 30 words.
import cProfile
import pstats
def clean(word):
return word.strip().lower()
def count_pairs(words):
pairs = 0
for i in range(len(words)):
for j in range(i + 1, len(words)):
if clean(words[i]) == clean(words[j]):
pairs += 1
return pairs
words = ["Ada", "bo", "ADA ", "Cy", "Bo", "ada"] * 5
with cProfile.Profile() as profiler:
print(count_pairs(words))
calls = pstats.Stats(profiler).get_stats_profile().func_profiles
print(calls["clean"].ncalls)
print(calls["count_pairs"].ncalls)
# ✅ Expected output:
# 160
# 870
# 1with cProfile.Profile() as profiler: profiles everything inside the block, and turns the profiler off when the block ends, even if the code inside fails. pstats.Stats(profiler) reads what it recorded, and get_stats_profile().func_profiles is a dictionary from each function's name to its row. A row's ncalls is the number of calls, as a string. The fix, cleaning each word once before the loops, is yours to write in the practice further down.
The usual way to read a profile is the whole table, printed by:
pstats.Stats(profiler).sort_stats("cumulative").print_stats(5)Add that line to the example and run it. Above the table is a line saying how many function calls there were in how many seconds. Then come the columns ncalls, tottime (the time spent in the function itself), percall, cumtime (the time including the functions it calls), percall again, and the function's file, line and name. The times change on every run, which is why the examples here print counts and names instead. print_stats(5) prints only the first five rows. To profile a whole script from a terminal, run:
python3 -m cProfile -s cumulative myscript.pyTimes change from run to run; use them to find where
The times in a profile move every time you run it, and profiling slows the program down: each call does extra work for the profiler. So don't read a profile's times as the program's real speed. Read them to find where the time goes. Sorted by tottime, the function doing the work comes first:
import cProfile
import pstats
def load():
return list(range(1000))
def analyse(numbers):
total = 0
for n in numbers:
for m in numbers:
total += n * m
return total
def main():
return analyse(load())
with cProfile.Profile() as profiler:
main()
ranked = pstats.Stats(profiler).sort_stats("tottime").get_stats_profile()
print(next(iter(ranked.func_profiles)))
# ✅ Expected output:
# analyseanalyse() led by a factor of hundreds in every run here, so its name is safe to print. Sorted by "cumulative", main() comes first instead, because its cumtime includes the analyse() call inside it. That's how to read the two columns: cumtime says which call leads to the slow part, tottime says which function is the slow part.
sort_stats() takes names such as "cumulative", "tottime", "calls" and "name". A name it doesn't know stops with a KeyError:
import cProfile
import pstats
with cProfile.Profile() as profiler:
sorted([3, 1, 2])
pstats.Stats(profiler).sort_stats("slowest")
# ✅ Expected error:
# KeyError: 'slowest'ncalls shows recursion as two numbers
For a function that calls itself, ncalls holds two numbers: 3/1 means 3 calls in all, 1 of them from outside the function.
import cProfile
import pstats
def flatten(items):
flat = []
for item in items:
if isinstance(item, list):
flat.extend(flatten(item))
else:
flat.append(item)
return flat
with cProfile.Profile() as profiler:
print(flatten([1, [2, [3, 4]], 5]))
print(pstats.Stats(profiler).get_stats_profile().func_profiles["flatten"].ncalls)
# ✅ Expected output:
# [1, 2, 3, 4, 5]
# 3/1runcall() profiles a single call and hands back its result, without a with block:
import cProfile
import pstats
def flatten(items):
flat = []
for item in items:
if isinstance(item, list):
flat.extend(flatten(item))
else:
flat.append(item)
return flat
profiler = cProfile.Profile()
result = profiler.runcall(flatten, [[1, 2], [3, [4]]])
print(result)
print(pstats.Stats(profiler).get_stats_profile().func_profiles["flatten"].ncalls)
# ✅ Expected output:
# [1, 2, 3, 4]
# 4/1timeit runs a statement string in a namespace of its own
timeit.timeit() runs a piece of code many times and returns the total time it took. Given a string, it runs that string in a namespace of its own, where your functions don't exist:
import timeit
def total():
return sum(range(100))
print(timeit.timeit("total()", number=1000))
# ✅ Expected error:
# NameError: name 'total' is not definedCalling the function instead hands timeit its result, a number, to run:
import timeit
def total():
return sum(range(100))
print(timeit.timeit(total(), number=1000))
# ✅ Expected error:
# ValueError: stmt is neither a string nor callablePass the function itself, or give the string your names with globals=globals(). The time is a float, and it changes from run to run, so the example only shows its type:
import timeit
def total():
return sum(range(100))
seconds = timeit.timeit(total, number=1000)
print(type(seconds).__name__)
seconds = timeit.timeit("total()", globals=globals(), number=1000)
print(seconds > 0)
# ✅ Expected output:
# float
# TruecProfile.run("main()") runs its string the same way, in the __main__ module's namespace. With python3 on a computer that works, but this site's editor runs your code in a namespace of its own, so there it stops with a NameError. That's why this lesson profiles with a with block.
Optimising means doing less work
Once the profile shows where the time goes, the big wins come from doing less there, not from making each line a little quicker. Counting the work makes the difference exact. Checking 1,000 items for a duplicate by comparing every pair takes 499,500 comparisons; remembering what you've seen in a set, as Sets taught, takes 1,000:
def has_duplicates_slow(items):
checks = 0
for i in range(len(items)):
for j in range(i + 1, len(items)):
checks += 1
if items[i] == items[j]:
return True, checks
return False, checks
def has_duplicates_fast(items):
checks = 0
seen = set()
for item in items:
checks += 1
if item in seen:
return True, checks
seen.add(item)
return False, checks
ids = list(range(1000))
print(has_duplicates_slow(ids))
print(has_duplicates_fast(ids))
# ✅ Expected output:
# (False, 499500)
# (False, 1000)Caching is the third common fix: work out a result once and keep it. Advanced Functions & Parameters Masterclass showed lru_cache cutting fib(30) from 2,692,537 calls to 31, and functools: lru_cache, partial, reduce covers it.
One timing proves little: compare in one run, repeat, trust big gaps
To compare two versions, time both in the same run, repeat each a few times, and take the smallest time: timeit.repeat()'s own documentation says the minimum is "probably the only number you should be interested in". Even then, trust only a big gap. Looking a number up in a list of 100,000 took thousands of times as long as in a set, in every run on every Python here and in the browser, so this result holds:
import timeit
ids = list(range(100_000))
id_set = set(ids)
list_time = min(timeit.repeat(lambda: 99_999 in ids, number=20, repeat=3))
set_time = min(timeit.repeat(lambda: 99_999 in id_set, number=20, repeat=3))
print(set_time < list_time)
# ✅ Expected output:
# TrueSmaller differences flip. sum() against a loop, a comprehension against append(), and a generator against a list all came out the other way round in some runs, or on some Python versions, so which one wins depends on the version and the data. Measure your own case, as Benchmarking with timeit shows. And cProfile records calls and times only, not memory: that's tracemalloc, in Memory Management & Garbage Collection Internals.
Practice: profile the fix
Clean each word once, before the loops, and profile the result. Fill the three blanks: the function that cleans a word, the class that profiles the block, and the row's count of calls. The 👉 lines under the code hold the answers, one per blank, from the top. The pair count stays 160, and clean() now runs 30 times instead of 870.
import cProfile
import pstats
def clean(word):
return word.strip().lower()
def count_pairs(words):
cleaned = [___(w) for w in words] # clean each word once, before the loops
pairs = 0
for i in range(len(cleaned)):
for j in range(i + 1, len(cleaned)):
if cleaned[i] == cleaned[j]:
pairs += 1
return pairs
words = ["Ada", "bo", "ADA ", "Cy", "Bo", "ada"] * 5
with cProfile.___() as profiler:
print(count_pairs(words))
calls = pstats.Stats(profiler).get_stats_profile().func_profiles
print(calls["clean"].___)
# 👉 replace ___ with clean
# 👉 replace ___ with Profile
# 👉 replace ___ with ncalls
# ✅ Expected output:
# 160
# 30Ask for calls["Clean"] with a capital letter and the program stops with KeyError: 'Clean': the keys are the functions' names exactly as written.
On your own: parse each line once
report(lines) parses comma-separated lines, keeps the ones with three fields, and adds up their third fields. Write it twice:
- report_slow() calls parse() in the filter and again for the row it keeps;
- report_fast() parses each line once and filters the parsed rows.
Use parse(line), which returns line.split(","), valid(fields), which checks for three fields, and total(rows), which adds up the third fields as numbers. Run both on ["a,b,1", "c,d,2", "bad", "e,f,3"] * 25, each under its own profiler, and print each function's name, its total and how many times parse() and valid() ran.
import cProfile
import pstats
# your code hereimport cProfile
import pstats
def parse(line):
return line.split(",")
def valid(fields):
return len(fields) == 3
def total(rows):
return sum(int(row[2]) for row in rows)
def report_slow(lines):
rows = [parse(line) for line in lines if valid(parse(line))]
return total(rows)
def report_fast(lines):
parsed = [parse(line) for line in lines]
rows = [fields for fields in parsed if valid(fields)]
return total(rows)
lines = ["a,b,1", "c,d,2", "bad", "e,f,3"] * 25
for report in (report_slow, report_fast):
with cProfile.Profile() as profiler:
result = report(lines)
calls = pstats.Stats(profiler).get_stats_profile().func_profiles
print(report.__name__, result, calls["parse"].ncalls, calls["valid"].ncalls)
# ✅ Expected output:
# report_slow 150 175 100
# report_fast 150 100 100The totals agree, and the slow version parses 75 more times: once more for each of the 75 lines it keeps. report.__name__ is each function's name, as in Lambda Functions.
Practice quiz
clean(word) returns word.lower(). count_pairs(words) compares every pair of words with clean(words[i]) == clean(words[j]). After profiling count_pairs(["a", "B", "b", "A"]), what is clean's ncalls?
- 4
- 6
- 16
- 12
Answer: 12. Four words make six pairs, and each pair calls clean() twice.
total() returns sum(range(100)). What does print(timeit.timeit("total()", number=1000)) do?
- It prints a float
- It stops with NameError: name 'total' is not defined
- It prints None
- It stops with ValueError: stmt is neither a string nor callable
Answer: It stops with NameError: name 'total' is not defined. The string runs in timeit's own namespace, where total doesn't exist. Pass globals=globals(), or the function itself.
flatten(items) calls itself on every item that is a list. After profiling flatten([[1], [2, [3]]]), what is flatten's ncalls?
- 4/1
- 1
- 3/1
- 4
Answer: 4/1. flatten runs 4 times: once from outside, and 3 times from inside itself, for [1], [2, [3]] and [3].
has_duplicates(items) compares every pair of items, counting each comparison, and returns (found, checks) as soon as two match. What does has_duplicates([1, 2, 3, 4, 5]) return?
- (False, 5)
- (False, 25)
- (False, 10)
- (False, 20)
Answer: (False, 10). Five items make 10 pairs, and none of them match.
total() returns sum(range(100)). What does print(timeit.timeit(total(), number=1000)) do?
- It prints a float
- It stops with ValueError: stmt is neither a string nor callable
- It stops with NameError: name 'total' is not defined
- It prints 4950
Answer: It stops with ValueError: stmt is neither a string nor callable. total() runs first and hands timeit the number 4950, which is neither a string nor a function.
count_pairs() now cleans each word once, into a list, before the loops. Profiled on ["a", "B", "b", "A"], it prints its count of matching pairs, then clean's ncalls is printed. What is shown?
- 2, then 12
- 4, then 2
- 2, then 1
- 2, then 4
Answer: 2, then 4. Two pairs match (a with A, and B with b), and clean() runs once per word.
fib(5) is profiled, then the program prints pstats.Stats(profiler).get_stats_profile().func_profiles["Fib"].ncalls. What happens?
- It prints 15
- It prints 0
- It stops with KeyError: 'Fib'
- It stops with NameError: name 'Fib' is not defined
Answer: It stops with KeyError: 'Fib'. func_profiles is a dictionary keyed by the functions' names as written: fib, not Fib.
The same has_duplicates() is given [7, 7, 1, 2]. What does it return?
- (True, 1)
- (True, 6)
- (False, 6)
- (True, 2)
Answer: (True, 1). The very first pair matches, so it returns after one check.
Continue this course
- Previous: Parallelism with concurrent.futures
- Next: Memory Management & Garbage Collection Internals — How Python allocates, tracks, and frees memory under the hood
- Quick reference: Python cheat sheet › Standard Library
- From the blog: Python Decorators: A Practical Guide
Frequently asked questions
Why does cProfile.run("main()") fail in this site's editor?
cProfile.run() runs the string in the __main__ module's namespace, and the editor runs your code in a namespace of its own. So it prints a table without your function in it, then stops with NameError: name 'main' is not defined. With python3 on a computer it works. In the editor, use with cProfile.Profile(), as this lesson does, or cProfile.runctx("main()", globals(), locals()).
How do I profile a whole script?
From a terminal: python3 -m cProfile -s cumulative myscript.py runs the script under the profiler and prints the table sorted by cumulative time.
Why does my program run slower under the profiler?
The profiler does extra work on every call. Here, a recursive function ran 2 to 9 times slower under cProfile on Python 3.10–3.14, and 7 to 11 times slower in this site's Run button. Compare profiled runs with profiled runs, and time the real program without the profiler.
Why does a second profiler fail on Python 3.12?
From 3.12, only one profiler can be switched on at a time, so starting a second one inside the first stops with a ValueError, whose wording changed again in 3.14. Python 3.10 and 3.11, and this site's Run button, allow two.
What's the difference between profile and cProfile?
They work the same way and count the same calls. profile is written in Python and slows your program down more; cProfile is written in C, so it's the one to use.
Can I see which line is slow?
cProfile works one function at a time. Timing each line needs a separate package, line_profiler, which isn't part of Python or of this site's Run button.
Should I time code with time.time() or time.perf_counter()?
time.perf_counter(): its docstring calls it the "Performance counter for benchmarking". time.time() is the time of day, meant for timestamps, and in this site's Run button it moves in steps of a millisecond.
Why does a generator expression appear in the profile?
Python runs a generator expression as a small function of its own, and the profiler counts every time it resumes. Profiling sum(n * n for n in range(10)) shows a <genexpr> row with 11 calls for its 10 values.