Profiling & Optimising Performance

Published by Brayan K

High-performance Python isn't about writing "faster code" — it's about finding bottlenecks and eliminating them with scientific precision. You cannot optimise what you do not measure.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Count the calls before you guess

This lesson finds where a program spends its time, with cProfile, and fixes it by making the program do less work. Three neighbours have lessons of their own: timing a small piece of code in detail is Benchmarking with timeit, measuring memory is Memory Management & Garbage Collection Internals, and caching results is functools: lru_cache, partial, reduce. Every example here runs in the browser too.

A profiler watches a program run and counts every function call. Reading the code, the function below that cleans a word looks cheap. The count shows how often it really runs: the nested loops call it twice for every pair of words, 870 times for 30 words.

import cProfile
import pstats

def clean(word):
    return word.strip().lower()

def count_pairs(words):
    pairs = 0
    for i in range(len(words)):
        for j in range(i + 1, len(words)):
            if clean(words[i]) == clean(words[j]):
                pairs += 1
    return pairs

words = ["Ada", "bo", "ADA ", "Cy", "Bo", "ada"] * 5

with cProfile.Profile() as profiler:
    print(count_pairs(words))

calls = pstats.Stats(profiler).get_stats_profile().func_profiles
print(calls["clean"].ncalls)
print(calls["count_pairs"].ncalls)

# ✅ Expected output:
# 160
# 870
# 1

with cProfile.Profile() as profiler: profiles everything inside the block, and turns the profiler off when the block ends, even if the code inside fails. pstats.Stats(profiler) reads what it recorded, and get_stats_profile().func_profiles is a dictionary from each function's name to its row. A row's ncalls is the number of calls, as a string. The fix, cleaning each word once before the loops, is yours to write in the practice further down.

The usual way to read a profile is the whole table, printed by:

pstats.Stats(profiler).sort_stats("cumulative").print_stats(5)

Add that line to the example and run it. Above the table is a line saying how many function calls there were in how many seconds. Then come the columns ncalls, tottime (the time spent in the function itself), percall, cumtime (the time including the functions it calls), percall again, and the function's file, line and name. The times change on every run, which is why the examples here print counts and names instead. print_stats(5) prints only the first five rows. To profile a whole script from a terminal, run:

python3 -m cProfile -s cumulative myscript.py

Times change from run to run; use them to find where

The times in a profile move every time you run it, and profiling slows the program down: each call does extra work for the profiler. So don't read a profile's times as the program's real speed. Read them to find where the time goes. Sorted by tottime, the function doing the work comes first:

import cProfile
import pstats

def load():
    return list(range(1000))

def analyse(numbers):
    total = 0
    for n in numbers:
        for m in numbers:
            total += n * m
    return total

def main():
    return analyse(load())

with cProfile.Profile() as profiler:
    main()

ranked = pstats.Stats(profiler).sort_stats("tottime").get_stats_profile()
print(next(iter(ranked.func_profiles)))

# ✅ Expected output:
# analyse

analyse() led by a factor of hundreds in every run here, so its name is safe to print. Sorted by "cumulative", main() comes first instead, because its cumtime includes the analyse() call inside it. That's how to read the two columns: cumtime says which call leads to the slow part, tottime says which function is the slow part.

sort_stats() takes names such as "cumulative", "tottime", "calls" and "name". A name it doesn't know stops with a KeyError:

import cProfile
import pstats

with cProfile.Profile() as profiler:
    sorted([3, 1, 2])

pstats.Stats(profiler).sort_stats("slowest")

# ✅ Expected error:
# KeyError: 'slowest'

ncalls shows recursion as two numbers

For a function that calls itself, ncalls holds two numbers: 3/1 means 3 calls in all, 1 of them from outside the function.

import cProfile
import pstats

def flatten(items):
    flat = []
    for item in items:
        if isinstance(item, list):
            flat.extend(flatten(item))
        else:
            flat.append(item)
    return flat

with cProfile.Profile() as profiler:
    print(flatten([1, [2, [3, 4]], 5]))

print(pstats.Stats(profiler).get_stats_profile().func_profiles["flatten"].ncalls)

# ✅ Expected output:
# [1, 2, 3, 4, 5]
# 3/1

runcall() profiles a single call and hands back its result, without a with block:

import cProfile
import pstats

def flatten(items):
    flat = []
    for item in items:
        if isinstance(item, list):
            flat.extend(flatten(item))
        else:
            flat.append(item)
    return flat

profiler = cProfile.Profile()
result = profiler.runcall(flatten, [[1, 2], [3, [4]]])
print(result)
print(pstats.Stats(profiler).get_stats_profile().func_profiles["flatten"].ncalls)

# ✅ Expected output:
# [1, 2, 3, 4]
# 4/1

timeit runs a statement string in a namespace of its own

timeit.timeit() runs a piece of code many times and returns the total time it took. Given a string, it runs that string in a namespace of its own, where your functions don't exist:

import timeit

def total():
    return sum(range(100))

print(timeit.timeit("total()", number=1000))

# ✅ Expected error:
# NameError: name 'total' is not defined

Calling the function instead hands timeit its result, a number, to run:

import timeit

def total():
    return sum(range(100))

print(timeit.timeit(total(), number=1000))

# ✅ Expected error:
# ValueError: stmt is neither a string nor callable

Pass the function itself, or give the string your names with globals=globals(). The time is a float, and it changes from run to run, so the example only shows its type:

import timeit

def total():
    return sum(range(100))

seconds = timeit.timeit(total, number=1000)
print(type(seconds).__name__)
seconds = timeit.timeit("total()", globals=globals(), number=1000)
print(seconds > 0)

# ✅ Expected output:
# float
# True

cProfile.run("main()") runs its string the same way, in the __main__ module's namespace. With python3 on a computer that works, but this site's editor runs your code in a namespace of its own, so there it stops with a NameError. That's why this lesson profiles with a with block.

Optimising means doing less work

Once the profile shows where the time goes, the big wins come from doing less there, not from making each line a little quicker. Counting the work makes the difference exact. Checking 1,000 items for a duplicate by comparing every pair takes 499,500 comparisons; remembering what you've seen in a set, as Sets taught, takes 1,000:

def has_duplicates_slow(items):
    checks = 0
    for i in range(len(items)):
        for j in range(i + 1, len(items)):
            checks += 1
            if items[i] == items[j]:
                return True, checks
    return False, checks

def has_duplicates_fast(items):
    checks = 0
    seen = set()
    for item in items:
        checks += 1
        if item in seen:
            return True, checks
        seen.add(item)
    return False, checks

ids = list(range(1000))
print(has_duplicates_slow(ids))
print(has_duplicates_fast(ids))

# ✅ Expected output:
# (False, 499500)
# (False, 1000)

Caching is the third common fix: work out a result once and keep it. Advanced Functions & Parameters Masterclass showed lru_cache cutting fib(30) from 2,692,537 calls to 31, and functools: lru_cache, partial, reduce covers it.

One timing proves little: compare in one run, repeat, trust big gaps

To compare two versions, time both in the same run, repeat each a few times, and take the smallest time: timeit.repeat()'s own documentation says the minimum is "probably the only number you should be interested in". Even then, trust only a big gap. Looking a number up in a list of 100,000 took thousands of times as long as in a set, in every run on every Python here and in the browser, so this result holds:

import timeit

ids = list(range(100_000))
id_set = set(ids)

list_time = min(timeit.repeat(lambda: 99_999 in ids, number=20, repeat=3))
set_time = min(timeit.repeat(lambda: 99_999 in id_set, number=20, repeat=3))
print(set_time < list_time)

# ✅ Expected output:
# True

Smaller differences flip. sum() against a loop, a comprehension against append(), and a generator against a list all came out the other way round in some runs, or on some Python versions, so which one wins depends on the version and the data. Measure your own case, as Benchmarking with timeit shows. And cProfile records calls and times only, not memory: that's tracemalloc, in Memory Management & Garbage Collection Internals.

Practice: profile the fix

Clean each word once, before the loops, and profile the result. Fill the three blanks: the function that cleans a word, the class that profiles the block, and the row's count of calls. The 👉 lines under the code hold the answers, one per blank, from the top. The pair count stays 160, and clean() now runs 30 times instead of 870.

import cProfile
import pstats

def clean(word):
    return word.strip().lower()

def count_pairs(words):
    cleaned = [___(w) for w in words]   # clean each word once, before the loops
    pairs = 0
    for i in range(len(cleaned)):
        for j in range(i + 1, len(cleaned)):
            if cleaned[i] == cleaned[j]:
                pairs += 1
    return pairs

words = ["Ada", "bo", "ADA ", "Cy", "Bo", "ada"] * 5

with cProfile.___() as profiler:
    print(count_pairs(words))

calls = pstats.Stats(profiler).get_stats_profile().func_profiles
print(calls["clean"].___)

# 👉 replace ___ with clean
# 👉 replace ___ with Profile
# 👉 replace ___ with ncalls

# ✅ Expected output:
# 160
# 30

Ask for calls["Clean"] with a capital letter and the program stops with KeyError: 'Clean': the keys are the functions' names exactly as written.

On your own: parse each line once

report(lines) parses comma-separated lines, keeps the ones with three fields, and adds up their third fields. Write it twice:

Use parse(line), which returns line.split(","), valid(fields), which checks for three fields, and total(rows), which adds up the third fields as numbers. Run both on ["a,b,1", "c,d,2", "bad", "e,f,3"] * 25, each under its own profiler, and print each function's name, its total and how many times parse() and valid() ran.

import cProfile
import pstats

# your code here
import cProfile
import pstats

def parse(line):
    return line.split(",")

def valid(fields):
    return len(fields) == 3

def total(rows):
    return sum(int(row[2]) for row in rows)

def report_slow(lines):
    rows = [parse(line) for line in lines if valid(parse(line))]
    return total(rows)

def report_fast(lines):
    parsed = [parse(line) for line in lines]
    rows = [fields for fields in parsed if valid(fields)]
    return total(rows)

lines = ["a,b,1", "c,d,2", "bad", "e,f,3"] * 25

for report in (report_slow, report_fast):
    with cProfile.Profile() as profiler:
        result = report(lines)
    calls = pstats.Stats(profiler).get_stats_profile().func_profiles
    print(report.__name__, result, calls["parse"].ncalls, calls["valid"].ncalls)

# ✅ Expected output:
# report_slow 150 175 100
# report_fast 150 100 100

The totals agree, and the slow version parses 75 more times: once more for each of the 75 lines it keeps. report.__name__ is each function's name, as in Lambda Functions.

Practice quiz

clean(word) returns word.lower(). count_pairs(words) compares every pair of words with clean(words[i]) == clean(words[j]). After profiling count_pairs(["a", "B", "b", "A"]), what is clean's ncalls?

  • 4
  • 6
  • 16
  • 12

Answer: 12. Four words make six pairs, and each pair calls clean() twice.

total() returns sum(range(100)). What does print(timeit.timeit("total()", number=1000)) do?

  • It prints a float
  • It stops with NameError: name 'total' is not defined
  • It prints None
  • It stops with ValueError: stmt is neither a string nor callable

Answer: It stops with NameError: name 'total' is not defined. The string runs in timeit's own namespace, where total doesn't exist. Pass globals=globals(), or the function itself.

flatten(items) calls itself on every item that is a list. After profiling flatten([[1], [2, [3]]]), what is flatten's ncalls?

  • 4/1
  • 1
  • 3/1
  • 4

Answer: 4/1. flatten runs 4 times: once from outside, and 3 times from inside itself, for [1], [2, [3]] and [3].

has_duplicates(items) compares every pair of items, counting each comparison, and returns (found, checks) as soon as two match. What does has_duplicates([1, 2, 3, 4, 5]) return?

  • (False, 5)
  • (False, 25)
  • (False, 10)
  • (False, 20)

Answer: (False, 10). Five items make 10 pairs, and none of them match.

total() returns sum(range(100)). What does print(timeit.timeit(total(), number=1000)) do?

  • It prints a float
  • It stops with ValueError: stmt is neither a string nor callable
  • It stops with NameError: name 'total' is not defined
  • It prints 4950

Answer: It stops with ValueError: stmt is neither a string nor callable. total() runs first and hands timeit the number 4950, which is neither a string nor a function.

count_pairs() now cleans each word once, into a list, before the loops. Profiled on ["a", "B", "b", "A"], it prints its count of matching pairs, then clean's ncalls is printed. What is shown?

  • 2, then 12
  • 4, then 2
  • 2, then 1
  • 2, then 4

Answer: 2, then 4. Two pairs match (a with A, and B with b), and clean() runs once per word.

fib(5) is profiled, then the program prints pstats.Stats(profiler).get_stats_profile().func_profiles["Fib"].ncalls. What happens?

  • It prints 15
  • It prints 0
  • It stops with KeyError: 'Fib'
  • It stops with NameError: name 'Fib' is not defined

Answer: It stops with KeyError: 'Fib'. func_profiles is a dictionary keyed by the functions' names as written: fib, not Fib.

The same has_duplicates() is given [7, 7, 1, 2]. What does it return?

  • (True, 1)
  • (True, 6)
  • (False, 6)
  • (True, 2)

Answer: (True, 1). The very first pair matches, so it returns after one check.

Continue this course

Frequently asked questions

Why does cProfile.run("main()") fail in this site's editor?

cProfile.run() runs the string in the __main__ module's namespace, and the editor runs your code in a namespace of its own. So it prints a table without your function in it, then stops with NameError: name 'main' is not defined. With python3 on a computer it works. In the editor, use with cProfile.Profile(), as this lesson does, or cProfile.runctx("main()", globals(), locals()).

How do I profile a whole script?

From a terminal: python3 -m cProfile -s cumulative myscript.py runs the script under the profiler and prints the table sorted by cumulative time.

Why does my program run slower under the profiler?

The profiler does extra work on every call. Here, a recursive function ran 2 to 9 times slower under cProfile on Python 3.10–3.14, and 7 to 11 times slower in this site's Run button. Compare profiled runs with profiled runs, and time the real program without the profiler.

Why does a second profiler fail on Python 3.12?

From 3.12, only one profiler can be switched on at a time, so starting a second one inside the first stops with a ValueError, whose wording changed again in 3.14. Python 3.10 and 3.11, and this site's Run button, allow two.

What's the difference between profile and cProfile?

They work the same way and count the same calls. profile is written in Python and slows your program down more; cProfile is written in C, so it's the one to use.

Can I see which line is slow?

cProfile works one function at a time. Timing each line needs a separate package, line_profiler, which isn't part of Python or of this site's Run button.

Should I time code with time.time() or time.perf_counter()?

time.perf_counter(): its docstring calls it the "Performance counter for benchmarking". time.time() is the time of day, meant for timestamps, and in this site's Run button it moves in steps of a millisecond.

Why does a generator expression appear in the profile?

Python runs a generator expression as a small function of its own, and the profiler counts every time it resumes. Profiling sum(n * n for n in range(10)) shows a <genexpr> row with 11 calls for its 10 values.

Related lessons