Benchmarking with timeit

Published by Brayan K

The timeit module measures how long small snippets of code take by running them many times, giving you reliable numbers instead of the noisy guesses a single timer produces.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Measure a defined operation after proving its result

A microbenchmark answers a narrow question: how long a particular operation takes under a particular setup. It does not prove that a whole application is fast or that one implementation always wins. Establish correctness and the work being counted before collecting timing data.

from timeit import Timer

calls = []
def operation():
    calls.append("called")
    return sum((1, 2, 3))

timer = Timer(operation)
elapsed = timer.timeit(number=4)
print(len(calls))
print(operation() == 6)
print(type(elapsed).__name__)

# ✅ Expected output:
# 4
# True
# float

timeit returns total elapsed seconds for the requested number of executions. The example keeps that measurement in elapsed and prints only deterministic facts about the experiment. Actual durations depend on the interpreter, machine and current load; a fixed published duration would not be a result you should expect on your own system.

The extra correctness call happens after timing, so the displayed count is taken before it. Instrumentation such as appending to calls itself costs time. This first experiment demonstrates the execution contract, not the performance of sum.

number and repeat are two different loops

from timeit import Timer

counts = {"setup": 0, "operation": 0}
state = []

def prepare():
    counts["setup"] += 1
    state[:] = [1, 2]

def operation():
    counts["operation"] += 1
    return state.pop()

samples = Timer(operation, setup=prepare).repeat(repeat=3, number=2)
print(counts)
print(len(samples))
print(state)

# ✅ Expected output:
# {'setup': 3, 'operation': 6}
# 3
# []

number controls how many executions occur inside one timing sample. repeat controls how many timing samples are collected. Setup runs once before each sample, outside that sample’s timed loop. Here the setup creates exactly enough mutable state for two operations, then another repetition resets it.

Calling timeit with no explicit number uses a large fixed default; it does not automatically calibrate a loop count. Timer.autorange can choose a count by trying increasing workloads. Calibration itself runs the code, so it can mutate state or trigger side effects before the final comparison. Choose a safe workload and account for that work.

Setup is not a reset before every iteration

from timeit import Timer

values = [1, 2]
Timer(values.pop).timeit(number=3)

# ✅ Expected error:
# IndexError: pop from empty list

The first two calls consume the list; the third is a different operation on empty data. A setup that runs once per sample would not make every iteration receive a fresh list. If the operation requires a fresh copy, decide whether copying belongs inside the measured work, or design a repeatable operation that does not consume its input.

from timeit import Timer

original = (1, 2, 3)
results = []
def operation():
    working = list(original)
    results.append(working.pop())

samples = Timer(operation).repeat(repeat=2, number=3)
print(results)
print(original)
print(len(samples))

# ✅ Expected output:
# [3, 3, 3, 3, 3, 3]
# (1, 2, 3)
# 2

This measured operation includes list construction, popping and result recording. It would be misleading to label the result “pop alone.” A fair comparison must measure equivalent work on equivalent inputs, including any setup cost intentionally placed in the timed statement.

A string statement does not inherit your caller’s local scope

from timeit import timeit

values = [1, 2, 3]
timeit("sum(values)", number=1)

# ✅ Expected error:
# NameError: name 'values' is not defined
from timeit import timeit

values = (1, 2, 3)
checks = []
namespace = {"values": values, "checks": checks}
elapsed = timeit("checks.append(sum(values))", globals=namespace, number=3)
print(checks)
print(type(elapsed).__name__)

# ✅ Expected output:
# [6, 6, 6]
# float

The globals argument supplies names used by the generated timing code. A callable is often clearer when you already have a function, but it includes the callable invocation overhead. Keep the form consistent across alternatives when that overhead matters. Scope: global, nonlocal & LEGB owns name resolution; this lesson makes the benchmark’s namespace explicit.

Do not interpolate untrusted input into a timed source string. A Timer executes Python code. An explicit callable or a fixed statement with a data namespace avoids turning a benchmark option into arbitrary code evaluation.

Understand the timer and garbage-collection choices

import gc
from timeit import Timer

before = gc.isenabled()
observed = []
Timer(lambda: observed.append(gc.isenabled())).timeit(number=2)
print(observed)
print(gc.isenabled() == before)

# ✅ Expected output:
# [False, False]
# True

By default timeit temporarily disables cyclic garbage collection during timing and restores the prior enabled state afterward. This can reduce noise, but it also means the experiment may omit a cost that matters for an allocation-heavy workload. If GC is part of the question, enable it deliberately in the setup and restore surrounding state explicitly when necessary.

The default timer is a high-resolution elapsed-time clock. Elapsed time can include waiting and scheduling effects; it is not automatically CPU time. Keep the timer choice and GC policy the same when comparing results. Do not claim that a browser’s Python runtime and a native installation have interchangeable performance.

Normalize by the work count and keep the samples

from timeit import Timer

values = tuple(range(20))
def by_loop():
    total = 0
    for value in values:
        total += value * value
    return total

def by_generator():
    return sum(value * value for value in values)

assert by_loop() == by_generator()
number = 50
loop_samples = Timer(by_loop).repeat(repeat=3, number=number)
generator_samples = Timer(by_generator).repeat(repeat=3, number=number)
loop_per_call = [sample / number for sample in loop_samples]
generator_per_call = [sample / number for sample in generator_samples]
print(by_loop() == by_generator())
print(len(loop_per_call), len(generator_per_call))
print(all(isinstance(value, float) for value in loop_per_call))

# ✅ Expected output:
# True
# 3 3
# True

Each sample is a total for number operations, so divide by that count to obtain a per-operation value. Retain the full sample list. A minimum can approximate the least-interrupted observed run for a small deterministic snippet; it is not a confidence interval, a universal speed ranking or a latency guarantee. Investigate variability instead of hiding it.

The code deliberately prints no “winner.” Even a yes/no timing comparison can change across runs and machines. Use your own retained samples to assess the specific environment, repeat under representative inputs, and report the operation, interpreter, loop count and relevant settings alongside any real benchmark result.

Complete the experimental counts

Collect three samples, with four executions in each sample. The operation records its calls so you can verify the loop contract independently from elapsed timing. Fill number and repeat using the literal hints.

from timeit import Timer

calls = []
def operation():
    calls.append(1)

samples = Timer(operation).repeat(number=___, repeat=___)
print(len(calls))
print(len(samples))

# 👉 replace ___ with 4
# 👉 replace ___ with 3

# ✅ Expected output:
# 12
# 3

Compare two correct transformations with a stated workload

Write a loop-based and a comprehension-based function that convert the same tuple of integers into a list of their doubles. Check both results against the same expected list before timing. Collect three samples of 20 executions per function, retain per-call measurements, and print only the correctness result and sample counts. Neither implementation may mutate the input.

from timeit import Timer

values = (1, 3, 5, 7)
def loop_version():
    result = []
    for value in values:
        result.append(value * 2)
    return result

def comprehension_version():
    return [value * 2 for value in values]

expected = [2, 6, 10, 14]
assert loop_version() == expected
assert comprehension_version() == expected
number = 20
measurements = {}
for name, operation in [("loop", loop_version),
                        ("comprehension", comprehension_version)]:
    totals = Timer(operation).repeat(repeat=3, number=number)
    measurements[name] = [total / number for total in totals]
print(loop_version() == comprehension_version() == expected)
print({name: len(samples) for name, samples in measurements.items()})
print(values)

# ✅ Expected output:
# True
# {'loop': 3, 'comprehension': 3}
# (1, 3, 5, 7)

Practice quiz

What does this program print? from timeit import Timer calls = [] Timer(lambda: calls.append(1)).repeat(repeat=2, number=3) print(len(calls))

  • 6
  • 2
  • 3
  • 5

Answer: 6. Two samples each execute the operation three times.

What does timeit(number=100) return?

  • A guaranteed duration per operation
  • Total seconds for the 100 executions
  • A ranking against another algorithm
  • The operation’s return value

Answer: Total seconds for the 100 executions. The total must be divided by the execution count for a per-call figure.

What does this program print? from timeit import Timer counts = [] Timer(lambda: None, setup=lambda: counts.append("setup")).repeat(repeat=3, number=4) print(len(counts))

  • 12
  • 4
  • 3
  • 1

Answer: 3. Setup runs once before each timing sample, not before every loop iteration.

A timed operation pops from one list repeatedly. What is the main validity concern?

  • The list automatically resets
  • All pops have identical input forever
  • timeit refuses mutable values
  • Later iterations can see different or exhausted state

Answer: Later iterations can see different or exhausted state. Mutation can change the workload or cause later iterations to fail.

What does this program print? from timeit import Timer values = (1, 2) samples = Timer(lambda: sum(values)).repeat(repeat=2, number=3) print(len(samples))

  • 2
  • 3
  • 6
  • 1

Answer: 2. repeat determines the number of returned timing samples.

Why can a string statement fail to find a variable from the surrounding function?

  • Strings cannot reference any variables
  • The timing namespace must receive the required names explicitly
  • timeit deletes every variable
  • number must equal the variable value

Answer: The timing namespace must receive the required names explicitly. Use an explicit globals mapping or a callable with the intended bindings.

What does this program print? import gc from timeit import Timer seen = [] Timer(lambda: seen.append(gc.isenabled())).timeit(number=1) print(seen)

  • [True]
  • []
  • [False]
  • None

Answer: [False]. The default timing loop temporarily disables cyclic GC.

What does one smaller minimum timing establish?

  • Universal superiority on all machines
  • A production latency guarantee
  • That correctness checks are unnecessary
  • A lower observed timing under that experiment’s conditions

Answer: A lower observed timing under that experiment’s conditions. Benchmark interpretation is bounded by the measured workload and environment.

Continue this course

Related lessons