Concurrency: Threads vs Processes
Published by Brayan K
Master Python's concurrency models and learn when to use threads, processes, or AsyncIO for maximum performance.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Threads help a program that waits, not one that computes
This lesson is about the choice between threads, processes and asyncio, and the mistakes each one invites. Measuring which is fastest for your program is Profiling & Optimising Python Performance 's subject, and the tools in depth have lessons of their own: Parallelism with concurrent.futures, Threading & the GIL and multiprocessing: True Parallelism.
This site's Run button can't start threads or processes: the Python in your browser runs in the page's only thread. An example that starts one shows a note there instead of its output, and its ✅ panel shows what Python prints on a computer. The examples that start neither run in the browser too.
A thread is a second line of work inside the same program. Threads help when the program spends its time waiting, for files, the network or a database: while one thread waits, another runs. In the usual CPython, though, only one thread runs Python code at a time. A lock called the GIL, the global interpreter lock, is handed between them: a thread lets go of it while it waits, and the running thread is asked to let go about every 5 milliseconds:
import sys
print(sys.getswitchinterval())
# ✅ Expected output:
# 0.005So threads don't make pure-Python calculations faster. C code that lets go of the GIL while it works, such as hashing a large buffer with hashlib or compressing with zlib, can run in several threads at once. A process is a separate running Python with its own memory and its own GIL, so processes can calculate on several cores at once, but starting one takes far longer than starting a thread, and its data has to be copied over.
How much faster is it on your machine? Run this with python3 on your computer and compare the three times; they depend on the computer, so none is printed here:
import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
def count(n):
total = 0
for i in range(n):
total += i
return total
if __name__ == "__main__":
jobs = [5_000_000] * 4
start = time.perf_counter()
for n in jobs:
count(n)
print("one after another:", round(time.perf_counter() - start, 2), "s")
for Pool in (ThreadPoolExecutor, ProcessPoolExecutor):
start = time.perf_counter()
with Pool(max_workers=4) as pool:
list(pool.map(count, jobs))
print(Pool.__name__, round(time.perf_counter() - start, 2), "s")A thread runs the function you pass, with its arguments in a tuple
threading.Thread(target=f, args=(...)) makes a thread that will call f with those arguments. start() starts it, and join() waits for it to finish. Calling run() instead calls the function right here, in the main thread, which shows what the thread would do without starting one:
import threading
def greet(name):
print("hello", name)
print("in the main thread?", threading.current_thread() is threading.main_thread())
t = threading.Thread(target=greet, args=("Ada",))
t.run()
print("started?", t.is_alive())
# ✅ Expected output:
# hello Ada
# in the main thread? True
# started? Falseargs supplies positional arguments as a list or tuple. ("Ada") is just a string in brackets; a one-item tuple needs a comma, as Tuples showed. So the thread passes the string's three letters as three arguments:
import threading
def greet(name):
print("hello", name)
t = threading.Thread(target=greet, args=("Ada"))
t.run()
# ✅ Expected error:
# TypeError: greet() takes 1 positional argument but 3 were giventarget=work() with brackets calls work at once, in the main thread, and gives the thread its return value, None, so the thread itself does nothing. No error says so:
import threading
def work():
print("working; main thread?", threading.current_thread() is threading.main_thread())
t = threading.Thread(target=work())
t.start()
t.join()
print("finished")
# ✅ Expected output:
# working; main thread? True
# finishedAnd join() before start() stops with RuntimeError: cannot join thread before it is started.
A read and a write can be split by another thread
Threads share the program's memory, so two of them can change the same data. Changing it takes steps, read the balance, check it, write the new one, and another thread can run between the read and the write. Then one thread's write is lost. Whether that happens depends on timing: in measured runs on Python 3.10 to 3.14, ten threads doing counter += 1 in a tight loop never lost an update, while ten that read, called a function, then wrote lost updates almost every time. A demo that usually works proves nothing either way, so the examples here force the bad timing every time.
First the timing played by hand, with two generators from Generators & Iterators Mastery: each yield is a point where the other withdrawal gets to run.
account = {"balance": 100}
def withdraw(name, amount):
seen = account["balance"] # read
yield # the other withdrawal runs here
if seen >= amount:
account["balance"] = seen - amount # write
print(name, "took", amount)
ada = withdraw("Ada", 80)
bo = withdraw("Bo", 80)
next(ada) # Ada reads the balance
next(bo) # Bo reads it too, before Ada writes
next(ada, None) # Ada writes
next(bo, None) # Bo writes, over Ada's write
print("balance:", account["balance"])
# ✅ Expected output:
# Ada took 80
# Bo took 80
# balance: 20Both withdrawals of 80 went through from a balance of 100, and the balance went down only once. With real threads, a threading.Barrier(2) forces the same timing: each thread waits at both_have_read.wait() until the other has arrived, so both read before either writes.
import threading
account = {"balance": 100}
both_have_read = threading.Barrier(2)
def withdraw(amount):
seen = account["balance"] # read
both_have_read.wait() # wait until the other thread has read too
account["balance"] = seen - amount # write
threads = [threading.Thread(target=withdraw, args=(80,)) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
print("balance:", account["balance"])
# ✅ Expected output:
# balance: 20A Lock lets one thread at a time into a with lock: block. With the read, the check and the write inside it, the second withdrawal sees the first one's result, and is refused:
import threading
account = {"balance": 100}
paid_out = []
lock = threading.Lock()
def withdraw(amount):
with lock: # one thread at a time in here
seen = account["balance"]
if seen >= amount:
account["balance"] = seen - amount
paid_out.append(amount)
threads = [threading.Thread(target=withdraw, args=(80,)) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
print("paid out:", sum(paid_out))
print("balance:", account["balance"])
# ✅ Expected output:
# paid out: 80
# balance: 20What a Lock does, without any threads: acquire(blocking=False) takes it if it's free and says whether it did. While it's held, nobody else gets it, the same thread included. with lock: takes it and gives it back:
import threading
lock = threading.Lock()
print(lock.acquire(blocking=False))
print(lock.acquire(blocking=False))
lock.release()
print(lock.acquire(blocking=False))
lock.release()
with lock:
print("inside:", lock.locked())
print("after:", lock.locked())
# ✅ Expected output:
# True
# False
# True
# inside: True
# after: FalseGiving back a lock that nobody holds is an error:
import threading
lock = threading.Lock()
with lock:
print("counting")
lock.release()
# ✅ Expected output:
# counting
# ✅ Expected error:
# RuntimeError: release unlocked lockCollect results from a pool instead of printing from threads
Lines printed by several threads come out in a different order from run to run, and two lines can run together: two threads printing three lines each gave 15 to 19 different outputs in 50 runs per Python version. So let each thread return its result, and print once at the end.
ThreadPoolExecutor runs a function on a list of inputs in a pool of threads. Its map() gives the results back in the order of the inputs, even when later ones finish first, and the with block waits for all of them:
from concurrent.futures import ThreadPoolExecutor
def work(n):
total = 0
for i in range(n):
total += i
return n
sizes = [300_000, 200_000, 100_000, 1]
with ThreadPoolExecutor(max_workers=4) as pool:
print(list(pool.map(work, sizes)))
# ✅ Expected output:
# [300000, 200000, 100000, 1]Unlike the built-in map() from Lambda Functions, which does nothing until you ask for values, the pool's map() submits all these jobs without waiting for results to be read. At most max_workers jobs run at once:
from concurrent.futures import ThreadPoolExecutor
done = []
def square(n):
done.append(n)
return n * n
lazy = map(square, [1, 2, 3]) # built-in map: nothing has run yet
print("built-in map ran:", done)
with ThreadPoolExecutor(max_workers=2) as pool:
results = pool.map(square, [1, 2, 3]) # submit all three jobs to the two workers
print("pool ran:", sorted(done))
print(list(results))
# ✅ Expected output:
# built-in map ran: []
# pool ran: [1, 2, 3]
# [1, 4, 9]To handle results in the order they finish instead, Parallelism with concurrent.futures has as_completed().
A process works on a copy of your data
ProcessPoolExecutor has the same map(), but runs the function in other processes. The same call gives the same results, but the list the function appends to stays empty in the main program, because each process changed its own copy:
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
saved = []
def save(item):
saved.append(item)
return item.upper()
if __name__ == "__main__":
for Pool in (ThreadPoolExecutor, ProcessPoolExecutor):
saved.clear()
with Pool(max_workers=2) as pool:
results = list(pool.map(save, ["a", "b", "c"]))
print(Pool.__name__, results, sorted(saved))
# ✅ Expected output:
# ThreadPoolExecutor ['A', 'B', 'C'] ['a', 'b', 'c']
# ProcessPoolExecutor ['A', 'B', 'C'] []The same with a single multiprocessing.Process:
import multiprocessing
results = []
def work():
results.append("from the process")
print("inside the process:", results)
if __name__ == "__main__":
p = multiprocessing.Process(target=work)
p.start()
p.join()
print("back in the main program:", results)
# ✅ Expected output:
# inside the process: ['from the process']
# back in the main program: []So results come back as return values. What goes to a process is pickled, as in Serialization with pickle: a lambda works in a thread pool, but sent to a process pool it stops the program with a PicklingError. Give a process pool a function defined with def at the top of the file.
The __main__ guard: a new process can run your file again
Modules and Packages showed if __name__ == "__main__":. With the spawn start method, a new process imports your file under the name __mp_main__. That start method is called spawn, and this example asks for it on any system:
import multiprocessing
print("this file is running as", __name__, flush=True)
def work():
print("work ran")
if __name__ == "__main__":
ctx = multiprocessing.get_context("spawn")
p = ctx.Process(target=work)
p.start()
p.join()
# ✅ Expected output:
# this file is running as __main__
# this file is running as __mp_main__
# work ranflush=True sends the parent's first line before it starts the child, so buffered output cannot make that line appear last. Code under the guard doesn't run again in the new process. Without the guard, the new process would try to start processes of its own, and Python stops it with a RuntimeError. Spawn is the default on Windows and macOS. Linux commonly used fork by default through Python 3.13; Python 3.14 uses forkserver by default where supported. An explicit context can choose a different supported method, so do not infer behavior from the operating-system name alone. Both spawn and forkserver require safely importable main-module code. Keep process creation behind the main guard.
An error in a worker: a thread's goes by, a pool's comes back with the result
An error in a plain thread ends that thread only. Python prints its traceback, headed Exception in thread and the thread's name, and the main program goes on as if nothing happened:
import threading
def work():
print("worker starting")
print(10 / 0)
t = threading.Thread(target=work)
t.start()
t.join()
print("main program still running")
# ✅ Expected output:
# worker starting
# main program still runningA pool keeps the error and raises it when you reach that result: the first two results print, then the error stops the loop.
from concurrent.futures import ThreadPoolExecutor
def parse(text):
return int(text)
with ThreadPoolExecutor(max_workers=2) as pool:
for n in pool.map(parse, ["1", "2", "three", "4"]):
print(n)
# ✅ Expected output:
# 1
# 2
# ✅ Expected error:
# ValueError: invalid literal for int() with base 10: 'three'Processes do the same:
from concurrent.futures import ProcessPoolExecutor
def share(total, people):
return total / people
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=2) as pool:
for n in pool.map(share, [10, 10, 10], [2, 0, 5]):
print(n)
# ✅ Expected output:
# 5.0
# ✅ Expected error:
# ZeroDivisionError: division by zeroThreads, processes or asyncio?
Choose by what the program spends its time on:
| The program mostly… | Use | Because |
|---|---|---|
| waits on calls that block, such as files, requests or a database | threads, with ThreadPoolExecutor | while one thread waits, another runs |
| waits on many network calls, with async libraries | asyncio | one thread switches tasks at each await |
| calculates in pure Python | processes, with ProcessPoolExecutor and the guard | each process has its own interpreter and GIL |
| calculates in C code that lets go of the GIL, such as hashlib or zlib | threads may be enough | measure it with your own data |
AsyncIO: Event Loop, Tasks & Futures and Advanced Async & Await Patterns cover asyncio, and Thread-Safe Queues (queue) passes work between threads.
Practice: one ticket, two buyers
Two buyers are generators that take turns, the way two threads could: each yield lets the other one run. A Lock should stop both of them buying the last ticket. Fill the three blanks: the Lock method that takes it, the one that gives it back, and the built-in function that runs a generator to its next yield. Each 👉 line under the code fills one blank, top to bottom. This one runs in the browser too.
import threading
stock = {"tickets": 1}
lock = threading.Lock()
def buyer(name):
while not lock.___(blocking=False):
yield # the lock is taken: let the other buyer go
left = stock["tickets"]
yield # a thread can be paused right here
if left > 0:
stock["tickets"] = left - 1
print(name, "got the ticket")
else:
print(name, "found it sold out")
lock.___()
buyers = [buyer("Ada"), buyer("Bo")]
for turn in range(5): # five rounds of turns is plenty
for b in buyers:
___(b, None)
print("tickets left:", stock["tickets"])
# 👉 replace ___ with acquire
# 👉 replace ___ with release
# 👉 replace ___ with next
# ✅ Expected output:
# Ada got the ticket
# Bo found it sold out
# tickets left: 0Without the lock, both buyers read one ticket, and both print that they got it. Leave out the line that gives the lock back, and Bo never gets a turn: only Ada's line prints, before the tickets left.
On your own: a sum in four processes
Add up the squares of 1 to 1,000,000 in four chunks of 250,000, using a ProcessPoolExecutor with four workers, under the __main__ guard. Print each chunk's sum, then the total and whether it equals n(n + 1)(2n + 1) / 6 for n = 1,000,000. Run it with python3 on your computer.
from concurrent.futures import ProcessPoolExecutor
# your code herefrom concurrent.futures import ProcessPoolExecutor
def sum_of_squares(bounds):
start, stop = bounds
return sum(n * n for n in range(start, stop))
if __name__ == "__main__":
chunks = [(1, 250_001), (250_001, 500_001), (500_001, 750_001), (750_001, 1_000_001)]
with ProcessPoolExecutor(max_workers=4) as pool:
parts = list(pool.map(sum_of_squares, chunks))
for part in parts:
print(part)
total = sum(parts)
n = 1_000_000
expected = n * (n + 1) * (2 * n + 1) // 6
print(total, total == expected)
# ✅ Expected output:
# 5208364583375000
# 36458427083375000
# 98958489583375000
# 192708552083375000
# 333333833333500000 TrueThe function has to be defined with def at the top of the file: a lambda in its place stops with a PicklingError.
Practice quiz
names = []. A thread runs add(), which does names.append("Ada"). The program starts the thread, joins it, then prints names. What is printed?
- ['Ada']
- []
- ['Ada', 'Ada']
- None
Answer: ['Ada']. Threads share the program's memory, so the main program sees the thread's append.
counter = {"value": 0}. Three multiprocessing.Process objects each run add_ten(), which adds 10 to counter["value"]. Under the __main__ guard, the program starts all three, joins them, then prints counter["value"]. What is printed?
- 30
- 10
- 0
- It stops with a KeyError
Answer: 0. Each process adds 10 to its own copy of the dict. The main program's copy is never touched.
count_up(n) adds up range(n) and returns n. What does print(list(pool.map(count_up, [200_000, 1, 1000]))) print, with a ThreadPoolExecutor of 3 workers?
- [1, 1000, 200000]
- [200000, 1, 1000]
- [1, 200000, 1000]
- The order changes from run to run
Answer: [200000, 1, 1000]. A pool's map() returns the results in the order of the inputs, however long each one took.
A file defines double(n), then has if __name__ == "main": with a ProcessPoolExecutor that prints list(pool.map(double, [1, 2, 3])), then print("end") at the end. Run with python3, what does it print?
- [2, 4, 6], then end
- It stops with a RuntimeError
- end
- [2, 4, 6]
Answer: end. __name__ is "__main__", not "main", so the if is false and the pool never runs. No error says so.
A file prints loading with flush=True at the top, defines work(), which prints working, and under if __name__ == "__main__": starts multiprocessing.get_context("spawn").Process(target=work) and joins it. What does it print?
- loading, loading, working
- loading, working
- working, loading
- It stops with a RuntimeError
Answer: loading, loading, working. A spawned process starts a fresh Python that runs the file again, under the name __mp_main__, so the top line prints twice. The guard keeps it from starting another process.
greet(name) prints hello and the name. t = threading.Thread(target=greet, args=("Ada")), then t.run(). What happens?
- It prints hello Ada
- It prints hello A
- It prints hello ('Ada',)
- It stops with TypeError: greet() takes 1 positional argument but 3 were given
Answer: It stops with TypeError: greet() takes 1 positional argument but 3 were given. ("Ada") is a string in brackets, not a tuple, so its three letters become three arguments. A one-item tuple needs a comma: ("Ada",).
Two threads each read account["balance"] (100), wait at a threading.Barrier(2) until both have read, then write the value they read minus 80. What does print(account["balance"]) show at the end?
- -60
- 20
- 100
- It stops with a ValueError
Answer: 20. Both read 100 before either writes, so both write 20. One withdrawal is lost.
done = []. record(n) appends n to done and returns it. Inside with ThreadPoolExecutor(max_workers=2) as pool:, the program does results = pool.map(record, [1, 2, 3]) and never reads results. After the with block, what does print(len(done)) show?
- 0
- 1
- It stops with a RuntimeError
- 3
Answer: 3. The pool's map() submits all three jobs without waiting for you to read results. At most two workers run them at once, and the with block waits for them all.
Continue this course
- Previous: Advanced Async & Await Patterns
- Next: Parallelism with concurrent.futures — Run CPU-bound tasks in parallel with ProcessPoolExecutor
- Quick reference: Python cheat sheet
Frequently asked questions
Does with lock: let go of the lock if the code inside raises an error?
Yes. If a ValueError("bad sale") is raised inside with lock:, the except outside catches it, and lock.locked() afterwards gives False. Context Managers & the with Statement in Depth shows why with always cleans up.
Can one thread take the same Lock twice?
No: a second acquire() by the thread that holds it waits forever. With acquire(blocking=False) you can see it: inside with lock:, lock.acquire(blocking=False) gives False. An RLock can be taken again by the thread that holds it: inside with rlock:, the same call gives True. Threading & the GIL covers RLock and the other tools.
How many workers does a pool use if I don't say?
A ThreadPoolExecutor uses the number of CPUs plus four, at most 32, and a ProcessPoolExecutor one per CPU. Python 3.13 counts only the CPUs this process is allowed to use. The number depends on the computer, so set max_workers when it matters.
Is the GIL gone in Python 3.13 or 3.14?
Not in the ordinary build. Since Python 3.13 there is a second, optional build of CPython without the GIL, called free-threaded, installed separately: experimental in 3.13, and officially supported from 3.14. The ordinary python3 still has the GIL: on 3.13 and 3.14, sys._is_gil_enabled() returns True, and asking it to drop the GIL stops Python with an error.
How do I share data between processes?
Send it in as arguments and get it back as return values, as the pools in this lesson do. multiprocessing: True Parallelism covers queues, pipes and shared memory, and Thread-Safe Queues (queue) covers queues between threads.