multiprocessing: True Parallelism
Published by Brayan K
The multiprocessing module runs your code in separate OS processes — each with its own Python interpreter and its own GIL — so CPU-heavy work can run on multiple cores at the same time.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
A child process owns separate ordinary Python objects
Processes can run work in separate interpreters, but ordinary lists and dictionaries are not shared application memory. Send inputs explicitly and return results through a communication channel. Starting a process is not a promise of a speedup: startup, serialization and data transfer can outweigh the calculation.
Run these examples as saved Python files on your computer. Each chooses the spawn context and puts process creation behind a main guard. That keeps the behavior consistent across the Python versions and operating systems used here. The browser editor cannot start operating-system processes.
from multiprocessing import get_context
def extend_copy(values, output):
values.append("child")
output.put(values)
if __name__ == "__main__":
context = get_context("spawn")
output = context.Queue()
values = ["parent"]
worker = context.Process(target=extend_copy, args=(values, output))
worker.start()
received = output.get()
worker.join()
print(received)
print(values)
print(worker.exitcode)
output.close()
output.join_thread()
worker.close()
# ✅ Expected output:
# ['parent', 'child']
# ['parent']
# 0The child receives a serialized copy of values and sends another serialized value back. The parent’s list is unchanged. Receive the queued result before joining a producer that may still be flushing data; joining first can deadlock when a large queued payload has not been drained. Concurrency in Python: Threads vs Processes owns choosing between concurrency models.
Spawn imports the module; the guard prevents recursive process creation
from multiprocessing import get_context
setting = "module default"
def report_setting(explicit, output):
output.put((setting, explicit))
if __name__ == "__main__":
setting = "parent change"
context = get_context("spawn")
output = context.Queue()
worker = context.Process(target=report_setting, args=(setting, output))
worker.start()
print(output.get())
worker.join()
output.close()
output.join_thread()
worker.close()
# ✅ Expected output:
# ('module default', 'parent change')The spawned interpreter imports the module, defining setting from its top-level assignment. It does not execute the guarded parent mutation. The argument carries the current parent value intentionally. Keep worker functions at module level so the child can import them; a function defined only inside the guarded block is not available when the child imports the file.
Default start methods differ by platform and Python version. Python 3.14 changed supported POSIX defaults to forkserver, while Windows and macOS use spawn by default. Libraries should accept or use an explicit compatible context rather than forcing a global method unexpectedly. Create communication and synchronization objects from that same context.
A nested function or lambda can work in a thread but fail as a spawn target because its name cannot be imported in the new interpreter. Serialization with pickle owns serialization and class/function name resolution. Replacing threads with processes can therefore require restructuring the callable and its inputs.
Join waits for process lifetime; a channel carries its result
from multiprocessing import get_context
def calculate():
return 42
if __name__ == "__main__":
context = get_context("spawn")
worker = context.Process(target=calculate)
worker.start()
joined = worker.join()
print(joined is None)
print(worker.exitcode)
print(worker.is_alive())
worker.close()
# ✅ Expected output:
# True
# 0
# FalseThe returned 42 is not delivered to the parent by Process. The parent gets a completion status and must use Queue, Pipe or a pool result API for data. A timed join does not terminate a running process; inspect is_alive to see whether it finished. Close a Process object after it has stopped when you no longer need its handles.
from multiprocessing import get_context
worker = get_context("spawn").Process()
worker.join()
# ✅ Expected error:
# AssertionError: can only join a started processUse Pool for ordered independent results and explicit error retrieval
from multiprocessing import get_context
def square(value):
return value * value
if __name__ == "__main__":
context = get_context("spawn")
with context.Pool(processes=2) as pool:
results = pool.map(square, [4, 1, 3])
print(results)
# ✅ Expected output:
# [16, 1, 9]map returns results in input order, regardless of which worker finishes first. Keep output in the parent when the order is part of the lesson’s contract. The context manager cleans up the pool; in this example map has completed before that exit. Do not submit asynchronous jobs and assume leaving a Pool context waits for all their results: retrieve the results you require first.
from multiprocessing import get_context
def positive(value):
if value < 1:
raise ValueError("value must be positive")
return value
if __name__ == "__main__":
context = get_context("spawn")
with context.Pool(processes=2) as pool:
result = pool.apply_async(positive, (0,))
try:
result.get()
except ValueError as error:
print(type(error).__name__)
print(str(error))
# ✅ Expected output:
# ValueError
# value must be positiveSubmitting a task is not proof that it succeeded. get retrieves the outcome and re-raises a worker exception in the caller. The example prints only the stable exception type and message, avoiding environment-specific traceback paths. Parallelism with concurrent.futures owns the related Future interface and executor cancellation rules.
Shared memory requires a shared synchronization rule
from multiprocessing import get_context
def add_many(total, amount):
for _ in range(100):
with total.get_lock():
total.value += amount
if __name__ == "__main__":
context = get_context("spawn")
total = context.Value("i", 0)
workers = [context.Process(target=add_many, args=(total, amount)) for amount in (1, 2)]
for worker in workers:
worker.start()
for worker in workers:
worker.join()
if worker.exitcode != 0:
raise RuntimeError("worker failed")
worker.close()
print(total.value)
# ✅ Expected output:
# 300The synchronized Value has a lock, but total.value += amount is a read followed by a write. Hold the lock across that whole operation. Separately protected property access is not the same as one atomic update. The fixed-width storage type also has a numeric range; choose a representation suitable for the values rather than assuming Python’s unbounded int behavior.
Managed proxy objects can share higher-level state through a manager process, but each interaction has communication cost and compound operations still need a synchronization policy. For independent jobs, returning values and combining them in the parent is often simpler than sharing mutable state.
Request a clean stop instead of treating termination as ordinary cleanup
from multiprocessing import get_context
def wait_for_stop(stop, messages):
messages.put("ready")
stop.wait()
messages.put("stopped")
if __name__ == "__main__":
context = get_context("spawn")
stop = context.Event()
messages = context.Queue()
worker = context.Process(target=wait_for_stop, args=(stop, messages))
worker.start()
print(messages.get())
stop.set()
print(messages.get())
worker.join()
print(worker.exitcode)
messages.close()
messages.join_thread()
worker.close()
# ✅ Expected output:
# ready
# stopped
# 0The parent knows the worker reached its waiting boundary before requesting shutdown. A real task must check the signal at appropriate boundaries or use operations that can be woken. terminate is abrupt: cleanup handlers may not run, and terminating a process using a queue or lock can leave communication or synchronization unusable. Design the normal shutdown path first.
Complete a portable pool calculation
Choose the explicit start context, protect creation with the main guard and retrieve ordered map results. Keep the worker definition importable at module level.
from multiprocessing import get_context
def double(value):
return value * 2
if __name__ == ___:
context = get_context(___)
with context.Pool(processes=2) as pool:
values = pool.___(double, [2, 4, 6])
print(values)
# 👉 replace ___ with "__main__"
# 👉 replace ___ with "spawn"
# 👉 replace ___ with map
# ✅ Expected output:
# [4, 8, 12]Combine independent chunk results in the parent
Define a module-level summarize function accepting a pair of a label and a tuple of integers. Return the label and the sum of their squares. Use a two-process spawn Pool to process north with 1,2,3 and south with 4,5. Print each returned label and subtotal in input order, then print the combined total and compare it with a sequential calculation over 1 through 5. Do not update a global total inside workers or print from them.
from multiprocessing import get_context
def summarize(chunk):
label, values = chunk
return label, sum(value * value for value in values)
if __name__ == "__main__":
chunks = [("north", (1, 2, 3)), ("south", (4, 5))]
context = get_context("spawn")
with context.Pool(processes=2) as pool:
results = pool.map(summarize, chunks)
for label, subtotal in results:
print(label, subtotal)
total = sum(subtotal for label, subtotal in results)
print(total)
print(total == sum(value * value for value in range(1, 6)))
# ✅ Expected output:
# north 14
# south 41
# 55
# TrueThis tiny workload demonstrates ownership and result ordering, not a performance improvement. Larger real workloads need measurement that includes process startup and serialization costs.
Practice quiz
After a spawned child appends to an ordinary list argument, what happens to the parent’s original list?
- It receives the append automatically
- It becomes a shared-memory array
- It remains unchanged unless the parent applies a returned result
- It is deleted when the child exits
Answer: It remains unchanged unless the parent applies a returned result. Spawn transfers a serialized value, not a shared reference to the parent list.
Why is process creation protected by if __name__ == "__main__"?
- To prevent child module import from starting the parent orchestration again
- To make every function picklable
- To remove startup cost
- To force the same default start method everywhere
Answer: To prevent child module import from starting the parent orchestration again. Spawn imports the module to find worker definitions; guarded orchestration must not repeat.
What does this program print? from multiprocessing import get_context def twice(n): return n * 2 if __name__ == "__main__": with get_context("spawn").Pool(2) as pool: print(pool.map(twice, [3, 1, 2]))
- [2, 4, 6]
- [3, 1, 2]
- None
- [6, 2, 4]
Answer: [6, 2, 4]. Pool.map preserves input order in its returned results.
Where should a spawn worker callable normally be defined?
- Inside the guarded main block only
- At importable module level
- Only inside another worker
- As an anonymous lambda passed inline
Answer: At importable module level. The child must be able to resolve the function when it imports the module.
Which error ends this program? from multiprocessing import get_context worker = get_context("spawn").Process() worker.join()
- None
- AssertionError: can only join a started process
- False
- RuntimeError: process returned no value
Answer: AssertionError: can only join a started process. join requires a process that has actually started.
Why acquire total.get_lock around total.value += amount?
- To make the number unlimited in size
- To avoid serializing any arguments
- To ensure every process uses a separate lock
- To protect the combined read-and-write operation
Answer: To protect the combined read-and-write operation. Individual synchronized property access does not make the whole update atomic.
When a child may be flushing a large Queue payload, what should the parent normally do before joining that producer?
- Drain the required queued data
- Close the queue without reading
- Wait for the child and ignore the queue
- Terminate the child immediately
Answer: Drain the required queued data. A producer waiting for its buffered data to flush can be blocked by a parent that joins before consuming.
What does this program print? from multiprocessing import get_context def work(): return 9 if __name__ == "__main__": p = get_context("spawn").Process(target=work) p.start() print(p.join() is None) print(p.exitcode) p.close()
- False 9
- True 9
- True 0
- False 0
Answer: True 0. join supplies no target return value; exitcode describes successful process completion.
Continue this course
- Previous: Threading & the GIL
- Next: Thread-Safe Queues (queue) — Coordinate producers and consumers safely with queue.Queue
- Quick reference: Python cheat sheet