Thread-Safe Queues (queue)
Published by Brayan K
A queue.Queue is a thread-safe container for passing items between threads — producers put() work in and consumers get() it out, with all the locking handled for you.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Removing a job and finishing a job are different events
Queue coordinates the handoff of items between threads. get removes an item from the waiting queue; task_done acknowledges that the consumer finished handling it. join waits for all submitted items to be acknowledged. It does not return results or prove that each job succeeded.
from queue import Queue
jobs = Queue()
for name in ["draft", "review"]:
jobs.put(name)
completed = []
for _ in range(2):
item = jobs.get()
try:
completed.append(item.upper())
finally:
jobs.task_done()
jobs.join()
print(completed)
# ✅ Expected output:
# ['DRAFT', 'REVIEW']Every put adds unfinished work, and each matching task_done removes one unit of that accounting. get alone does not reduce the unfinished count. If a consumer forgets the acknowledgement, join can wait forever even though no items remain in the waiting buffer. Use finally around handling so the accounting is completed even when handling raises.
from queue import Queue
jobs = Queue()
jobs.task_done()
# ✅ Expected error:
# ValueError: task_done() called too many timesA failure still needs its own outcome record. Acknowledging a failed job means its handling attempt is finished, not that the requested operation succeeded. Retry policy and error reporting belong in the application rather than being inferred from join returning.
A size observation does not reserve the item you saw
from queue import Queue, Empty
jobs = Queue()
jobs.put("A")
print(jobs.empty())
other_consumer_item = jobs.get_nowait()
jobs.task_done()
try:
mine = jobs.get_nowait()
except Empty:
print("no item available")
print(other_consumer_item)
# ✅ Expected output:
# False
# no item available
# AThis single-threaded sequence deliberately models an interleaving: the queue was nonempty, then another consumer removed the item. In concurrent code, empty, full and qsize are observations, not reservations. Request the operation you need and handle Empty or Full, or use the blocking operation with an appropriate waiting policy.
Do not write a worker loop that exits whenever empty becomes true if producers may still add work. An empty buffer can mean only that no item is waiting right now. Use a separate completion protocol, such as explicit stop messages after production ends.
A bounded Queue applies backpressure instead of discarding old work
from queue import Queue, Full
jobs = Queue(maxsize=1)
jobs.put_nowait("first")
try:
jobs.put_nowait("second")
except Full:
print("producer must wait or handle refusal")
print(jobs.get_nowait())
jobs.task_done()
jobs.join()
# ✅ Expected output:
# producer must wait or handle refusal
# firstThe first item is retained. A normal put would wait for space; put_nowait raises Full instead. This differs from a bounded deque that evicts an item to retain recent history, covered in deque & Queues. Choose the data structure from the loss and waiting policy, not from the fact that both contain a sequence.
A bounded queue limits waiting items, not necessarily every item currently being processed by workers. Start consumers before producing more items than the buffer can hold. Otherwise the producer can block before it reaches the code that would start those consumers.
Choose retrieval order independently from worker completion order
from queue import Queue, LifoQueue
for queue_type in (Queue, LifoQueue):
jobs = queue_type()
for item in ["A", "B", "C"]:
jobs.put(item)
order = []
for _ in range(3):
order.append(jobs.get_nowait())
jobs.task_done()
jobs.join()
print(order)
# ✅ Expected output:
# ['A', 'B', 'C']
# ['C', 'B', 'A']FIFO describes retrieval from the waiting queue. With multiple workers, a later retrieved job can still finish before an earlier one. If reporting must follow input order, keep a job ID and order the completed results explicitly.
from queue import PriorityQueue
jobs = PriorityQueue()
jobs.put((2, 0, {"name": "routine"}))
jobs.put((1, 1, {"name": "urgent A"}))
jobs.put((1, 2, {"name": "urgent B"}))
order = []
for _ in range(3):
priority, sequence, payload = jobs.get_nowait()
order.append(payload["name"])
jobs.task_done()
jobs.join()
print(order)
# ✅ Expected output:
# ['urgent A', 'urgent B', 'routine']The unique sequence settles equal priorities before tuple comparison reaches the dictionary payload. Heaps with heapq (Priority Queues) owns heap invariants and tie-breaking in depth. PriorityQueue adds synchronized handoff and task accounting around that ordering.
Give every worker a stop message and join both kinds of lifetime
from queue import Queue, SimpleQueue
from threading import Thread
jobs = Queue(maxsize=2)
results = SimpleQueue()
stop = object()
def worker():
while True:
item = jobs.get()
try:
if item is stop:
return
index, value = item
results.put((index, value * value))
finally:
jobs.task_done()
workers = [Thread(target=worker) for _ in range(2)]
for thread in workers:
thread.start()
for index, value in enumerate([3, 1, 2]):
jobs.put((index, value))
for thread in workers:
jobs.put(stop)
jobs.join()
for thread in workers:
thread.join()
print(sorted(results.get() for _ in range(3)))
print(all(not thread.is_alive() for thread in workers))
# ✅ Expected output:
# [(0, 9), (1, 1), (2, 4)]
# TrueThere is one stop message per worker, after all jobs have been enqueued. The sentinel is acknowledged in finally too, because it was submitted with put. jobs.join waits for acknowledgements; thread.join waits for the corresponding thread to exit. These are different promises, even though both methods are named join.
SimpleQueue is used for results because this example needs an unbounded result channel without task tracking. It has no task_done or join accounting. Threading & the GIL owns the thread lifecycle itself. The worker example requires local Python because the browser cannot start threads.
Record a failed job without abandoning queue accounting
from queue import Queue
jobs = Queue()
for text in ["3", "many", "5"]:
jobs.put(text)
outcomes = []
for _ in range(3):
text = jobs.get()
try:
outcomes.append((text, "ok", int(text)))
except ValueError:
outcomes.append((text, "error", "not an integer"))
finally:
jobs.task_done()
jobs.join()
print(outcomes)
# ✅ Expected output:
# [('3', 'ok', 3), ('many', 'error', 'not an integer'), ('5', 'ok', 5)]The failed conversion is reported as a failure and still acknowledged. In a real worker, decide whether to retry, reject or stop the pipeline. If retrying by putting the job back into a bounded queue from inside every consumer, consider how all workers could block while trying to put; a retry design must not eliminate the consumers needed to free space.
Queue from this module is for threads in one process. A multiprocessing queue has different transport and lifecycle rules, including serialization; multiprocessing: True Parallelism owns that cross-process channel.
Complete one acknowledgement per successful get
Fill the nonblocking retrieval, the acknowledgement and the completion wait. The finally block must acknowledge the retrieved item exactly once.
from queue import Queue
jobs = Queue()
jobs.put("review")
item = jobs.___()
try:
print(item.upper())
finally:
jobs.___()
jobs.___()
print("all handled")
# 👉 replace ___ with get_nowait
# 👉 replace ___ with task_done
# 👉 replace ___ with join
# ✅ Expected output:
# REVIEW
# all handledBuild a bounded worker pipeline with ordered error reporting
Use two non-daemon workers and a Queue with capacity 2. Submit the indexed strings "4", "bad" and "7". Each worker should convert a value to int and double it, or record a ValueError outcome. Acknowledge every item in finally, including one sentinel per worker. Collect exactly three outcomes in a SimpleQueue, wait for all work and both threads, then print outcomes sorted by input index. Do not use empty or qsize to decide that the pipeline has finished.
from queue import Queue, SimpleQueue
from threading import Thread
jobs = Queue(maxsize=2)
outcomes = SimpleQueue()
stop = object()
def worker():
while True:
item = jobs.get()
try:
if item is stop:
return
index, text = item
try:
value = int(text)
except ValueError:
outcomes.put((index, "error", "not an integer"))
else:
outcomes.put((index, "ok", value * 2))
finally:
jobs.task_done()
workers = [Thread(target=worker) for _ in range(2)]
for thread in workers:
thread.start()
inputs = ["4", "bad", "7"]
for item in enumerate(inputs):
jobs.put(item)
for thread in workers:
jobs.put(stop)
jobs.join()
for thread in workers:
thread.join()
for outcome in sorted(outcomes.get() for _ in inputs):
print(outcome)
# ✅ Expected output:
# (0, 'ok', 8)
# (1, 'error', 'not an integer')
# (2, 'ok', 14)The fixed result count follows from the stated contract: each input produces one outcome, even on conversion failure. Sentinels produce no outcome. If a worker can fail outside that handled boundary, a production pipeline needs a separate fatal-error and shutdown policy rather than waiting forever for a missing result.
Practice quiz
Which event does task_done acknowledge?
- An item has merely been looked at
- Handling of one submitted item has finished
- Every worker has exited
- The queue has allocated more capacity
Answer: Handling of one submitted item has finished. It decrements unfinished-work accounting for one submitted item.
Which error ends this program? from queue import Queue q = Queue() q.task_done()
- None
- queue.Empty
- False
- ValueError: task_done() called too many times
Answer: ValueError: task_done() called too many times. Acknowledging nonexistent unfinished work raises a ValueError.
Why can get block after empty returned False?
- Another consumer may remove the item before get
- False means the queue is full
- get always blocks regardless of contents
- The queue sorts items before returning them
Answer: Another consumer may remove the item before get. The observation does not reserve a value for the observing caller.
What does this program print? from queue import Queue, Full q = Queue(maxsize=1) q.put_nowait("A") try: q.put_nowait("B") except Full: print("full") print(q.get_nowait()) q.task_done()
- full B
- A
- full A
- B
Answer: full A. A bounded Queue refuses the new item rather than evicting the old one.
What does Queue.join returning establish?
- All worker threads exited
- All jobs succeeded
- No producer can submit another item
- All submitted items have matching acknowledgements so far
Answer: All submitted items have matching acknowledgements so far. Task accounting is distinct from thread lifetime and success reporting.
What does this program print? from queue import LifoQueue q = LifoQueue() q.put("A") q.put("B") print(q.get_nowait()) q.task_done() print(q.get_nowait()) q.task_done() q.join()
- B A
- A B
- A A
- B B
Answer: B A. LIFO retrieves the most recently queued waiting item first.
For two workers using the sentinel pattern shown here, how many stop messages are submitted?
- One, because they share a queue
- Two, one for each worker
- None, because qsize eventually reaches zero
- One per result plus one
Answer: Two, one for each worker. Each worker consumes one sentinel and returns.
Why are completed outcomes sorted by job index before printing?
- Queue does not preserve any retrieval order
- task_done sorts the queue
- Worker completion order can differ from input order
- SimpleQueue cannot store tuples
Answer: Worker completion order can differ from input order. Explicit IDs let the report restore input order independently of scheduling.
Continue this course
- Previous: multiprocessing: True Parallelism
- Next: contextlib: contextmanager, suppress, ExitStack — Write your own with-statements and manage dynamic resources
- Quick reference: Python cheat sheet