Checkpoint: Concurrency & I/O

Published by Brayan K

A checkpoint is a consolidation lesson where you recap and combine the skills from a whole module — here, the eight tools that let Python do many things at once and work with files and the network.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Define the boundaries before starting the workers

Build a small batch processor: receive fixed byte fragments, assemble newline-terminated records, validate each record in workers and write one report. The input represents nonnegative integer amounts. Invalid records must be counted without losing valid ones. Reporting must follow input order even if workers finish in another order.

This checkpoint uses supplied byte fragments as an offline transport fixture. It does not connect to a server or prove network behavior. Network Programming with sockets owns stream framing and connection lifetime. Here framing is one boundary in a larger pipeline.

from pathlib import Path
from queue import Queue, SimpleQueue
from tempfile import TemporaryDirectory
from threading import Thread

def collect_lines(chunks):
    buffer = bytearray()
    records = []
    for chunk in chunks:
        buffer.extend(chunk)
        if len(buffer) > 64:
            raise ValueError("input buffer exceeds limit")
        while b"\n" in buffer:
            line, _, rest = buffer.partition(b"\n")
            records.append(bytes(line).decode("utf-8"))
            buffer = bytearray(rest)
    if buffer:
        raise EOFError("incomplete final record")
    return records

def run_batch(records):
    jobs = Queue(maxsize=2)
    outcomes = SimpleQueue()
    stop = object()

    def worker():
        while True:
            job = jobs.get()
            try:
                if job is stop:
                    return
                index, text = job
                try:
                    value = int(text)
                    if value < 0:
                        raise ValueError("negative value")
                    outcomes.put((index, "ok", value))
                except ValueError:
                    outcomes.put((index, "invalid", 0))
            finally:
                jobs.task_done()

    workers = [Thread(target=worker) for _ in range(2)]
    for thread in workers:
        thread.start()
    for job in enumerate(records):
        jobs.put(job)
    for _ in workers:
        jobs.put(stop)
    jobs.join()
    for thread in workers:
        thread.join()
    return sorted(outcomes.get() for _ in records)

chunks = [b"10\nb", b"ad\n2", b"0\n"]
records = collect_lines(chunks)
outcomes = run_batch(records)
total = sum(value for _, status, value in outcomes if status == "ok")
rejected = sum(status == "invalid" for _, status, _ in outcomes)

with TemporaryDirectory() as directory:
    root = Path(directory)
    report_file = root / "report.txt"
    report_file.write_text(
        f"accepted total: {total}\nrejected: {rejected}", encoding="utf-8"
    )
    report = report_file.read_text(encoding="utf-8")
print(outcomes)
print(report)
print(root.exists())

# ✅ Expected output:
# [(0, 'ok', 10), (1, 'invalid', 0), (2, 'ok', 20)]
# accepted total: 30
# rejected: 1
# False

Read the output as evidence of these contracts: the outcomes are ordered by their input index, the rejected record has an explicit status, the report contains only accepted values, and the final existence check happens after cleanup. The workers never print or write the report, so their scheduling cannot reorder the displayed result.

An incomplete record is an input failure, not a shorter valid record

chunks = [b"10\n2", b"0\n"]
buffer = bytearray()
records = []
for chunk in chunks:
    buffer.extend(chunk)
    while b"\n" in buffer:
        line, _, remaining = buffer.partition(b"\n")
        records.append(bytes(line).decode("utf-8"))
        buffer = bytearray(remaining)
print(records)
print(len(buffer))

# ✅ Expected output:
# ['10', '20']
# 0

The delimiter belongs to the input protocol. A tail left at the end is not automatically another record: this protocol requires the final delimiter too. The full example also imposes a small buffer limit before parsing each fixture chunk. That limit is a deliberately simple batch contract, not a general high-throughput network parser.

remaining = b"20"
if remaining:
    raise EOFError("incomplete final record")

# ✅ Expected error:
# EOFError: incomplete final record

Do not begin concurrent work until the input has been converted to a valid batch in this exercise. A streaming service may process earlier records while later input is still arriving, but it then needs additional cancellation, partial-result and backpressure decisions. That is a different contract, not a harmless optimization of this one.

An outcome keeps identity even when processing fails

records = ["10", "bad", "-2", "20"]
outcomes = []
for index, text in enumerate(records):
    try:
        value = int(text)
        if value < 0:
            raise ValueError("negative value")
        outcomes.append((index, "ok", value))
    except ValueError:
        outcomes.append((index, "invalid", 0))
print(outcomes)
print(sum(value for _, status, value in outcomes if status == "ok"))

# ✅ Expected output:
# [(0, 'ok', 10), (1, 'invalid', 0), (2, 'invalid', 0), (3, 'ok', 20)]
# 30

The invalid status is essential: a rejected value and a valid zero both carry a numeric zero, but they mean different things. Preserve that distinction in the outcome instead of making the reporting stage guess from the number. The error boundary catches only expected validation failures; an unrelated programming error should remain visible.

The single-threaded version establishes the validation contract before concurrency is added. Threading & the GIL owns thread lifecycle, and Thread-Safe Queues (queue) owns queue accounting. The checkpoint combines those contracts without changing validation semantics.

Completion accounting is independent of result order

completed = [(2, "ok", 20), (0, "ok", 10), (1, "invalid", 0)]
ordered = sorted(completed)
print([index for index, _, _ in ordered])
print([status for _, status, _ in ordered])
print(sum(value for _, status, value in ordered if status == "ok"))

# ✅ Expected output:
# [0, 1, 2]
# ['ok', 'invalid', 'ok']
# 30

This fixture intentionally supplies results out of order. The integer index is a stable identity, so sorting restores report order without relying on timing. It is not enough to assert that a FIFO queue guarantees ordered completion: different workers can take different amounts of time.

In the full program, jobs.join waits for submitted items to receive task_done. Each thread.join then waits for that worker to exit. Both are necessary for the stated shutdown contract. One stop message is enqueued per worker after all jobs, and each stop message is acknowledged in the same finally block.

Read the deliverable before releasing its scratch storage

from pathlib import Path
from tempfile import TemporaryDirectory

def make_report(total, rejected):
    with TemporaryDirectory() as directory:
        root = Path(directory)
        report = root / "report.txt"
        report.write_text(f"total={total}; rejected={rejected}", encoding="utf-8")
        contents = report.read_text(encoding="utf-8")
    return contents, root.exists()

text, still_exists = make_report(30, 1)
print(text)
print(still_exists)

# ✅ Expected output:
# total=30; rejected=1
# False

Returning the path would return a reference to a file that has already been removed. This helper returns the report text instead. A real application that needs a permanent file must deliberately publish it to an approved destination before releasing scratch storage. Temporary Files & Directories owns temporary lifetimes; High-Level File Operations (shutil) owns copying and moving a chosen result.

Complete the acknowledgement even when validation fails

This smaller single-consumer exercise isolates queue accounting. Fill the acknowledgement and completion-wait calls. The invalid item still has to be acknowledged after its failure is recorded.

from queue import Queue

jobs = Queue()
for text in ["10", "bad", "20"]:
    jobs.put(text)
accepted = []
rejected = 0
for _ in range(3):
    text = jobs.get()
    try:
        accepted.append(int(text))
    except ValueError:
        rejected += 1
    finally:
        jobs.___()
jobs.___()
print(sum(accepted))
print(rejected)

# 👉 replace ___ with task_done
# 👉 replace ___ with join

# ✅ Expected output:
# 30
# 1

Extend the batch while preserving every contract

Adapt the complete processor to the records 7, -1, bad and 5, split across chunks. Accept only nonnegative integers. Produce one indexed outcome for every record, then create a temporary report containing the accepted count, rejected count and accepted total. Read it before cleanup. Use two workers, bounded waiting capacity, one stop message per worker, and explicit joins. Do not print from the workers.

from pathlib import Path
from queue import Queue, SimpleQueue
from tempfile import TemporaryDirectory
from threading import Thread

def collect_lines(chunks):
    buffer = bytearray()
    records = []
    for chunk in chunks:
        buffer.extend(chunk)
        if len(buffer) > 64:
            raise ValueError("input buffer exceeds limit")
        while b"\n" in buffer:
            line, _, rest = buffer.partition(b"\n")
            records.append(bytes(line).decode("utf-8"))
            buffer = bytearray(rest)
    if buffer:
        raise EOFError("incomplete final record")
    return records

def run_batch(records):
    jobs = Queue(maxsize=2)
    outcomes = SimpleQueue()
    stop = object()

    def worker():
        while True:
            job = jobs.get()
            try:
                if job is stop:
                    return
                index, text = job
                try:
                    value = int(text)
                    if value < 0:
                        raise ValueError("negative value")
                    outcomes.put((index, "ok", value))
                except ValueError:
                    outcomes.put((index, "invalid", 0))
            finally:
                jobs.task_done()

    workers = [Thread(target=worker) for _ in range(2)]
    for thread in workers:
        thread.start()
    for job in enumerate(records):
        jobs.put(job)
    for _ in workers:
        jobs.put(stop)
    jobs.join()
    for thread in workers:
        thread.join()
    return sorted(outcomes.get() for _ in records)

chunks = [b"7\n-1\nb", b"ad\n5\n"]
records = collect_lines(chunks)
outcomes = run_batch(records)
accepted = sum(status == "ok" for _, status, _ in outcomes)
total = sum(value for _, status, value in outcomes if status == "ok")
rejected = sum(status == "invalid" for _, status, _ in outcomes)

with TemporaryDirectory() as directory:
    root = Path(directory)
    report_file = root / "report.txt"
    report_file.write_text(
        f"accepted: {accepted}\nrejected: {rejected}\ntotal: {total}", encoding="utf-8"
    )
    report = report_file.read_text(encoding="utf-8")
print(outcomes)
print(report)
print(root.exists())

# ✅ Expected output:
# [(0, 'ok', 7), (1, 'invalid', 0), (2, 'invalid', 0), (3, 'ok', 5)]
# accepted: 2
# rejected: 2
# total: 12
# False

Compare the two full programs by their contracts, not by how many lines changed. Input framing, outcome identity, queue acknowledgements, worker shutdown and file lifetime all remain explicit. This small batch does not claim production handling of unexpected worker death, a hanging task, an unbounded input stream or network reconnection. Those requirements would need their own failure and cancellation design.

Practice quiz

What does this program print? outcomes = [(2, "ok", 5), (0, "ok", 7), (1, "invalid", 0)] print([item[0] for item in sorted(outcomes)])

  • [0, 1, 2]
  • [2, 0, 1]
  • [0, 2]
  • [1, 0, 2]

Answer: [0, 1, 2]. Stable input indices restore the required reporting order.

A worker catches a validation error. What must still happen for the retrieved queue item?

  • Nothing; failed work does not count
  • task_done in the finally path
  • Another put of the same item automatically
  • Only a print from the worker

Answer: task_done in the finally path. A handling attempt still owns exactly one acknowledgement.

What does this program print? outcomes = [(0, "ok", 0), (1, "invalid", 0)] print(sum(status == "ok" for _, status, _ in outcomes))

  • 0
  • 2
  • 1
  • False

Answer: 1. An explicit status distinguishes accepted zero from a rejected record.

Why does the full processor call both jobs.join and thread.join?

  • They are interchangeable and both are unnecessary
  • They make the GIL disappear
  • They force FIFO worker completion
  • They wait for work accounting and worker lifetimes respectively

Answer: They wait for work accounting and worker lifetimes respectively. Acknowledged queue items and exited worker threads are different conditions.

What does this program print? records = ["7", "5"] print(sum(int(text) for text in records))

  • 12
  • 75
  • 2
  • 0

Answer: 12. Validation produces integers before reporting adds them.

What should a helper return if its TemporaryDirectory is removed before returning?

  • A promise that the removed file remains readable
  • The report data already read, or a deliberately published permanent result
  • Only the temporary pathname
  • An open handle whose lifetime was already closed

Answer: The report data already read, or a deliberately published permanent result. A pathname reference does not preserve a removed resource.

What does this program print? buffer = b"12 3" line, _, rest = buffer.partition(b" ") print(line) print(rest)

  • b'12 3' b''
  • b'3' b'12'
  • b'12' b'3'
  • b'12' b''

Answer: b'12' b'3'. The unfinished tail belongs to a later record and must be retained.

What do the fixed byte-chunk fixtures establish?

  • That a real server is reachable
  • That operating-system timeouts work
  • That TLS authentication succeeded
  • That the framing logic handles the tested fragmentation patterns

Answer: That the framing logic handles the tested fragmentation patterns. A deterministic transport double checks protocol logic, not live networking.

Continue this course

Related lessons