Advanced Async & Await

Published by Brayan K

Asynchronous programming is at the heart of modern Python applications. Whether you're building high-performance web APIs, data pipelines, websocket apps, scrapers, or microservices — mastering advanced async/await patterns gives you the ability to build systems that scale effortlessly.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A timeout cancels the job it gives up on

This lesson builds on AsyncIO: Event Loop, Tasks & Futures: async def, await, asyncio.run(), tasks and gather(). Like that lesson, it uses await asyncio.sleep(0) where a real program would wait for a reply, so the order of the printed lines shows what runs when. Web APIs, websocket apps and scrapers have lessons of their own: Building APIs with FastAPI, Network Programming with sockets and Web Scraping with BeautifulSoup.

asyncio.wait_for() gives a coroutine a deadline in seconds. When the time is up, it doesn't just stop waiting: it cancels the job, which gets a CancelledError at the line where it's paused, waits for the job to stop, and then raises a timeout error for you. The job below would take 10 seconds; the deadline is 0.01:

import asyncio

async def slow_report():
    try:
        await asyncio.sleep(10)
        return "report"
    except asyncio.CancelledError:
        print("slow_report was cancelled")
        raise

async def main():
    try:
        print(await asyncio.wait_for(slow_report(), timeout=0.01))
    except asyncio.TimeoutError:
        print("gave up after 0.01 s")

asyncio.run(main())

# ✅ Expected output:
# slow_report was cancelled
# gave up after 0.01 s

Within the deadline, you get the result as usual:

import asyncio

async def quick_report():
    await asyncio.sleep(0)
    return "report ready"

async def main():
    print(await asyncio.wait_for(quick_report(), timeout=1))

asyncio.run(main())

# ✅ Expected output:
# report ready

Catch the timeout as asyncio.TimeoutError, which works on every version. From Python 3.11 on it's the built-in TimeoutError; on 3.10 it's a class of its own, so an uncaught timeout is named differently there. Python 3.11 also added asyncio.timeout(), which puts one deadline on a whole block:

import asyncio

async def main():
    try:
        async with asyncio.timeout(0.01):
            await asyncio.sleep(10)
    except TimeoutError:
        print("timed out")

asyncio.run(main())

# ✅ Expected output:
# timed out

Catching CancelledError doesn't make cancelling safe

task.cancel() asks a task to stop: Python raises CancelledError inside it, at the await where it's paused. A handler that catches the error and carries on refuses the request, and the task then counts as finished, not cancelled:

import asyncio

async def worker():
    try:
        while True:
            await asyncio.sleep(0)
    except asyncio.CancelledError:
        print("Worker cancelled safely")

async def main():
    task = asyncio.create_task(worker())
    await asyncio.sleep(0)
    task.cancel()
    await task
    print("task.cancelled():", task.cancelled())

asyncio.run(main())

# ✅ Expected output:
# Worker cancelled safely
# task.cancelled(): False

The same refusal breaks a deadline. wait_for() cancels the job and then waits for it to stop, so a job that catches the CancelledError and carries on keeps it waiting, and what the job returns comes back from wait_for() with no timeout error:

import asyncio

async def stubborn_report():
    try:
        await asyncio.sleep(10)
        return "report"
    except asyncio.CancelledError:
        print("stubborn_report ignores the cancel")
        await asyncio.sleep(0.2)
        return "late report"

async def main():
    try:
        print(await asyncio.wait_for(stubborn_report(), timeout=0.01))
    except asyncio.TimeoutError:
        print("gave up after 0.01 s")

asyncio.run(main())

# ✅ Expected output:
# stubborn_report ignores the cancel
# late report

The deadline was 0.01 seconds, but wait_for() waited another 0.2 for the handler and handed back "late report", so main()'s except never ran.

Clean up in finally instead, and let the error go on. Whoever awaits the task gets the CancelledError, and the task counts as cancelled. If you do catch it, to log something, end the handler with raise, as slow_report() does above.

import asyncio

async def worker():
    try:
        while True:
            await asyncio.sleep(0)
    finally:
        print("worker cleans up")

async def main():
    task = asyncio.create_task(worker())
    await asyncio.sleep(0)
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("main: worker is cancelled")
    print("task.cancelled():", task.cancelled())

asyncio.run(main())

# ✅ Expected output:
# worker cleans up
# main: worker is cancelled
# task.cancelled(): True

Awaiting a cancelled task without catching the error stops the program with it:

import asyncio

async def job():
    await asyncio.sleep(10)

async def main():
    task = asyncio.create_task(job())
    await asyncio.sleep(0)
    task.cancel()
    await task

asyncio.run(main())

# ✅ Expected error:
# asyncio.exceptions.CancelledError

A bare except: is worse than it looks here. Exception Handling showed that KeyboardInterrupt and SystemExit aren't an Exception; neither is CancelledError, but except: with nothing after it catches it anyway:

import asyncio

print(issubclass(asyncio.CancelledError, Exception))
print(issubclass(asyncio.CancelledError, BaseException))

# ✅ Expected output:
# False
# True

So a retry loop with a bare except: swallows every cancel():

import asyncio

async def retry(attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            await asyncio.sleep(10)
            return "ok"
        except:
            print("attempt", attempt, "failed, retrying")
    return "gave up"

async def main():
    task = asyncio.create_task(retry())
    await asyncio.sleep(0)
    for _ in range(3):
        task.cancel()
        await asyncio.sleep(0)
    print(await task)
    print("cancelled?", task.cancelled())

asyncio.run(main())

# ✅ Expected output:
# attempt 1 failed, retrying
# attempt 2 failed, retrying
# attempt 3 failed, retrying
# gave up
# cancelled? False

With except Exception:, one cancel() is enough:

import asyncio

async def retry(attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            await asyncio.sleep(10)
            return "ok"
        except Exception:
            print("attempt", attempt, "failed, retrying")
    return "gave up"

async def main():
    task = asyncio.create_task(retry())
    await asyncio.sleep(0)
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("cancelled?", task.cancelled())

asyncio.run(main())

# ✅ Expected output:
# cancelled? True

Start everything, but limit how many run at once

Starting a thousand downloads at once can swamp the server, or your own connection. asyncio.Semaphore(n) lets at most n tasks into an async with block at a time; the others wait at the door until one leaves. Four jobs, two at a time:

import asyncio

async def job(n, sem):
    async with sem:
        print("start", n)
        await asyncio.sleep(0)
        print("end", n)

async def main():
    sem = asyncio.Semaphore(2)
    await asyncio.gather(*(job(n, sem) for n in range(1, 5)))

asyncio.run(main())

# ✅ Expected output:
# start 1
# start 2
# end 1
# end 2
# start 3
# start 4
# end 3
# end 4

Jobs 3 and 4 start only after 1 and 2 have left the block. Writing with sem: without async is an error: a TypeError saying the Semaphore doesn't support the context manager protocol on Python 3.11 and later, and a different one on 3.10. From 3.14 the message also asks whether you meant async with.

Queues: put, get, join and task_done instead of sleeping and checking

An asyncio.Queue hands work from one task to others. await get() waits for an item, task_done() says one is finished, and join() waits until every item put in has been marked done. These cooks loop forever, so main() cancels them once the orders are done:

import asyncio

async def cook(name, orders):
    while True:
        dish = await orders.get()
        print(name, "cooks", dish)
        await asyncio.sleep(0)
        orders.task_done()

async def main():
    orders = asyncio.Queue()
    for dish in ["soup", "salad", "pie", "tea"]:
        orders.put_nowait(dish)
    cooks = [asyncio.create_task(cook(name, orders)) for name in ["Ana", "Ben"]]
    await orders.join()
    for c in cooks:
        c.cancel()
    print("all orders done")

asyncio.run(main())

# ✅ Expected output:
# Ana cooks soup
# Ben cooks salad
# Ana cooks pie
# Ben cooks tea
# all orders done

A queue with a maxsize makes put() wait while it's full, so a fast producer can't run ahead of the consumer. No sleeping, no checking the size:

import asyncio

async def producer(q):
    for item in ["a", "b", "c"]:
        await q.put(item)
        print("put", item)
    await q.put(None)

async def consumer(q):
    while True:
        item = await q.get()
        if item is None:
            break
        print("got", item)

async def main():
    q = asyncio.Queue(maxsize=1)
    await asyncio.gather(producer(q), consumer(q))

asyncio.run(main())

# ✅ Expected output:
# put a
# got a
# put b
# got b
# put c
# got c

get_nowait() doesn't wait, so on an empty queue it raises QueueEmpty:

import asyncio

async def main():
    q = asyncio.Queue()
    q.get_nowait()

asyncio.run(main())

# ✅ Expected error:
# asyncio.queues.QueueEmpty

An asyncio.Queue is for tasks on one event loop, and takes no lock. A queue between threads is queue.Queue, in Thread-Safe Queues (queue).

One failure: gather() lets the others run on, and a TaskGroup stops them

AsyncIO: Event Loop, Tasks & Futures showed that gather() raises the first error but leaves the other tasks running. With return_exceptions=True, it raises nothing: an error becomes that coroutine's result, and you sort them out afterwards:

import asyncio

async def ok():
    await asyncio.sleep(0)
    return "ok"

async def bad():
    raise ValueError("bad item")

async def main():
    results = await asyncio.gather(ok(), bad(), return_exceptions=True)
    for r in results:
        if isinstance(r, Exception):
            print("failed:", type(r).__name__, r)
        else:
            print("worked:", r)

asyncio.run(main())

# ✅ Expected output:
# worked: ok
# failed: ValueError bad item

A TaskGroup, new in Python 3.11, does the opposite: when one task fails, it cancels the others, waits for them, and raises the errors together as an ExceptionGroup. except* catches the errors of one type from it:

import asyncio

async def slow():
    try:
        for _ in range(3):
            await asyncio.sleep(0)
        print("slow finished")
    except asyncio.CancelledError:
        print("slow was cancelled")
        raise

async def bad():
    raise ValueError("bad item")

async def main():
    try:
        async with asyncio.TaskGroup() as tg:
            tg.create_task(slow())
            tg.create_task(bad())
    except* ValueError as group:
        print("caught:", [str(e) for e in group.exceptions])

asyncio.run(main())

# ✅ Expected output:
# slow was cancelled
# caught: ['bad item']

Uncaught, an ExceptionGroup's traceback ends with a box drawn around the errors, so its last line isn't the error itself. TaskGroup, ExceptionGroup and except* don't exist in Python 3.10.

async with and async for await inside

async with calls __aenter__ and __aexit__ and awaits them, as with calls __enter__ and __exit__ in Context Managers & the with Statement in Depth. async for asks an async generator, a function with both async def and yield (Generators & Iterators Mastery), for each value and awaits it, so other tasks can run between the values:

import asyncio

class Connection:
    async def __aenter__(self):
        await asyncio.sleep(0)
        print("connected")
        return self

    async def __aexit__(self, exc_type, exc, tb):
        await asyncio.sleep(0)
        print("closed")

async def rows(n):
    for i in range(1, n + 1):
        await asyncio.sleep(0)
        yield f"row {i}"

async def main():
    async with Connection():
        async for row in rows(3):
            print(row)

asyncio.run(main())

# ✅ Expected output:
# connected
# row 1
# row 2
# row 3
# closed

Comprehensions work with async for too:

import asyncio

async def rows(n):
    for i in range(1, n + 1):
        await asyncio.sleep(0)
        yield i * i

async def main():
    print([x async for x in rows(4)])

asyncio.run(main())

# ✅ Expected output:
# [1, 4, 9, 16]

A plain for can't loop over an async generator:

import asyncio

async def rows(n):
    for i in range(n):
        yield i

async def main():
    for row in rows(3):
        print(row)

asyncio.run(main())

# ✅ Expected error:
# TypeError: 'async_generator' object is not iterable

And async for works only inside async def:

async def rows():
    yield 1

def main():
    async for r in rows():
        print(r)

print("never printed")

# ✅ Expected error:
# SyntaxError: 'async for' outside async function

In the same way, with without async on an async context manager such as Connection is an error, a TypeError on Python 3.11 and later.

A blocking call stops every task

Tasks take turns only at an await that pauses. time.sleep() doesn't pause the coroutine, it stops the whole thread, so nothing else runs while it sleeps. await asyncio.sleep() pauses only the coroutine that calls it:

import asyncio
import time

async def nap(blocking):
    print("nap starts")
    if blocking:
        time.sleep(0.05)
    else:
        await asyncio.sleep(0.05)
    print("nap ends")

async def ticker():
    for i in range(2):
        print("tick", i)
        await asyncio.sleep(0)

async def main():
    print("-- time.sleep")
    await asyncio.gather(nap(True), ticker())
    print("-- asyncio.sleep")
    await asyncio.gather(nap(False), ticker())

asyncio.run(main())

# ✅ Expected output:
# -- time.sleep
# nap starts
# nap ends
# tick 0
# tick 1
# -- asyncio.sleep
# nap starts
# tick 0
# tick 1
# nap ends

A long calculation with no await holds up the others in the same way, as AsyncIO: Event Loop, Tasks & Futures showed. For a blocking call you can't rewrite, such as one from a library without async support, asyncio.to_thread() runs it in a thread of its own and lets the loop carry on meanwhile:

import asyncio
import time

def read_sensor():
    time.sleep(0.01)
    return 21.5

async def main():
    print(await asyncio.to_thread(read_sensor))

asyncio.run(main())

# ✅ Expected output:
# 21.5

This site's Run button can't start threads, so there to_thread() runs the function on the page's only thread: you get the same value, but nothing else runs meanwhile. For calculations rather than waiting, Concurrency in Python: Threads vs Processes compares threads and processes.

One thread doesn't mean no races

Tasks switch only at an await, but that's enough for a race. Two buyers check the shop, then pause before buying. Both see one apple, and both take it:

import asyncio

async def buy(shop, name):
    if shop["apples"] > 0:
        await asyncio.sleep(0)
        shop["apples"] -= 1
        print(name, "bought an apple")

async def main():
    shop = {"apples": 1}
    await asyncio.gather(buy(shop, "Ana"), buy(shop, "Ben"))
    print("apples left:", shop["apples"])

asyncio.run(main())

# ✅ Expected output:
# Ana bought an apple
# Ben bought an apple
# apples left: -1

An asyncio.Lock around the check and the change makes the second buyer wait until the first has finished:

import asyncio

async def buy(shop, lock, name):
    async with lock:
        if shop["apples"] > 0:
            await asyncio.sleep(0)
            shop["apples"] -= 1
            print(name, "bought an apple")
        else:
            print(name, "found none left")

async def main():
    shop = {"apples": 1}
    lock = asyncio.Lock()
    await asyncio.gather(buy(shop, lock, "Ana"), buy(shop, lock, "Ben"))
    print("apples left:", shop["apples"])

asyncio.run(main())

# ✅ Expected output:
# Ana bought an apple
# Ben found none left
# apples left: 0

An asyncio.Lock can't be taken twice by the same task: a second await lock.acquire() before release() waits forever.

Practice: five uploads, two at a time

Each upload() counts itself in and out of the limited block, so most at once records the busiest moment. Fill the three blanks: the statement that enters the semaphore's block, the asyncio class that limits how many tasks are inside, and the function that runs all five and collects their sizes. The hints under the code give the answers, in the order the blanks appear.

import asyncio

async def upload(n, sem, stats):
    ___ sem:
        stats["now"] += 1
        stats["most"] = max(stats["most"], stats["now"])
        await asyncio.sleep(0)
        stats["now"] -= 1
    return n * 10

async def main():
    sem = asyncio.___(2)
    stats = {"now": 0, "most": 0}
    sizes = await asyncio.___(*(upload(n, sem, stats) for n in range(1, 6)))
    print(sizes)
    print("most at once:", stats["most"])

asyncio.run(main())

# 👉 replace ___ with async with
# 👉 replace ___ with Semaphore
# 👉 replace ___ with gather

# ✅ Expected output:
# [10, 20, 30, 40, 50]
# most at once: 2

With Semaphore(5) instead, all five get in at once and most at once is 5. Without the * before the generator expression, gather() gets one generator instead of five coroutines and stops with an error, worded differently on Python 3.12 and later.

On your own: three quotes with a deadline each

get_quote(source, delay) waits delay seconds, then returns the source and a price. Ask three sources at the same time: north and east answer at once, and south takes 10 seconds. Give each one its own deadline of 0.05 seconds, show a source that misses it as timed out, and print the answers in the order you asked.

import asyncio

async def get_quote(source, delay):
    await asyncio.sleep(delay)
    return f"{source}: 9.99"

# your code here
import asyncio

async def get_quote(source, delay):
    await asyncio.sleep(delay)
    return f"{source}: 9.99"

async def ask(source, delay):
    try:
        return await asyncio.wait_for(get_quote(source, delay), timeout=0.05)
    except asyncio.TimeoutError:
        return f"{source}: timed out"

async def main():
    answers = await asyncio.gather(ask("north", 0), ask("south", 10), ask("east", 0))
    for answer in answers:
        print(answer)

asyncio.run(main())

# ✅ Expected output:
# north: 9.99
# south: timed out
# east: 9.99

One wait_for() around the whole gather() would lose all three answers when south runs out of time, north and east included.

Practice quiz

load_page() sleeps 10 seconds; if it's cancelled it prints load_page cancelled and re-raises. main() prints await asyncio.wait_for(load_page(), timeout=0.01), and prints too slow on asyncio.TimeoutError. What is printed?

  • load_page cancelled, then too slow
  • too slow
  • page
  • It stops with TimeoutError

Answer: load_page cancelled, then too slow. When the deadline passes, wait_for() cancels the job first, so its handler prints, and then raises the timeout in main().

worker() loops on await asyncio.sleep(0), and its except asyncio.CancelledError: prints stopping, with no raise. main() creates the task, awaits asyncio.sleep(0) so the worker starts, cancels it, awaits it, then prints task.cancelled(). What is printed?

  • stopping, then True
  • True
  • stopping, then False
  • It stops with CancelledError

Answer: stopping, then False. The worker was already running when cancellation was requested. Its handler swallowed CancelledError, so the task ended normally and is not marked cancelled.

What does print(issubclass(asyncio.CancelledError, Exception), issubclass(asyncio.CancelledError, BaseException)) print?

  • True True
  • False True
  • False False
  • True False

Answer: False True. CancelledError is a BaseException but not an Exception, so except Exception: lets it through, while a bare except: catches it.

Ten jobs share asyncio.Semaphore(3). Each one, inside async with sem:, adds 1 to a "now" counter, records the highest value it has seen, pauses with await asyncio.sleep(0), then takes 1 away. What is the highest value printed at the end?

  • 10
  • 1
  • 7
  • 3

Answer: 3. The semaphore lets at most three jobs inside its block at once, so the counter never goes over 3.

buy(shop, name) checks shop["apples"] > 0, pauses with await asyncio.sleep(0), then takes 1 away. The shop has 1 apple, and gather() runs buy() for Ana, Ben and Cy, with no lock. What does print("apples left:", shop["apples"]) show?

  • apples left: -2
  • apples left: 0
  • apples left: 1
  • It stops with a ValueError

Answer: apples left: -2. All three check before anyone has bought, because each pauses between the check and the change, so all three take an apple.

ok() returns 1 and bad() raises KeyError("x"). main() does results = await asyncio.gather(ok(), bad(), return_exceptions=True), then print(len(results), type(results[1]).__name__). What is printed?

  • 1 KeyError
  • It stops with KeyError: 'x'
  • 2 KeyError
  • 2 NoneType

Answer: 2 KeyError. With return_exceptions=True, an error becomes that coroutine's result instead of being raised.

Three cooks, Ana, Ben and Cy, are tasks that each take an order from a shared asyncio.Queue, print their name, cooks and the dish, pause, then call task_done(). The queue holds soup, salad and pie. main() awaits orders.join(), cancels the cooks and prints all orders done. What is printed?

  • Ana cooks soup, Ana cooks salad, Ana cooks pie, all orders done
  • all orders done, Ana cooks soup, Ben cooks salad, Cy cooks pie
  • Ana cooks soup, Ben cooks salad, all orders done
  • Ana cooks soup, Ben cooks salad, Cy cooks pie, all orders done

Answer: Ana cooks soup, Ben cooks salad, Cy cooks pie, all orders done. Each cook takes one order and pauses, which lets the next cook take the next one. join() returns only after task_done() has been called for all three.

ticker() prints tick 0 and tick 1, with await asyncio.sleep(0) after each. nap() prints nap starts, calls time.sleep(0.05), then prints nap ends. What does await asyncio.gather(ticker(), nap()) print?

  • tick 0, tick 1, nap starts, nap ends
  • tick 0, nap starts, nap ends, tick 1
  • nap starts, tick 0, tick 1, nap ends
  • tick 0, nap starts, tick 1, nap ends

Answer: tick 0, nap starts, nap ends, tick 1. time.sleep() blocks the whole thread, so nothing else runs until nap() ends. asyncio.sleep() would have let the ticker go on.

Continue this course

Frequently asked questions

Do I need async versions of my libraries?

For anything that waits, yes. A call that blocks, such as requests.get() or time.sleep(), holds up every task, as the blocking-call example in this lesson shows. Use a library made for asyncio where there is one, or hand the call to asyncio.to_thread().

How do I test async functions with pytest?

pytest doesn't run async def tests on its own: pytest 9 fails one with "async def functions are not natively supported" and lists plugins to install, such as pytest-asyncio and anyio. Testing with pytest: Fixtures, Parametrisation & Mocks covers pytest.

Can I put @functools.lru_cache on an async function?

No. It caches the coroutine object the first call made, and a coroutine can be awaited only once, so awaiting the second call stops with RuntimeError: cannot reuse already awaited coroutine. functools: lru_cache, partial, reduce covers lru_cache for ordinary functions.

What does asyncio.shield() do?

It keeps a task running when whoever awaits it is cancelled: you write result = await asyncio.shield(task). It isn't a context manager, so async with asyncio.shield(task): stops with an error.

Related lessons