Advanced Async & Await
Published by Brayan K
Asynchronous programming is at the heart of modern Python applications. Whether you're building high-performance web APIs, data pipelines, websocket apps, scrapers, or microservices — mastering advanced async/await patterns gives you the ability to build systems that scale effortlessly.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
A timeout cancels the job it gives up on
This lesson builds on AsyncIO: Event Loop, Tasks & Futures: async def, await, asyncio.run(), tasks and gather(). Like that lesson, it uses await asyncio.sleep(0) where a real program would wait for a reply, so the order of the printed lines shows what runs when. Web APIs, websocket apps and scrapers have lessons of their own: Building APIs with FastAPI, Network Programming with sockets and Web Scraping with BeautifulSoup.
asyncio.wait_for() gives a coroutine a deadline in seconds. When the time is up, it doesn't just stop waiting: it cancels the job, which gets a CancelledError at the line where it's paused, waits for the job to stop, and then raises a timeout error for you. The job below would take 10 seconds; the deadline is 0.01:
import asyncio
async def slow_report():
try:
await asyncio.sleep(10)
return "report"
except asyncio.CancelledError:
print("slow_report was cancelled")
raise
async def main():
try:
print(await asyncio.wait_for(slow_report(), timeout=0.01))
except asyncio.TimeoutError:
print("gave up after 0.01 s")
asyncio.run(main())
# ✅ Expected output:
# slow_report was cancelled
# gave up after 0.01 sWithin the deadline, you get the result as usual:
import asyncio
async def quick_report():
await asyncio.sleep(0)
return "report ready"
async def main():
print(await asyncio.wait_for(quick_report(), timeout=1))
asyncio.run(main())
# ✅ Expected output:
# report readyCatch the timeout as asyncio.TimeoutError, which works on every version. From Python 3.11 on it's the built-in TimeoutError; on 3.10 it's a class of its own, so an uncaught timeout is named differently there. Python 3.11 also added asyncio.timeout(), which puts one deadline on a whole block:
import asyncio
async def main():
try:
async with asyncio.timeout(0.01):
await asyncio.sleep(10)
except TimeoutError:
print("timed out")
asyncio.run(main())
# ✅ Expected output:
# timed outCatching CancelledError doesn't make cancelling safe
task.cancel() asks a task to stop: Python raises CancelledError inside it, at the await where it's paused. A handler that catches the error and carries on refuses the request, and the task then counts as finished, not cancelled:
import asyncio
async def worker():
try:
while True:
await asyncio.sleep(0)
except asyncio.CancelledError:
print("Worker cancelled safely")
async def main():
task = asyncio.create_task(worker())
await asyncio.sleep(0)
task.cancel()
await task
print("task.cancelled():", task.cancelled())
asyncio.run(main())
# ✅ Expected output:
# Worker cancelled safely
# task.cancelled(): FalseThe same refusal breaks a deadline. wait_for() cancels the job and then waits for it to stop, so a job that catches the CancelledError and carries on keeps it waiting, and what the job returns comes back from wait_for() with no timeout error:
import asyncio
async def stubborn_report():
try:
await asyncio.sleep(10)
return "report"
except asyncio.CancelledError:
print("stubborn_report ignores the cancel")
await asyncio.sleep(0.2)
return "late report"
async def main():
try:
print(await asyncio.wait_for(stubborn_report(), timeout=0.01))
except asyncio.TimeoutError:
print("gave up after 0.01 s")
asyncio.run(main())
# ✅ Expected output:
# stubborn_report ignores the cancel
# late reportThe deadline was 0.01 seconds, but wait_for() waited another 0.2 for the handler and handed back "late report", so main()'s except never ran.
Clean up in finally instead, and let the error go on. Whoever awaits the task gets the CancelledError, and the task counts as cancelled. If you do catch it, to log something, end the handler with raise, as slow_report() does above.
import asyncio
async def worker():
try:
while True:
await asyncio.sleep(0)
finally:
print("worker cleans up")
async def main():
task = asyncio.create_task(worker())
await asyncio.sleep(0)
task.cancel()
try:
await task
except asyncio.CancelledError:
print("main: worker is cancelled")
print("task.cancelled():", task.cancelled())
asyncio.run(main())
# ✅ Expected output:
# worker cleans up
# main: worker is cancelled
# task.cancelled(): TrueAwaiting a cancelled task without catching the error stops the program with it:
import asyncio
async def job():
await asyncio.sleep(10)
async def main():
task = asyncio.create_task(job())
await asyncio.sleep(0)
task.cancel()
await task
asyncio.run(main())
# ✅ Expected error:
# asyncio.exceptions.CancelledErrorA bare except: is worse than it looks here. Exception Handling showed that KeyboardInterrupt and SystemExit aren't an Exception; neither is CancelledError, but except: with nothing after it catches it anyway:
import asyncio
print(issubclass(asyncio.CancelledError, Exception))
print(issubclass(asyncio.CancelledError, BaseException))
# ✅ Expected output:
# False
# TrueSo a retry loop with a bare except: swallows every cancel():
import asyncio
async def retry(attempts=3):
for attempt in range(1, attempts + 1):
try:
await asyncio.sleep(10)
return "ok"
except:
print("attempt", attempt, "failed, retrying")
return "gave up"
async def main():
task = asyncio.create_task(retry())
await asyncio.sleep(0)
for _ in range(3):
task.cancel()
await asyncio.sleep(0)
print(await task)
print("cancelled?", task.cancelled())
asyncio.run(main())
# ✅ Expected output:
# attempt 1 failed, retrying
# attempt 2 failed, retrying
# attempt 3 failed, retrying
# gave up
# cancelled? FalseWith except Exception:, one cancel() is enough:
import asyncio
async def retry(attempts=3):
for attempt in range(1, attempts + 1):
try:
await asyncio.sleep(10)
return "ok"
except Exception:
print("attempt", attempt, "failed, retrying")
return "gave up"
async def main():
task = asyncio.create_task(retry())
await asyncio.sleep(0)
task.cancel()
try:
await task
except asyncio.CancelledError:
print("cancelled?", task.cancelled())
asyncio.run(main())
# ✅ Expected output:
# cancelled? TrueStart everything, but limit how many run at once
Starting a thousand downloads at once can swamp the server, or your own connection. asyncio.Semaphore(n) lets at most n tasks into an async with block at a time; the others wait at the door until one leaves. Four jobs, two at a time:
import asyncio
async def job(n, sem):
async with sem:
print("start", n)
await asyncio.sleep(0)
print("end", n)
async def main():
sem = asyncio.Semaphore(2)
await asyncio.gather(*(job(n, sem) for n in range(1, 5)))
asyncio.run(main())
# ✅ Expected output:
# start 1
# start 2
# end 1
# end 2
# start 3
# start 4
# end 3
# end 4Jobs 3 and 4 start only after 1 and 2 have left the block. Writing with sem: without async is an error: a TypeError saying the Semaphore doesn't support the context manager protocol on Python 3.11 and later, and a different one on 3.10. From 3.14 the message also asks whether you meant async with.
Queues: put, get, join and task_done instead of sleeping and checking
An asyncio.Queue hands work from one task to others. await get() waits for an item, task_done() says one is finished, and join() waits until every item put in has been marked done. These cooks loop forever, so main() cancels them once the orders are done:
import asyncio
async def cook(name, orders):
while True:
dish = await orders.get()
print(name, "cooks", dish)
await asyncio.sleep(0)
orders.task_done()
async def main():
orders = asyncio.Queue()
for dish in ["soup", "salad", "pie", "tea"]:
orders.put_nowait(dish)
cooks = [asyncio.create_task(cook(name, orders)) for name in ["Ana", "Ben"]]
await orders.join()
for c in cooks:
c.cancel()
print("all orders done")
asyncio.run(main())
# ✅ Expected output:
# Ana cooks soup
# Ben cooks salad
# Ana cooks pie
# Ben cooks tea
# all orders doneA queue with a maxsize makes put() wait while it's full, so a fast producer can't run ahead of the consumer. No sleeping, no checking the size:
import asyncio
async def producer(q):
for item in ["a", "b", "c"]:
await q.put(item)
print("put", item)
await q.put(None)
async def consumer(q):
while True:
item = await q.get()
if item is None:
break
print("got", item)
async def main():
q = asyncio.Queue(maxsize=1)
await asyncio.gather(producer(q), consumer(q))
asyncio.run(main())
# ✅ Expected output:
# put a
# got a
# put b
# got b
# put c
# got cget_nowait() doesn't wait, so on an empty queue it raises QueueEmpty:
import asyncio
async def main():
q = asyncio.Queue()
q.get_nowait()
asyncio.run(main())
# ✅ Expected error:
# asyncio.queues.QueueEmptyAn asyncio.Queue is for tasks on one event loop, and takes no lock. A queue between threads is queue.Queue, in Thread-Safe Queues (queue).
One failure: gather() lets the others run on, and a TaskGroup stops them
AsyncIO: Event Loop, Tasks & Futures showed that gather() raises the first error but leaves the other tasks running. With return_exceptions=True, it raises nothing: an error becomes that coroutine's result, and you sort them out afterwards:
import asyncio
async def ok():
await asyncio.sleep(0)
return "ok"
async def bad():
raise ValueError("bad item")
async def main():
results = await asyncio.gather(ok(), bad(), return_exceptions=True)
for r in results:
if isinstance(r, Exception):
print("failed:", type(r).__name__, r)
else:
print("worked:", r)
asyncio.run(main())
# ✅ Expected output:
# worked: ok
# failed: ValueError bad itemA TaskGroup, new in Python 3.11, does the opposite: when one task fails, it cancels the others, waits for them, and raises the errors together as an ExceptionGroup. except* catches the errors of one type from it:
import asyncio
async def slow():
try:
for _ in range(3):
await asyncio.sleep(0)
print("slow finished")
except asyncio.CancelledError:
print("slow was cancelled")
raise
async def bad():
raise ValueError("bad item")
async def main():
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(slow())
tg.create_task(bad())
except* ValueError as group:
print("caught:", [str(e) for e in group.exceptions])
asyncio.run(main())
# ✅ Expected output:
# slow was cancelled
# caught: ['bad item']Uncaught, an ExceptionGroup's traceback ends with a box drawn around the errors, so its last line isn't the error itself. TaskGroup, ExceptionGroup and except* don't exist in Python 3.10.
async with and async for await inside
async with calls __aenter__ and __aexit__ and awaits them, as with calls __enter__ and __exit__ in Context Managers & the with Statement in Depth. async for asks an async generator, a function with both async def and yield (Generators & Iterators Mastery), for each value and awaits it, so other tasks can run between the values:
import asyncio
class Connection:
async def __aenter__(self):
await asyncio.sleep(0)
print("connected")
return self
async def __aexit__(self, exc_type, exc, tb):
await asyncio.sleep(0)
print("closed")
async def rows(n):
for i in range(1, n + 1):
await asyncio.sleep(0)
yield f"row {i}"
async def main():
async with Connection():
async for row in rows(3):
print(row)
asyncio.run(main())
# ✅ Expected output:
# connected
# row 1
# row 2
# row 3
# closedComprehensions work with async for too:
import asyncio
async def rows(n):
for i in range(1, n + 1):
await asyncio.sleep(0)
yield i * i
async def main():
print([x async for x in rows(4)])
asyncio.run(main())
# ✅ Expected output:
# [1, 4, 9, 16]A plain for can't loop over an async generator:
import asyncio
async def rows(n):
for i in range(n):
yield i
async def main():
for row in rows(3):
print(row)
asyncio.run(main())
# ✅ Expected error:
# TypeError: 'async_generator' object is not iterableAnd async for works only inside async def:
async def rows():
yield 1
def main():
async for r in rows():
print(r)
print("never printed")
# ✅ Expected error:
# SyntaxError: 'async for' outside async functionIn the same way, with without async on an async context manager such as Connection is an error, a TypeError on Python 3.11 and later.
A blocking call stops every task
Tasks take turns only at an await that pauses. time.sleep() doesn't pause the coroutine, it stops the whole thread, so nothing else runs while it sleeps. await asyncio.sleep() pauses only the coroutine that calls it:
import asyncio
import time
async def nap(blocking):
print("nap starts")
if blocking:
time.sleep(0.05)
else:
await asyncio.sleep(0.05)
print("nap ends")
async def ticker():
for i in range(2):
print("tick", i)
await asyncio.sleep(0)
async def main():
print("-- time.sleep")
await asyncio.gather(nap(True), ticker())
print("-- asyncio.sleep")
await asyncio.gather(nap(False), ticker())
asyncio.run(main())
# ✅ Expected output:
# -- time.sleep
# nap starts
# nap ends
# tick 0
# tick 1
# -- asyncio.sleep
# nap starts
# tick 0
# tick 1
# nap endsA long calculation with no await holds up the others in the same way, as AsyncIO: Event Loop, Tasks & Futures showed. For a blocking call you can't rewrite, such as one from a library without async support, asyncio.to_thread() runs it in a thread of its own and lets the loop carry on meanwhile:
import asyncio
import time
def read_sensor():
time.sleep(0.01)
return 21.5
async def main():
print(await asyncio.to_thread(read_sensor))
asyncio.run(main())
# ✅ Expected output:
# 21.5This site's Run button can't start threads, so there to_thread() runs the function on the page's only thread: you get the same value, but nothing else runs meanwhile. For calculations rather than waiting, Concurrency in Python: Threads vs Processes compares threads and processes.
One thread doesn't mean no races
Tasks switch only at an await, but that's enough for a race. Two buyers check the shop, then pause before buying. Both see one apple, and both take it:
import asyncio
async def buy(shop, name):
if shop["apples"] > 0:
await asyncio.sleep(0)
shop["apples"] -= 1
print(name, "bought an apple")
async def main():
shop = {"apples": 1}
await asyncio.gather(buy(shop, "Ana"), buy(shop, "Ben"))
print("apples left:", shop["apples"])
asyncio.run(main())
# ✅ Expected output:
# Ana bought an apple
# Ben bought an apple
# apples left: -1An asyncio.Lock around the check and the change makes the second buyer wait until the first has finished:
import asyncio
async def buy(shop, lock, name):
async with lock:
if shop["apples"] > 0:
await asyncio.sleep(0)
shop["apples"] -= 1
print(name, "bought an apple")
else:
print(name, "found none left")
async def main():
shop = {"apples": 1}
lock = asyncio.Lock()
await asyncio.gather(buy(shop, lock, "Ana"), buy(shop, lock, "Ben"))
print("apples left:", shop["apples"])
asyncio.run(main())
# ✅ Expected output:
# Ana bought an apple
# Ben found none left
# apples left: 0An asyncio.Lock can't be taken twice by the same task: a second await lock.acquire() before release() waits forever.
Practice: five uploads, two at a time
Each upload() counts itself in and out of the limited block, so most at once records the busiest moment. Fill the three blanks: the statement that enters the semaphore's block, the asyncio class that limits how many tasks are inside, and the function that runs all five and collects their sizes. The hints under the code give the answers, in the order the blanks appear.
import asyncio
async def upload(n, sem, stats):
___ sem:
stats["now"] += 1
stats["most"] = max(stats["most"], stats["now"])
await asyncio.sleep(0)
stats["now"] -= 1
return n * 10
async def main():
sem = asyncio.___(2)
stats = {"now": 0, "most": 0}
sizes = await asyncio.___(*(upload(n, sem, stats) for n in range(1, 6)))
print(sizes)
print("most at once:", stats["most"])
asyncio.run(main())
# 👉 replace ___ with async with
# 👉 replace ___ with Semaphore
# 👉 replace ___ with gather
# ✅ Expected output:
# [10, 20, 30, 40, 50]
# most at once: 2With Semaphore(5) instead, all five get in at once and most at once is 5. Without the * before the generator expression, gather() gets one generator instead of five coroutines and stops with an error, worded differently on Python 3.12 and later.
On your own: three quotes with a deadline each
get_quote(source, delay) waits delay seconds, then returns the source and a price. Ask three sources at the same time: north and east answer at once, and south takes 10 seconds. Give each one its own deadline of 0.05 seconds, show a source that misses it as timed out, and print the answers in the order you asked.
import asyncio
async def get_quote(source, delay):
await asyncio.sleep(delay)
return f"{source}: 9.99"
# your code hereimport asyncio
async def get_quote(source, delay):
await asyncio.sleep(delay)
return f"{source}: 9.99"
async def ask(source, delay):
try:
return await asyncio.wait_for(get_quote(source, delay), timeout=0.05)
except asyncio.TimeoutError:
return f"{source}: timed out"
async def main():
answers = await asyncio.gather(ask("north", 0), ask("south", 10), ask("east", 0))
for answer in answers:
print(answer)
asyncio.run(main())
# ✅ Expected output:
# north: 9.99
# south: timed out
# east: 9.99One wait_for() around the whole gather() would lose all three answers when south runs out of time, north and east included.
Practice quiz
load_page() sleeps 10 seconds; if it's cancelled it prints load_page cancelled and re-raises. main() prints await asyncio.wait_for(load_page(), timeout=0.01), and prints too slow on asyncio.TimeoutError. What is printed?
- load_page cancelled, then too slow
- too slow
- page
- It stops with TimeoutError
Answer: load_page cancelled, then too slow. When the deadline passes, wait_for() cancels the job first, so its handler prints, and then raises the timeout in main().
worker() loops on await asyncio.sleep(0), and its except asyncio.CancelledError: prints stopping, with no raise. main() creates the task, awaits asyncio.sleep(0) so the worker starts, cancels it, awaits it, then prints task.cancelled(). What is printed?
- stopping, then True
- True
- stopping, then False
- It stops with CancelledError
Answer: stopping, then False. The worker was already running when cancellation was requested. Its handler swallowed CancelledError, so the task ended normally and is not marked cancelled.
What does print(issubclass(asyncio.CancelledError, Exception), issubclass(asyncio.CancelledError, BaseException)) print?
- True True
- False True
- False False
- True False
Answer: False True. CancelledError is a BaseException but not an Exception, so except Exception: lets it through, while a bare except: catches it.
Ten jobs share asyncio.Semaphore(3). Each one, inside async with sem:, adds 1 to a "now" counter, records the highest value it has seen, pauses with await asyncio.sleep(0), then takes 1 away. What is the highest value printed at the end?
- 10
- 1
- 7
- 3
Answer: 3. The semaphore lets at most three jobs inside its block at once, so the counter never goes over 3.
buy(shop, name) checks shop["apples"] > 0, pauses with await asyncio.sleep(0), then takes 1 away. The shop has 1 apple, and gather() runs buy() for Ana, Ben and Cy, with no lock. What does print("apples left:", shop["apples"]) show?
- apples left: -2
- apples left: 0
- apples left: 1
- It stops with a ValueError
Answer: apples left: -2. All three check before anyone has bought, because each pauses between the check and the change, so all three take an apple.
ok() returns 1 and bad() raises KeyError("x"). main() does results = await asyncio.gather(ok(), bad(), return_exceptions=True), then print(len(results), type(results[1]).__name__). What is printed?
- 1 KeyError
- It stops with KeyError: 'x'
- 2 KeyError
- 2 NoneType
Answer: 2 KeyError. With return_exceptions=True, an error becomes that coroutine's result instead of being raised.
Three cooks, Ana, Ben and Cy, are tasks that each take an order from a shared asyncio.Queue, print their name, cooks and the dish, pause, then call task_done(). The queue holds soup, salad and pie. main() awaits orders.join(), cancels the cooks and prints all orders done. What is printed?
- Ana cooks soup, Ana cooks salad, Ana cooks pie, all orders done
- all orders done, Ana cooks soup, Ben cooks salad, Cy cooks pie
- Ana cooks soup, Ben cooks salad, all orders done
- Ana cooks soup, Ben cooks salad, Cy cooks pie, all orders done
Answer: Ana cooks soup, Ben cooks salad, Cy cooks pie, all orders done. Each cook takes one order and pauses, which lets the next cook take the next one. join() returns only after task_done() has been called for all three.
ticker() prints tick 0 and tick 1, with await asyncio.sleep(0) after each. nap() prints nap starts, calls time.sleep(0.05), then prints nap ends. What does await asyncio.gather(ticker(), nap()) print?
- tick 0, tick 1, nap starts, nap ends
- tick 0, nap starts, nap ends, tick 1
- nap starts, tick 0, tick 1, nap ends
- tick 0, nap starts, tick 1, nap ends
Answer: tick 0, nap starts, nap ends, tick 1. time.sleep() blocks the whole thread, so nothing else runs until nap() ends. asyncio.sleep() would have let the ticker go on.
Continue this course
- Previous: AsyncIO: Event Loop, Tasks & Futures
- Next: Concurrency in Python: Threads vs Processes — Understand the GIL and choose between threading and multiprocessing
- Quick reference: Python cheat sheet
Frequently asked questions
Do I need async versions of my libraries?
For anything that waits, yes. A call that blocks, such as requests.get() or time.sleep(), holds up every task, as the blocking-call example in this lesson shows. Use a library made for asyncio where there is one, or hand the call to asyncio.to_thread().
How do I test async functions with pytest?
pytest doesn't run async def tests on its own: pytest 9 fails one with "async def functions are not natively supported" and lists plugins to install, such as pytest-asyncio and anyio. Testing with pytest: Fixtures, Parametrisation & Mocks covers pytest.
Can I put @functools.lru_cache on an async function?
No. It caches the coroutine object the first call made, and a coroutine can be awaited only once, so awaiting the second call stops with RuntimeError: cannot reuse already awaited coroutine. functools: lru_cache, partial, reduce covers lru_cache for ordinary functions.
What does asyncio.shield() do?
It keeps a task running when whoever awaits it is cancelled: you write result = await asyncio.shield(task). It isn't a context manager, so async with asyncio.shield(task): stops with an error.