2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

(https://superfastpython.com/)

Python Asyncio: The Complete

Guide
OCTOBER 10, 2023 by JASON BROWNLEE (/ABOUT) in PYTHON ASYNCIO
(HTTPS://SUPERFASTPYTHON.COM/CATEGORY/ASYNCIO/)

Last Updated on November 22, 2023

Python Asyncio provides asynchronous programming with coroutines.

Asynchronous programming is a popular programming paradigm that allows a

large number of lightweight tasks to run concurrently with very little memory
overhead, compared to threads.

This makes asyncio very attractive and widely used for Python web development,
Python APIs that make web calls, and concurrency for socket programming.

This book-length guide provides a detailed and comprehensive walkthrough of

Python Asyncio.

Some tips:

1. You may want to bookmark this guide and read it over a few sittings.
2. You can download a zip (https://superfastpython.com/wp-
content/uploads/2023/11/superfastpython-asyncio-guide.zip) of all code used in
this guide.
3. You can get help, ask a question in the comments or email me
(https://superfastpython.com/contact/).
4. You can jump to the topics that interest you via the table of contents (below).
https://superfastpython.com/python-asyncio/ 1/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Let’s dive in.

PYTHON ASYNCIO: THE COMPLETE GUIDE

Download Now: Free Asyncio PDF Cheat Sheet (https://marvelous-

writer-6152.ck.page/d29b7d8dfb)

Table of Contents
1. What is Asynchronous Programming
1.1. Asynchronous Tasks
1.2. Asynchronous Programming
1.3. Asynchronous Programming in Python
2. What is Asyncio
2.1. Changes to Python to add Support for Coroutines

https://superfastpython.com/python-asyncio/ 2/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

2.2. The asyncio Module

3. When to Use Asyncio
3.1. Reasons to Use Asyncio in Python
3.2. Other Reasons to Use Asyncio
3.3. When to Not Use Asyncio
4. Coroutines in Python
4.1. What is a Coroutine
4.2. Coroutine vs Routine and Subroutine
4.3. Coroutine vs Generator
4.4. Coroutine vs Task
4.5. Coroutine vs Thread
4.6. Coroutine vs Process
4.7. When Were Coroutines Added to Python
5. Define, Create and Run Coroutines
5.1. How to Define a Coroutine
5.2. How to Create a Coroutine
5.3. How to Run a Coroutine From Python
6. What is the Event Loop
6.1. What is the Asyncio Event Loop
6.2. How To Start and Get An Event Loop
6.3. What is an Event Loop Object
6.4. Why Get Access to The Event Loop
7. Create and Run Asyncio Tasks
7.1. What is an Asyncio Task
7.2. How to Create a Task
7.3. When Does a Task Run?
8. Work With and Query Tasks
8.1. Task Life-Cycle
8.2. How to Check Task Status
8.3. How to Get Task Result
8.4. How to Get Task Exception
8.5. How to Cancel a Task
8.6. How to Use Callback With a Task
8.7. How to Set the Task Name
9. Current and Running Tasks
9.1. How to Get the Current Task
9.2. How to Get All Tasks
10. Run Many Coroutines Concurrently
10.1. What is Asyncio gather()

https://superfastpython.com/python-asyncio/ 3/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

10.2. How to use Asyncio gather()

10.3. Example of gather() For Many Coroutines in a List
11. Wait for A Collection of Tasks
11.1. What is asyncio.wait()
11.2. How to Use asyncio.wait()
11.3. Example of Waiting for All Tasks
12. Manage Multiple Coroutines in a Group
12.1. What is an asyncio.TaskGroup
12.2. How to Create an asyncio.TaskGroup
12.3. How to Create Tasks Using asyncio.TaskGroup
12.4. How to Wait on Tasks Using asyncio.TaskGroup
12.5. How to Cancel All Tasks If One Task Fails Using asyncio.TaskGroup
12.6. Example of Waiting on Multiple Tasks with a TaskGroup
13. Wait for a Coroutine with a Time Limit
13.1. What is Asyncio wait_for()
13.2. How to Use Asyncio wait_for()
13.3. Example of Asyncio wait_for() With a Timeout
14. Shield Tasks from Cancellation
14.1. What is Asyncio shield()
14.2. How to Use Asyncio shield()
14.3. Example of Asyncio shield() for a Task
15. Run a Blocking Task in Asyncio
15.1. Need to Run Blocking Tasks in Asyncio
15.2. How to Run Blocking Tasks
15.3. Example of Running I/O-Bound Task in Asyncio with to_thread()
16. Asynchronous Iterators
16.1. What Are Asynchronous Iterators
16.2. What is the “async for” loop?
16.3. How to Use Asynchronous Iterators
16.4. Example of an Asynchronous Iterator
17. Asynchronous Generators
17.1. What Are Asynchronous Generators
17.2. How to Use an Asynchronous Generator
17.3. Example of an Asynchronous Generator
18. Asynchronous Context Managers
18.1. What is an Asynchronous Context Manager
18.2. How to Use Asynchronous Context Managers
18.3. Example of an Asynchronous Context Manager and “async with”

https://superfastpython.com/python-asyncio/ 4/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

19. Asynchronous Comprehensions

19.1. What are Asynchronous Comprehensions
19.2. Comprehensions
19.3. Asynchronous Comprehensions
19.4. Await Comprehensions
20. Run Commands in Non-Blocking Subprocesses
20.1. What is asyncio.subprocess.Process
20.2. How to Run a Command Directly
20.3. How to Run a Command Via the Shell
21. Non-Blocking Streams
21.1. Asyncio Streams
21.2. How to Open a Connection
21.3. How to Start a Server
21.4. How to Write Data with the StreamWriter
21.5. How to Read Data with the StreamReader
21.6. How to Close Connection
22. Example of Checking Website Status
22.1. How to Check HTTP Status with Asyncio
22.2. Open HTTP Connection
22.3. Write HTTP Request
22.4. Read HTTP Response
22.5. Close HTTP Connection
22.6. Example of Checking HTTP Status Sequentially
22.7. Example of Checking Website Status Concurrently
23. Python Asyncio Common Errors
23.1. Error 1: Trying to Run Coroutines by Calling Them
23.2. Error 2: Not Letting Coroutines Run in the Event Loop
23.3. Error 3: Using the Low-Level Asyncio API
23.4. Error 4: Exiting the Main Coroutine Too Early
23.5. Error 5: Assuming Race Conditions and Deadlocks are Impossible
24. Python Asyncio Common Questions
24.1. How to Stop a Task?
24.2. How to Wait for a Task To Finish?
24.3. How to Get a Return Value from a Task?
24.4. How to Run a Task in the Background?
24.5. How to Wait for All Background Tasks?
24.6. Does a Running Task Stop the Event Loop from Exiting?
24.7. How to Show Progress of Running Tasks?
24.8. How to Run a Task After a Delay?

https://superfastpython.com/python-asyncio/ 5/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

24.9. How to Run a Follow-Up Task?

24.10. How to Execute a Blocking I/O or CPU-bound Function in Asyncio?
25. Common Objections to Using Asyncio
25.1. What About the Global Interpreter Lock (GIL)?
25.2. Are Python Coroutines “Real“?
25.3. Isn’t Python Concurrency Buggy?
25.4. Isn’t Python a Bad Choice for Concurrency?
25.5. Why Not Use Threads Instead?
26. Further Reading
27. Conclusions

What is Asynchronous Programming

Asynchronous programming is a programming paradigm that does not block.

Instead, requests and function calls are issued and executed somehow in the
background at some future time. This frees the caller to perform other activities and
handle the results of issued calls at a later time when results are available or when
the caller is interested.

Let’s get a handle on asynchronous programming before we dive into asyncio.

Asynchronous Tasks
Asynchronous means not at the same time, as opposed to synchronous or at the
same time.

“ asynchronous: not simultaneous or concurrent in time

— MERRIAM-WEBSTER DICTIONARY (HTTPS://WWW.MERRIAM-
WEBSTER.COM/DICTIONARY/ASYNCHRONOUS)

https://superfastpython.com/python-asyncio/ 6/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

When programming, asynchronous

(https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)) means that the
action is requested, although not performed at the time of the request. It is
performed later.

“ Asynchronous: Separate execution streams that can run concurrently in any

order relative to each other are asynchronous.

— PAGE 265, THE ART OF CONCURRENCY (HTTPS://AMZN.TO/3TKCUWX), 2009.

For example, we can make an asynchronous function call

(https://en.wikipedia.org/wiki/Asynchronous_procedure_call).

This will issue the request to make the function call and will not wait around for the
call to complete. We can choose to check on the status or result of the function call
later.

Asynchronous Function Call: Request that a function is called at some time and
in some manner, allowing the caller to resume and perform other activities.

The function call will happen somehow and at some time, in the background, and the
program can perform other tasks or respond to other events.

This is key. We don’t have control over how or when the request is handled, only that
we would like it handled while the program does other things.

Issuing an asynchronous function call often results in some handle on the request
that the caller can use to check on the status of the call or get results. This is often
called a future.

Future: A handle on an asynchronous function call allowing the status of the call
to be checked and results to be retrieved.

https://superfastpython.com/python-asyncio/ 7/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The combination of the asynchronous function call and future together is often
referred to as an asynchronous task. This is because it is more elaborate than a
function call, such as allowing the request to be canceled and more.

Asynchronous Task: Used to refer to the aggregate of an asynchronous function

call and resulting future.

Asynchronous Programming
Issuing asynchronous tasks and making asynchronous function calls is referred to as
asynchronous programming.

“ So what is asynchronous programming? It means that a particular long-

running task can be run in the background separate from the main
application. Instead of blocking all other application code waiting for that
long-running task to be completed, the system is free to do other work that
is not dependent on that task. Then, once the long-running task is
completed, we’ll be notified that it is done so we can process the result.

— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO (HTTPS://AMZN.TO/3CZ7ZH6),

2022.

Asynchronous Programming: The use of asynchronous techniques, such as

issuing asynchronous tasks or function calls.

Asynchronous programming is primarily used with non-blocking I/O, such as reading

and writing from socket connections with other processes or other systems.

https://superfastpython.com/python-asyncio/ 8/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ In non-blocking mode, when we write bytes to a socket, we can just fire and
forget the write or read, and our application can go on to perform other
tasks.

— PAGE 18, PYTHON CONCURRENCY WITH ASYNCIO (HTTPS://AMZN.TO/3CZ7ZH6),

2022.

Non-blocking I/O is a way of performing I/O where reads and writes are requested,
although performed asynchronously. The caller does not need to wait for the
operation to complete before returning.

The read and write operations are performed somehow (e.g. by the underlying
operating system or systems built upon it), and the status of the action and/or data is
retrieved by the caller later, once available, or when the caller is ready.

Non-blocking I/O: Performing I/O operations via asynchronous requests and

responses, rather than waiting for operations to complete.

As such, we can see how non-blocking I/O is related to asynchronous programming.

In fact, we use non-blocking I/O via asynchronous programming, or non-blocking I/O
is implemented via asynchronous programming.

The combination of non-blocking I/O with asynchronous programming is so common

that it is commonly referred to by the shorthand of asynchronous I/O
(https://en.wikipedia.org/wiki/Asynchronous_I/O).

Asynchronous I/O: A shorthand that refers to combining asynchronous

programming with non-blocking I/O.

Next, let’s consider asynchronous programming support in Python.

https://superfastpython.com/python-asyncio/ 9/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Asynchronous Programming in Python

Broadly, asynchronous programming in Python refers to making requests and not
blocking to wait for them to complete.

We can implement asynchronous programming in Python in various ways, although a

few are most relevant for Python concurrency.

The first and obvious example is the asyncio module

(https://docs.python.org/3/library/asyncio.html). This module directly offers an
asynchronous programming environment using the async/await syntax and non-
blocking I/O with sockets and subprocesses.

“ asyncio is short for asynchronous I/O. It is a Python library that allows us to

run code using an asynchronous programming model. This lets us handle
multiple I/O operations at once, while still allowing our application to
remain responsive.

— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO (HTTPS://AMZN.TO/3CZ7ZH6),

2022.

It is implemented using coroutines that run in an event loop that itself runs in a single
thread.

Asyncio: An asynchronous programming environment provided in Python via the

asyncio module.

More broadly, Python offers threads and processes that can execute tasks
asynchronously.

For example, one thread can start a second thread to execute a function call and
resume other activities. The operating system will schedule and execute the second
thread at some time and the first thread may or may not check on the status of the
https://superfastpython.com/python-asyncio/ 10/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

task, manually.

“ Threads are asynchronous, meaning that they may run at different speeds,
and any thread can halt for an unpredictable duration at any time.

— PAGE 76, THE ART OF MULTIPROCESSOR PROGRAMMING

(HTTPS://AMZN.TO/3CC82J2), 2020.

More concretely, Python provides executor-based thread pools and process pools in
the ThreadPoolExecutor (https://superfastpython.com/threadpoolexecutor-in-
python/) and ProcessPoolExeuctor
(https://superfastpython.com/processpoolexecutor-in-python/) classes.

These classes use the same interface and support asynchronous tasks via the
submit() method that returns a Future object.

“ The concurrent.futures module provides a high-level interface for

asynchronously executing callables. The asynchronous execution can be
performed with threads, using ThreadPoolExecutor, or separate processes,
using ProcessPoolExecutor.

— CONCURRENT.FUTURES — LAUNCHING PARALLEL TASKS

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/CONCURRENT.FUTURES.HTML)

The multiprocessing module

(https://docs.python.org/3/library/multiprocessing.html) also provides pools of
workers using processes and threads in the Pool
(https://superfastpython.com/multiprocessing-pool-python/) and ThreadPool
(https://superfastpython.com/threadpool-python/) classes, forerunners to the
ThreadPoolExecutor and ProcessPoolExeuctor classes.

https://superfastpython.com/python-asyncio/ 11/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The capabilities of these classes are described in terms of worker execution tasks
asynchronously. They explicitly provide synchronous (blocking) and asynchronous
(non-blocking) versions of each method for executing tasks.

For example, one may issue a one-off function call synchronously via the apply()
method or asynchronously via the apply_async() method.

“ A process pool object which controls a pool of worker processes to which

jobs can be submitted. It supports asynchronous results with timeouts and
callbacks and has a parallel map implementation.

— MULTIPROCESSING — PROCESS-BASED PARALLELISM

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/MULTIPROCESSING.HTML)

There are other aspects of asynchronous programming in Python that are less strictly
related to Python concurrency.

For example, Python processes receive or handle signals asynchronously. Signals are
fundamentally asynchronous events sent from other processes.

This is primarily supported by the signal module

(https://docs.python.org/3/library/signal.html).

Now that we know about asynchronous programming, let’s take a closer look at
asyncio.

Run loops using all CPUs, download your FREE book

(https://superfastpython.com/plip-incontent) to learn how.

https://superfastpython.com/python-asyncio/ 12/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

What is Asyncio
Broadly, asyncio refers to the ability to implement asynchronous programming in
Python using coroutines.

Specifically, it refers to two elements:

1. The addition of the “asyncio” module to the Python standard library in Python
3.4.
2. The addition of async/await expressions to the Python language in Python 3.5.

Together, the module and changes to the language facilitate the development of
Python programs that support coroutine-based concurrency, non-blocking I/O, and
asynchronous programming.

“ Python 3.4 introduced the asyncio library, and Python 3.5 produced the
async and await keywords to use it palatably. These new additions allow so-
called asynchronous programming.

— PAGE VII, USING ASYNCIO IN PYTHON (HTTPS://AMZN.TO/3MQC92E), 2020.

Let’s take a closer look at these two aspects of asyncio, starting with the changes to
the language.

Changes to Python to add Support for Coroutines

The Python language was changed to accommodate asyncio with the addition of
expressions and types.

More specifically, it was changed to support coroutines as first-class concepts. In turn,

coroutines are the unit of concurrency used in asyncio programs.

https://superfastpython.com/python-asyncio/ 13/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

A coroutine is a function that can be suspended and resumed.

“ coroutine: Coroutines are a more generalized form of subroutines.

Subroutines are entered at one point and exited at another point.
Coroutines can be entered, exited, and resumed at many different points.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML#TERM-

COROUTINE)

A coroutine can be defined via the “async def” expression

(https://superfastpython.com/asyncio-async-def/). It can take arguments and return a
value, just like a function.

For example:

1 # define a coroutine
2 async def custom_coro():
3 # ...

Calling a coroutine function will create a coroutine object, this is a new class. It does
not execute the coroutine function.

1 ...
2 # create a coroutine object
3 coro = custom_coro()

A coroutine can execute another coroutine via the await expression.

This suspends the caller and schedules the target for execution.

1 ...
2 # suspend and schedule the target
3 await custom_coro()

An asynchronous iterator is an iterator that yields awaitables.

https://superfastpython.com/python-asyncio/ 14/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ asynchronous iterator: An object that implements the __aiter__() and

__anext__() methods. __anext__ must return an awaitable object. async for
resolves the awaitables returned by an asynchronous iterator’s __anext__()
method until it raises a StopAsyncIteration exception.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

An asynchronous iterator can be traversed using the “async for” expression

(https://superfastpython.com/asyncio-async-for/).

1 ...
2 # traverse an asynchronous iterator
3 async for item in async_iterator:
4 print(item)

This does not execute the for-loop in parallel.

Instead, the calling coroutine that executes the for loop will suspend and internally
await each awaitable yielded from the iterator.

An asynchronous context manager is a context manager that can await the enter and
exit methods.

“ An asynchronous context manager is a context manager that is able to

suspend execution in its enter and exit methods.

— ASYNCHRONOUS CONTEXT MANAGERS AND “ASYNC WITH”

(HTTPS://PEPS.PYTHON.ORG/PEP-0492/#ASYNCHRONOUS-CONTEXT-MANAGERS -
AND-ASYNC-WITH)

The “async with” expression (https://superfastpython.com/asyncio-async-with/) is for

creating and using asynchronous context managers.

The calling coroutine will suspend and await the context manager before entering the
block for the context manager, and similarly when leaving the context manager block.
https://superfastpython.com/python-asyncio/ 15/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

These are the sum of the major changes to Python language to support coroutines.

Next, let’s look at the asyncio module.

The asyncio Module

The “asyncio” module provides functions and objects for developing coroutine-based
programs using the asynchronous programming paradigm.

Specifically, it supports non-blocking I/O with subprocesses (for executing commands)

and with streams (for TCP socket programming).

“ asyncio is a library to write concurrent code using the async/await syntax.

— ASYNCIO — ASYNCHRONOUS I/O
(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO.HTML)

Central to the asyncio module is the event loop.

This is the mechanism that runs a coroutine-based program and implements

cooperative multitasking between coroutines.

“ The event loop is the core of every asyncio application. Event loops run
asynchronous tasks and callbacks, perform network IO operations, and run
subprocesses.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

EVENTLOOP.HTML)

The module provides both a high-level and low-level API.

https://superfastpython.com/python-asyncio/ 16/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The high-level API is for us Python application developers. The low-level API is for
framework developers, not us, in most cases.

Most use cases are satisfied using the high-level API that provides utilities for working
with coroutines, streams, synchronization primitives, subprocesses, and queues for
sharing data between coroutines.

The lower-level API provides the foundation for the high-level API and includes the
internals of the event loop, transport protocols, policies, and more.

“ … there are low-level APIs for library and framework developers

— ASYNCIO — ASYNCHRONOUS I/O
(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO.HTML)

Now that we know what asyncio is, broadly, and that it is for Asynchronous
programming.

Next, let’s explore when we should consider using asyncio in our Python programs.

Download Now: Free Asyncio PDF Cheat Sheet (https://marvelous-

writer-6152.ck.page/d29b7d8dfb)

When to Use Asyncio

Asyncio, broadly, is new, popular, much discussed, and exciting.

Nevertheless, there is a lot of confusion over when it should be adopted in a project.

When should we use asyncio in Python?

https://superfastpython.com/python-asyncio/ 17/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Reasons to Use Asyncio in Python

There are perhaps 3 top-level reasons to use asyncio in a Python project.

They are:

1. Use asyncio in order to adopt coroutines in your program.

2. Use asyncio in order to use the asynchronous programming paradigm.
3. Use asyncio in order to use non-blocking I/O.

Reason 1: To Use Coroutines

We may choose to use asyncio because we want to use coroutines.

We may want to use coroutines because we can have many more concurrent
coroutines in our program than concurrent threads.

Coroutines are another unit of concurrency, like threads and processes.

Thread-based concurrency is provided by the threading module and is supported by

the underlying operating system. It is suited to blocking I/O tasks such reading and
writing from files, sockets, and devices.

Process-based concurrency is provided by the multiprocessing module and is also

supported by the underlying operating system, like threads. It is suited to CPU-bound
tasks that do not require much inter-process communication, such as compute tasks.

Coroutines are an alternative that is provided by the Python language and runtime
(standard interpreter) and further supported by the asyncio module. They are suited
to non-blocking I/O with subprocesses and sockets, however, blocking I/O and CPU-
bound tasks can be used in a simulated non-blocking manner using threads and
processes under the covers.

https://superfastpython.com/python-asyncio/ 18/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This last point is subtle and key. Although we can choose to use coroutines for the
capability for which they were introduced into Python, non-blocking, we may in fact
use them with any tasks. Any program written with threads or processes can be
rewritten or instead written using coroutines if we so desire.

Threads and processes achieve multitasking via the operating system that chooses
which threads and processes should run, when, and for how long. The operating
switches between threads and processes rapidly, suspending those that are not
running and resuming those granted time to run. This is called preemptive
multitasking.

Coroutines in Python provide an alternative type of multitasking called cooperating

multitasking.

A coroutine is a subroutine (function) that can be suspended and resumed. It is

suspended by the await expression and resumed once the await expression is
resolved.

This allows coroutines to cooperate by design, choosing how and when to suspend
their execution.

It is an alternate, interesting, and powerful approach to concurrency, different from

thread-based and process-based concurrency.

This alone may make it a reason to adopt it for a project.

Another key aspect of coroutines is that they are lightweight.

They are more lightweight than threads. This means they are faster to start and use
less memory. Essentially a coroutine is a special type of function, whereas a thread is
represented by a Python object and is associated with a thread in the operating
system with which the object must interact.

https://superfastpython.com/python-asyncio/ 19/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

As such, we may have thousands of threads in a Python program, but we could easily
have tens or hundreds of thousands of coroutines all in one thread.

We may choose coroutines for their scalability.

Reason 2: To Use Asynchronous Programming

We may choose to use asyncio because we want to use asynchronous programming
in our program.

That is, we want to develop a Python program that uses the asynchronous
programming paradigm.

Asynchronous means not at the same time, as opposed to synchronous or at the

same time.

When programming, asynchronous means that the action is requested, although not
performed at the time of the request. It is performed later.

Asynchronous programming often means going all in and designing the program
around the concept of asynchronous function calls and tasks.

Although there are other ways to achieve elements of asynchronous programming,

full asynchronous programming in Python requires the use of coroutines and the
asyncio module.

“ It is a Python library that allows us to run code using an asynchronous

programming model.

— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO (HTTPS://AMZN.TO/3CZ7ZH6),

2022.

https://superfastpython.com/python-asyncio/ 20/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We may choose to use asyncio because we want to use the asynchronous

programming module in our program, and that is a defensible reason.

To be crystal clear, this reason is independent of using non-blocking I/O.

Asynchronous programming can be used independently of non-blocking I/O.

As we saw previously, coroutines can execute non-blocking I/O asynchronously, but

the asyncio module also provides the facility for executing blocking I/O and CPU-
bound tasks in an asynchronous manner, simulating non-blocking under the covers
via threads and processes.

Reason 3: To Use Non-Blocking I/O

We may choose to use asyncio because we want or require non-blocking I/O in our
program.

Input/Output or I/O for short means reading or writing from a resource.

Common examples include:

Hard disk drives: Reading, writing, appending, renaming, deleting, etc. files.
Peripherals: mouse, keyboard, screen, printer, serial, camera, etc.
Internet: Downloading and uploading files, getting a webpage, querying RSS, etc.
Database: Select, update, delete, etc. SQL queries.
Email: Send mail, receive mail, query inbox, etc.

These operations are slow, compared to calculating things with the CPU.

The common way these operations are implemented in programs is to make the read
or write request and then wait for the data to be sent or received.

As such, these operations are commonly referred to as blocking I/O tasks.

https://superfastpython.com/python-asyncio/ 21/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The operating system can see that the calling thread is blocked and will context
switch to another thread that will make use of the CPU.

This means that the blocking call does not slow down the entire system. But it does
halt or block the thread or program making the blocking call.

You can learn more about blocking calls in the tutorial:

Thread Blocking Call in Python (https://superfastpython.com/thread-blocking-call-

in-python/)

Non-blocking I/O is an alternative to blocking I/O.

It requires support in the underlying operating system, just like blocking I/O, and all
modern operating systems provide support for some form of non-blocking I/O.

Non-blocking I/O allows read and write calls to be made as asynchronous requests.

The operating system will handle the request and notify the calling program when the
results are available.

Non-blocking I/O: Performing I/O operations via asynchronous requests and

responses, rather than waiting for operations to complete.

As such, we can see how non-blocking I/O is related to asynchronous programming.

In fact, we use non-blocking I/O via asynchronous programming, or non-blocking I/O
is implemented via asynchronous programming.

The combination of non-blocking I/O with asynchronous programming is so common

that it is commonly referred to by the shorthand of asynchronous I/O
(https://en.wikipedia.org/wiki/Asynchronous_I/O).

Asynchronous I/O: A shorthand that refers to combining asynchronous

programming with non-blocking I/O.
https://superfastpython.com/python-asyncio/ 22/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The asyncio module in Python was added specifically to add support for non-blocking
I/O with subprocesses (e.g. executing commands on the operating system) and with
streams (e.g. TCP socket programming) to the Python standard library.

We could simulate non-blocking I/O using threads and the asynchronous

programming capability provided by Python thread pools or thread pool executors.

The asyncio module provides first-class asynchronous programming for non-blocking

I/O via coroutines, event loops, and objects to represent non-blocking subprocesses
and streams.

We may choose to use asyncio because we want to use asynchronous I/O in our
program, and that is a defensible reason.

Other Reasons to Use Asyncio

Ideally, we would choose a reason that is defended in the context of the
requirements of the project.

Sometimes we have control over the function and non-functional requirements and
other times not. In the cases we do, we may choose to use asyncio for one of the
reasons listed above. In the cases we don’t, we may be led to choose asyncio in order
to deliver a program that solves a specific problem.

Some other reasons we may use asyncio include:

1. Use asyncio because someone else made the decision for you.
2. Use asyncio because the project you have joined is already using it.
3. Use asyncio because you want to learn more about it.

We don’t always have full control over the projects we work on.

It is common to start a new job, new role, or new project and be told by the line
manager or lead architect of specific design and technology decisions.
https://superfastpython.com/python-asyncio/ 23/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Using asyncio may be one of these decisions.

We may use asyncio on a project because the project is already using it. You must use
asyncio, rather than you choose to use asyncio.

A related example might be the case of a solution to a problem that uses asyncio that
you wish to adopt.

For example:

Perhaps you need to use a third-party API and the code examples use asyncio.
Perhaps you need to integrate an existing open-source solution that uses
asyncio.
Perhaps you stumble across some code snippets that do what you need, yet they
use asyncio.

For lack of alternate solutions, asyncio may be thrust upon you by your choice of
solution.

Finally, we may choose asyncio for our Python project to learn more about.

You may scoff, “what about the requirements?”

You may choose to adopt asyncio just because you want to try it out and it can be a
defensible reason.

Using asyncio in a project will make its workings concrete for you.

When to Not Use Asyncio

We have spent a lot of time on reasons why we should use asyncio.

It is probably a good idea to spend at least a moment on why we should not use it.

https://superfastpython.com/python-asyncio/ 24/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

One reason to not use asyncio is that you cannot defend its use using one of the
reasons above.

This is not foolproof. There may be other reasons to use it, not listed above.

But, if you pick a reason to use asyncio and the reason feels thin or full of holes for
your specific case. Perhaps asyncio is not the right solution.

I think the major reason to not use asyncio is that it does not deliver the benefit that
you think it does.

There are many misconceptions about Python concurrency, especially around

asyncio.

For example:

Asyncio will work around the global interpreter lock.

Asyncio is faster than threads.
Asyncio avoids the need for mutex locks and other synchronization primitives.
Asyncio is easier to use than threads.

These are all false.

Only a single coroutine can run at a time by design, they cooperate to execute. This is
just like threads under the GIL. In fact, the GIL is an orthogonal concern and probably
irrelevant in most cases when using asyncio.

Any program you can write with asyncio, you can write with threads and it will
probably be as fast or faster. It will also probably be simpler and easier to read and
interpret by fellow developers.

Any concurrency failure mode you might expect with threads, you can encounter with
coroutines. You must make coroutines safe from deadlocks and race conditions, just
like threads.
https://superfastpython.com/python-asyncio/ 25/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Another reason to not use asyncio is that you don’t like asynchronous programming.

Asynchronous programming has been popular for some time now in a number of
different programming communities, most notably the JavaScript community.

It is different from procedural, object-oriented, and functional programming, and

some developers just don’t like it.

No problem. If you don’t like it, don’t use it. It’s a fair reason.

You can achieve the same effect in many ways, notably by sprinkling a few
asynchronous calls in via thread or process executors as needed.

Now that we are familiar with when to use asyncio, let’s look at coroutines in more
detail.

Free Python Asyncio Course

Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-
day crash course on the Asyncio API.

Discover how to use the Python asyncio module including how to define, create, and
run new coroutines and how to use non-blocking I/O.

Learn more (https://marvelous-writer-6152.ck.page/d29b7d8dfb)

Coroutines in Python
Python provides first-class coroutines with a “coroutine” type and new expressions
like “async def” and “await“.
https://superfastpython.com/python-asyncio/ 26/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

It provides the “asyncio” module for running coroutines and developing

asynchronous programs.

In this section, we will take a much closer look at coroutines.

What is a Coroutine
A coroutine is a function (https://en.wikipedia.org/wiki/Coroutine) that can be
suspended and resumed.

It is often defined as a generalized subroutine.

A subroutine can be executed, starting at one point and finishing at another point.
Whereas, a coroutine can be executed then suspended, and resumed many times
before finally terminating.

Specifically, coroutines have control over when exactly they suspend their execution.

This may involve the use of a specific expression, such as an “await” expression in
Python, like a yield expression in a Python generator.

“ A coroutine is a method that can be paused when we have a potentially

long-running task and then resumed when that task is finished. In Python
version 3.5, the language implemented first-class support for coroutines and
asynchronous programming when the keywords async and await were
explicitly added to the language.

— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO (HTTPS://AMZN.TO/3ENILNS),

2022.

A coroutine may suspend for many reasons, such as executing another coroutine, e.g.
awaiting another task, or waiting for some external resources, such as a socket
connection or process to return data.
https://superfastpython.com/python-asyncio/ 27/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Coroutines are used for concurrency.

“ Coroutines let you have a very large number of seemingly simultaneous

functions in your Python programs.

— PAGE 267, EFFECTIVE PYTHON (HTTPS://AMZN.TO/3ECLKFE), 2019.

Many coroutines can be created and executed at the same time. They have control
over when they will suspend and resume, allowing them to cooperate as to when
concurrent tasks are executed.

This is called cooperative multitasking

(https://en.wikipedia.org/wiki/Cooperative_multitasking) and is different from the
multitasking typically used with threads called preemptive multitasking tasking.

“ … in order to run multiple applications concurrently, processes voluntarily

yield control periodically or when idle or logically blocked. This type of
multitasking is called cooperative because all programs must cooperate for
the scheduling scheme to work.

— COOPERATIVE MULTITASKING, WIKIPEDIA

(HTTPS://EN.WIKIPEDIA.ORG/WIKI/COOPERATIVE_MULTITASKING)

Preemptive multitasking involves the operating system choosing what threads to

suspend and resume and when to do so, as opposed to the tasks themselves
deciding in the case of cooperative multitasking.

Now that we have some idea of what a coroutine is, let’s deepen this understanding
by comparing them to other familiar programming constructs.

https://superfastpython.com/python-asyncio/ 28/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Coroutine vs Routine and Subroutine

A “routine” and “subroutine” often refer to the same thing in modern programming.

Perhaps more correctly, a routine is a program, whereas a subroutine is a function

(https://en.wikipedia.org/wiki/Function_(computer_programming)) in the program.

A routine has subroutines.

It is a discrete module of expressions that is assigned a name, may take arguments

and may return a value.

Subroutine: A module of instructions that can be executed on demand, typically

named, and may take arguments and return a value. also called a function

A subroutine is executed, runs through the expressions, and returns somehow.

Typically, a subroutine is called by another subroutine.

A coroutine is an extension of a subroutine. This means that a subroutine is a special

type of a coroutine.

A coroutine is like a subroutine in many ways, such as:

They both are discrete named modules of expressions.

They both can take arguments, or not.
They both can return a value, or not.

The main difference is that it chooses to suspend and resume its execution many
times before returning and exiting.

Both coroutines and subroutines can call other examples of themselves. A subroutine
can call other subroutines. A coroutine executes other coroutines. However, a
coroutine can also execute other subroutines.

https://superfastpython.com/python-asyncio/ 29/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

When a coroutine executes another coroutine, it must suspend its execution and
allow the other coroutine to resume once the other coroutine has completed.

This is like a subroutine calling another subroutine. The difference is the suspension
of the coroutine may allow any number of other coroutines to run as well.

This makes a coroutine calling another coroutine more powerful than a subroutine
calling another subroutine. It is central to the cooperating multitasking facilitated by
coroutines.

Coroutine vs Generator
A generator is a special function that can suspend its execution.

“ generator: A function which returns a generator iterator. It looks like a

normal function except that it contains yield expressions for producing a
series of values usable in a for-loop or that can be retrieved one at a time
with the next() function.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

A generator function can be defined like a normal function although it uses a yield
expression at the point it will suspend its execution and return a value.

A generator function will return a generator iterator object that can be traversed,
such as via a for-loop. Each time the generator is executed, it runs from the last point
it was suspended to the next yield statement.

https://superfastpython.com/python-asyncio/ 30/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ generator iterator: An object created by a generator function. Each yield

temporarily suspends processing, remembering the location execution state
(including local variables and pending try-statements). When the generator
iterator resumes, it picks up where it left off (in contrast to functions which
start fresh on every invocation).

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

A coroutine can suspend or yield to another coroutine using an “await” expression. It

will then resume from this point once the awaited coroutine has been completed.

“ Using this paradigm, an await statement is similar in function to a yield

statement; the execution of the current function gets paused while other
code is run. Once the await or yield resolves with data, the function is
resumed.

— PAGE 218, HIGH PERFORMANCE PYTHON (HTTPS://AMZN.TO/3RY7CQE), 2020.

We might think of a generator as a special type of coroutine and cooperative

multitasking used in loops.

“ Generators, also known as semicoroutines, are a subset of coroutines.

— COROUTINE, WIKIPEDIA (HTTPS://EN.WIKIPEDIA.ORG/WIKI/COROUTINE).

Before coroutines were developed, generators were extended so that they might be
used like coroutines in Python programs.

This required a lot of technical knowledge of generators and the development of

custom task schedulers.

https://superfastpython.com/python-asyncio/ 31/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ To implement your own concurrency using generators, you first need a

fundamental insight concerning generator functions and the yield statement.
Specifically, the fundamental behavior of yield is that it causes a generator
to suspend its execution. By suspending execution, it is possible to write a
scheduler that treats generators as a kind of “task” and alternates their
execution using a kind of cooperative task switching.

— PAGE 524, PYTHON COOKBOOK (HTTPS://AMZN.TO/3D002LI), 2013.

This was made possible via changes to the generators and the introduction of the
“yield from” expression.

These were later deprecated in favor of the modern async/await expressions.

Coroutine vs Task
A subroutine and a coroutine may represent a “task” in a program.

However, in Python, there is a specific object called an asyncio.Task object

(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task).

“ A Future-like object that runs a Python coroutine. […] Tasks are used to run
coroutines in event loops.

— ASYNCIO TASK OBJECT (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

TASK.HTML#ASYNCIO.TASK)

A coroutine can be wrapped in an asyncio.Task object and executed independently,

as opposed to being executed directly within a coroutine. The Task object provides a
handle on the asynchronously execute coroutine.

Task: A wrapped coroutine that can be executed independently.

https://superfastpython.com/python-asyncio/ 32/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This allows the wrapped coroutine to execute in the background. The calling
coroutine can continue executing instructions rather than awaiting another
coroutine.

A Task cannot exist on its own, it must wrap a coroutine.

Therefore a Task is a coroutine, but a coroutine is not a task.

You can learn more about asyncio.Task objects in the tutorial:

Coroutine vs Thread
A coroutine is more lightweight than a thread.

Thread: heavyweight compared to a coroutine

Coroutine: lightweight compared to a thread.

A coroutine is defined as a function.

A thread is an object created and managed by the underlying operating system and
represented in Python as a threading.Thread object.

Thread: Managed by the operating system, represented by a Python object.

This means that coroutines are typically faster to create and start executing and take
up less memory. Conversely, threads are slower than coroutines to create and start
and take up more memory.

“ The cost of starting a coroutine is a function call. Once a coroutine is active,

it uses less than 1 KB of memory until it’s exhausted.

— PAGE 267, EFFECTIVE PYTHON (HTTPS://AMZN.TO/3ECLKFE), 2019.

https://superfastpython.com/python-asyncio/ 33/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Coroutines execute within one thread, therefore a single thread may execute many
coroutines.

“ Many separate async functions advanced in lockstep all seem to run

simultaneously, mimicking the concurrent behavior of Python threads.
However, coroutines do this without the memory overhead, startup and
context switching costs, or complex locking and synchronization code that’s
required for threads.

— PAGE 267, EFFECTIVE PYTHON (HTTPS://AMZN.TO/3ECLKFE), 2019.

You can learn more about threads in the guide:

Python Threading: The Complete Guide (https://superfastpython.com/threading-

in-python/)

Coroutine vs Process
A coroutine is more lightweight than a process.

In fact, a thread is more lightweight than a process.

A process is a computer program. It may have one or many threads.

A Python process is in fact a separate instance of the Python interpreter.

Processes, like threads, are created and managed by the underlying operating system
and are represented by a multiprocessing.Process object.

Process: Managed by the operating system, represented by a Python object.

This means that coroutines are significantly faster than a process to create and start
and take up much less memory.
https://superfastpython.com/python-asyncio/ 34/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

A coroutine is just a special function, whereas a Process is an instance of the

interpreter that has at least one thread.

You can learn more about Python processes in the guide:

Python Multiprocessing: The Complete Guide

(https://superfastpython.com/multiprocessing-in-python/)

When Were Coroutines Added to Python

Coroutines extend generators in Python.

Generators have slowly been migrating towards becoming first-class coroutines for a
long time.

We can explore some of the major changes to Python to add coroutines, which we
might consider a subset of the probability addition of asyncio.

New methods like send() and close() were added to generator objects to allow them
to act more like coroutines.

These were added in Python 2.5 and described in PEP 342

(https://peps.python.org/pep-0342/).

“ This PEP proposes some enhancements to the API and syntax of generators,
to make them usable as simple coroutines.

— PEP 342 – COROUTINES VIA ENHANCED GENERATORS

(HTTPS://PEPS.PYTHON.ORG/PEP-0342/)

Later, allowing generators to emit a suspension exception as well as a stop exception

described in PEP 334 (https://peps.python.org/pep-0334/).

https://superfastpython.com/python-asyncio/ 35/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ This PEP proposes a limited approach to coroutines based on an extension

to the iterator protocol. Currently, an iterator may raise a StopIteration
exception to indicate that it is done producing values. This proposal adds
another exception to this protocol, SuspendIteration, which indicates that
the given iterator may have more values to produce, but is unable to do so
at this time.

— PEP 334 – SIMPLE COROUTINES VIA SUSPENDITERATION

(HTTPS://PEPS.PYTHON.ORG/PEP-0334/)

The vast majority of the capabilities for working with modern coroutines in Python via
the asyncio module were described in PEP 3156 (https://peps.python.org/pep-3156/),
added in Python 3.3.

“ This is a proposal for asynchronous I/O in Python 3, starting at Python 3.3.

Consider this the concrete proposal that is missing from PEP 3153. The
proposal includes a pluggable event loop, transport and protocol
abstractions similar to those in Twisted, and a higher-level scheduler based
on yield from (PEP 380). The proposed package name is asyncio.

— PEP 3156 – ASYNCHRONOUS IO SUPPORT REBOOTED: THE “ASYNCIO” MODULE

(HTTPS://PEPS.PYTHON.ORG/PEP-3156/)

A second approach to coroutines, based on generators, was added to Python 3.4

(https://docs.python.org/3.4/library/asyncio-task.html) as an extension to Python
generators.

A coroutine was defined as a function that used the @asyncio.coroutine decorator.

Coroutines were executed using an asyncio event loop, via the asyncio module.

https://superfastpython.com/python-asyncio/ 36/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

A coroutine could suspend and execute another coroutine via the “yield from”
expression

For example:

1 # define a custom coroutine in Python 3.4

2 @asyncio.coroutine
3 def custom_coro():
4 # suspend and execute another coroutine
5 yield from asyncio.sleep(1)

The “yield from” expression was defined in PEP 380 (https://peps.python.org/pep-

0380/).

“ A syntax is proposed for a generator to delegate part of its operations to

another generator. This allows a section of code containing ‘yield’ to be
factored out and placed in another generator.

— PEP 380 – SYNTAX FOR DELEGATING TO A SUBGENERATOR

(HTTPS://PEPS.PYTHON.ORG/PEP-0380/)

The “yield from” expression

(https://docs.python.org/3/reference/expressions.html#yield-expressions) is still
available for use in generators, although is a deprecated approach to suspending
execution in coroutines, in favor of the “await” expression
(https://docs.python.org/3/reference/expressions.html#await-expression).

“ Note: Support for generator-based coroutines is deprecated and is removed

in Python 3.11. Generator-based coroutines predate async/await syntax.
They are Python generators that use yield from expressions to await on
Futures and other coroutines.

— ASYNCIO COROUTINES AND TASKS

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-TASK.HTML)

https://superfastpython.com/python-asyncio/ 37/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We might say that coroutines were added as first-class objects to Python in version
3.5.

This included changes to the Python language, such as the “async def“, “await“,
“async with“, and “async for” expressions, as well as a coroutine type.

These changes were described in PEP 492 (https://peps.python.org/pep-0492/).

“ It is proposed to make coroutines a proper standalone concept in Python,

and introduce new supporting syntax. The ultimate goal is to help establish
a common, easily approachable, mental model of asynchronous
programming in Python and make it as close to synchronous programming
as possible.

— PEP 492 – COROUTINES WITH ASYNC AND AWAIT SYNTAX

(HTTPS://PEPS.PYTHON.ORG/PEP-0492/)

Now that we know what a coroutine is, let’s take a closer look at how to use them in
Python.

Overwhelmed by the python concurrency APIs?

Find relief, download my FREE Python Concurrency Mind Maps (https://marvelous-
writer-6152.ck.page/8f23adb076)

Define, Create and Run Coroutines

We can define coroutines in our Python programs, just like defining new subroutines
(functions).

Once defined, a coroutine function can be used to create a coroutine object.

https://superfastpython.com/python-asyncio/ 38/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The “asyncio” module provides tools to run our coroutine objects in an event loop,
which is a runtime for coroutines.

How to Define a Coroutine

A coroutine can be defined via the “async def” expression.

This is an extension of the “def” expression for defining subroutines.

It defines a coroutine that can be created and returns a coroutine object.

For example:

1 # define a coroutine
2 async def custom_coro():
3 # ...

A coroutine defined with the “async def” expression is referred to as a “coroutine

function“.

“ coroutine function: A function which returns a coroutine object. A coroutine

function may be defined with the async def statement, and may contain
await, async for, and async with keywords.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

A coroutine can then use coroutine-specific expressions within it, such as await,
async for, and async with.

https://superfastpython.com/python-asyncio/ 39/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Execution of Python coroutines can be suspended and resumed at many

points (see coroutine). await expressions, async for and async with can only
be used in the body of a coroutine function.

— COROUTINE FUNCTION DEFINITION

(HTTPS://DOCS.PYTHON.ORG/3/REFERENCE/COMPOUND_STMTS.HTML#ASYNC-DEF)

For example:

1 # define a coroutine
2 async def custom_coro():
3 # await another coroutine
4 await asyncio.sleep(1)

How to Create a Coroutine

Once a coroutine is defined, it can be created.

This looks like calling a subroutine.

For example:

1 ...
2 # create a coroutine
3 coro = custom_coro()

This does not execute the coroutine.

It returns a “coroutine” object

(https://docs.python.org/3/reference/datamodel.html#coroutines).

https://superfastpython.com/python-asyncio/ 40/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ You can think of a coroutine function as a factory for coroutine objects;

more directly, remember that calling a coroutine function does not cause
any user-written code to execute, but rather just builds and returns a
coroutine object.

— PAGE 516, PYTHON IN A NUTSHELL (HTTPS://AMZN.TO/3TAZSBW), 2017.

A “coroutine” Python object has methods, such as send() and close(). It is a type.

We can demonstrate this by creating an instance of a coroutine and calling the type()
built-in function in order to report its type.

For example:

1 # SuperFastPython.com
2 # check the type of a coroutine
3
4 # define a coroutine
5 async def custom_coro():
6 # await another coroutine
7 await asyncio.sleep(1)
8
9 # create the coroutine
10 coro = custom_coro()
11 # check the type of the coroutine
12 print(type(coro))

Running the example reports that the created coroutine is a “coroutine” class.

We also get a RuntimeError because the coroutine was created but never executed,
we will explore that in the next section.

1 <class 'coroutine'>
2 sys:1: RuntimeWarning: coroutine 'custom_coro' was never awaited

A coroutine object is an awaitable.

This means it is a Python type that implements the __await__() method.

https://superfastpython.com/python-asyncio/ 41/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ An awaitable object generally implements an __await__() method. Coroutine

objects returned from async def functions are awaitable.

— AWAITABLE OBJECTS
(HTTPS://DOCS.PYTHON.ORG/3/REFERENCE/DATAMODEL.HTML#COROUTINES)

You can learn more about awaitables in the tutorial:

What is an Asyncio Awaitable in Python (/asyncio-awaitable)

How to Run a Coroutine From Python

Coroutines can be defined and created, but they can only be executed within an
event loop.

“ The event loop is the core of every asyncio application. Event loops run
asynchronous tasks and callbacks, perform network IO operations, and run
subprocesses.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

EVENTLOOP.HTML)

The event loop that executes coroutines, manages the cooperative multitasking
between coroutines.

“ Coroutine objects can only run when the event loop is running.
— PAGE 517, PYTHON IN A NUTSHELL (HTTPS://AMZN.TO/3TAZSBW), 2017.

The typical way to start a coroutine event loop is via the asyncio.run() function
(https://docs.python.org/3/library/asyncio-task.html#asyncio.run).
https://superfastpython.com/python-asyncio/ 42/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This function takes one coroutine and returns the value of the coroutine. The
provided coroutine can be used as the entry point into the coroutine-based program.

For example:

1 # SuperFastPython.com
2 # example of running a coroutine
3 import asyncio
4 # define a coroutine
5 async def custom_coro():
6 # await another coroutine
7 await asyncio.sleep(1)
8
9 # main coroutine
10 async def main():
11 # execute my custom coroutine
12 await custom_coro()
13
14 # start the coroutine program
15 asyncio.run(main())

Now that we know how to define, create, and run a coroutine, let’s take a moment to
understand the event loop.

(https://superfastpython.com/paj-incontent)
Loving The Tutorials?
Why not take the next step? Get the book.

Learn more (https://superfastpython.com/paj-incontent)

What is the Event Loop

The heart of asyncio programs is the event loop.

In this section, we will take a moment to look at the asyncio event loop.

https://superfastpython.com/python-asyncio/ 43/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

What is the Asyncio Event Loop

The event loop is an environment for executing coroutines in a single thread.

“ asyncio is a library to execute these coroutines in an asynchronous fashion

using a concurrency model known as a single-threaded event loop.

— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO (HTTPS://AMZN.TO/3VVAQ59),

2022.

The event loop is the core of an asyncio program.

It does many things, such as:

1. Execute coroutines.
2. Execute callbacks.
3. Perform network input/output.
4. Run subprocesses.

“ The event loop is the core of every asyncio application. Event loops run
asynchronous tasks and callbacks, perform network IO operations, and run
subprocesses.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

EVENTLOOP.HTML)

Event loops are a common design pattern and became very popular in recent times
given their use in JavaScript.

https://superfastpython.com/python-asyncio/ 44/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ JavaScript has a runtime model based on an event loop, which is responsible

for executing the code, collecting and processing events, and executing
queued sub-tasks. This model is quite different from models in other
languages like C and Java.

— THE EVENT LOOP, MOZILLA (HTTPS://DEVELOPER.MOZILLA.ORG/EN-

US/DOCS/WEB/JAVASCRIPT/EVENTLOOP).

The event loop, as its name suggests, is a loop. It manages a list of tasks (coroutines)
and attempts to progress each in sequence in each iteration of the loop, as well as
perform other tasks like executing callbacks and handling I/O.

The “asyncio” module provides functions for accessing and interacting with the event
loop.

This is not required for typical application development.

Instead, access to the event loop is provided for framework developers, those that
want to build on top of the asyncio module or enable asyncio for their library.

“ Application developers should typically use the high-level asyncio functions,

such as asyncio.run(), and should rarely need to reference the loop object or
call its methods.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

EVENTLOOP.HTML)

The asyncio module provides a low-level API for getting access to the current event
loop object, as well as a suite of methods that can be used to interact with the event
loop.

https://superfastpython.com/python-asyncio/ 45/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The low-level API is intended for framework developers that will extend, complement
and integrate asyncio into third-party libraries.

We rarely need to interact with the event loop in asyncio programs, in favor of using
the high-level API instead.

Nevertheless, we can briefly explore how to get the event loop.

How To Start and Get An Event Loop

The typical way we create an event loop in asyncio applications is via the
asyncio.run() function.

“ This function always creates a new event loop and closes it at the end. It
should be used as a main entry point for asyncio programs, and should
ideally only be called once.

— ASYNCIO COROUTINES AND TASKS

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-TASK.HTML)

The function takes a coroutine and will execute it to completion.

We typically pass it to our main coroutine and run our program from there.

There are low-level functions for creating and accessing the event loop.

The asyncio.new_event_loop() function (https://docs.python.org/3/library/asyncio-

eventloop.html#asyncio.new_event_loop) will create a new event loop and return
access to it.

https://superfastpython.com/python-asyncio/ 46/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Create and return a new event loop object.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
EVENTLOOP.HTML)

For example:

1 ...
2 # create and access a new asyncio event loop
3 loop = asyncio.new_event_loop()

We can demonstrate this with a worked example.

In the example below we will create a new event loop and then report its details.

1 # SuperFastPython.com
2 # example of creating an event loop
3 import asyncio
4
5 # create and access a new asyncio event loop
6 loop = asyncio.new_event_loop()
7 # report defaults of the loop
8 print(loop)

Running the example creates the event loop, then reports the details of the object.

We can see that in this case the event loop has the type _UnixSelectorEventLoop
and is not running, but is also not closed.

1 <_UnixSelectorEventLoop running=False closed=False debug=False>

If an asyncio event loop is already running, we can get access to it via the
asyncio.get_running_loop() function (https://docs.python.org/3/library/asyncio-
eventloop.html#asyncio.get_running_loop).

https://superfastpython.com/python-asyncio/ 47/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Return the running event loop in the current OS thread. If there is no running
event loop a RuntimeError is raised. This function can only be called from a
coroutine or a callback.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

EVENTLOOP.HTML)

For example:

1 ...
2 # access he running event loop
3 loop = asyncio.get_running_loop()

There is also a function for getting or starting the event loop called
asyncio.get_event_loop() (https://docs.python.org/3/library/asyncio-
eventloop.html#asyncio.get_event_loop), but it was deprecated in Python 3.10 and
should not be used.

What is an Event Loop Object

An event loop is implemented as a Python object.

The event loop object defines how the event loop is implemented and provides a
common API for interacting with the loop, defined on the AbstractEventLoop class
(https://github.com/python/cpython/blob/3.10/Lib/asyncio/events.py#L204).

There are different implementations (https://docs.python.org/3/library/asyncio-

eventloop.html#event-loop-implementations) of the event loop for different
platforms.

For example, Windows and Unix-based operations systems will implement the event
loop in different ways, given the different underlying ways that non-blocking I/O is
implemented on these platforms.

https://superfastpython.com/python-asyncio/ 48/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The SelectorEventLoop type event loop (https://docs.python.org/3/library/asyncio-

eventloop.html#asyncio.SelectorEventLoop) is the default on Unix-based operating
systems like Linux and macOS.

The ProactorEventLoop type event loop (https://docs.python.org/3/library/asyncio-

eventloop.html#asyncio.ProactorEventLoop) is the default on Windows.

Third-party libraries may implement their own event loops to optimize for specific
features.

Why Get Access to The Event Loop

Why would we want access to an event loop outside of an asyncio program?

There are many reasons why we may want access to the event loop from outside of a
running asyncio program.

For example:

1. To monitor the progress of tasks.

2. To issue and get results from tasks.
3. To fire and forget one-off tasks.

An asyncio event loop can be used in a program as an alternative to a thread pool for
coroutine-based tasks.

An event loop may also be embedded within a normal asyncio program and accessed
as needed.

Now that we know a little about the event loop, let’s look at asyncio tasks.

https://superfastpython.com/python-asyncio/ 49/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Create and Run Asyncio Tasks

You can create Task objects from coroutines in asyncio programs.

Tasks provide a handle on independently scheduled and running coroutines and

allow the task to be queried, canceled, and results and exceptions to be retrieved
later.

The asyncio event loop manages tasks. As such, all coroutines become and are
managed as tasks within the event loop.

Let’s take a closer look at asyncio tasks.

What is an Asyncio Task

A Task is an object that schedules and independently runs an asyncio coroutine.

It provides a handle on a scheduled coroutine that an asyncio program can query and
use to interact with the coroutine.

“ A Task is an object that manages an independently running coroutine.

— PEP 3156 – ASYNCHRONOUS IO SUPPORT REBOOTED: THE “ASYNCIO” MODULE
(HTTPS://PEPS.PYTHON.ORG/PEP-3156/)

A task is created from a coroutine. It requires a coroutine object, wraps the coroutine,
schedules it for execution, and provides ways to interact with it.

A task is executed independently. This means it is scheduled in the asyncio event loop
and will execute regardless of what else happens in the coroutine that created it. This
is different from executing a coroutine directly, where the caller must wait for it to
complete.

https://superfastpython.com/python-asyncio/ 50/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Tasks are used to schedule coroutines concurrently. When a coroutine is

wrapped into a Task with functions like asyncio.create_task() the coroutine is
automatically scheduled to run soon

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

TASK.HTML)

The asyncio.Task class (https://docs.python.org/3/library/asyncio-task.html#task-

object) extends the asyncio.Future class (https://docs.python.org/3/library/asyncio-
future.html#asyncio.Future) and an instance are awaitable.

A Future is a lower-level class that represents a result that will eventually arrive.

“ A Future is a special low-level awaitable object that represents an eventual

result of an asynchronous operation.

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

TASK.HTML)

Classes that extend the Future class are often referred to as Future-like.

“ A Future-like object that runs a Python coroutine.

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
TASK.HTML)

Because a Task is awaitable it means that a coroutine can wait for a task to be done
using the await expression.

For example:

https://superfastpython.com/python-asyncio/ 51/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # wait for a task to be done
3 await task

Now that we know what an asyncio task is, let’s look at how we might create one.

How to Create a Task

A task is created using a provided coroutine instance.

Recall that a coroutine is defined using the async def expression and looks like a
function.

For example:

1 # define a coroutine
2 async def task_coroutine():
3 # ...

A task can only be created and scheduled within a coroutine.

There are two main ways to create and schedule a task, they are:

1. Create Task With High-Level API (preferred)

2. Create Task With Low-Level API

Let’s take a closer look at each in turn.

Create Task With High-Level API

A task can be created using the asyncio.create_task() function

(https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task).

The asyncio.create_task() function takes a coroutine instance and an optional name

for the task and returns an asyncio.Task instance.

For example:

https://superfastpython.com/python-asyncio/ 52/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # create a coroutine
3 coro = task_coroutine()
4 # create a task from a coroutine
5 task = asyncio.create_task(coro)

This can be achieved with a compound statement on a single line.

For example:

1 ...
2 # create a task from a coroutine
3 task = asyncio.create_task(task_coroutine())

This will do a few things:

1. Wrap the coroutine in a Task instance.

2. Schedule the task for execution in the current event loop.
3. Return a Task instance

The task instance can be discarded, interacted with via methods, and awaited by a
coroutine.

This is the preferred way to create a Task from a coroutine in an asyncio program.

Create Task With Low-Level API

A task can also be created from a coroutine using the lower-level asyncio API.

The first way is to use the asyncio.ensure_future() function

(https://docs.python.org/3/library/asyncio-future.html#asyncio.ensure_future).

This function takes a Task, Future, or Future-like object, such as a coroutine, and
optionally the loop in which to schedule it.

If a loop is not provided, it will be scheduled in the current event loop.

If a coroutine is provided to this function, it is wrapped in a Task instance for us,

which is returned.

https://superfastpython.com/python-asyncio/ 53/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # create and schedule the task
3 task = asyncio.ensure_future(task_coroutine())

Another low-level function that we can use to create and schedule a Task is the
loop.create_task() method (https://docs.python.org/3/library/asyncio-
eventloop.html#asyncio.loop.create_task).

This function requires access to a specific event loop in which to execute the
coroutine as a task.

We can acquire an instance to the current event loop within an asyncio program via
the asyncio.get_event_loop() function (https://docs.python.org/3/library/asyncio-
eventloop.html#asyncio.get_event_loop).

This can then be used to call the create_task() method to create a Task instance and
schedule it for execution.

For example:

1 ...
2 # get the current event loop
3 loop = asyncio.get_event_loop()
4 # create and schedule the task
5 task = loop.create_task(task_coroutine())

When Does a Task Run?

A common question after creating a task is when does it run?

This is a good question.

Although we can schedule a coroutine to run independently as a task with the

create_task() function, it may not run immediately.

In fact, the task will not execute until the event loop has an opportunity to run.

https://superfastpython.com/python-asyncio/ 54/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This will not happen until all other coroutines are not running and it is the task’s turn
to run.

For example, if we had an asyncio program with one coroutine that created and
scheduled a task, the scheduled task will not run until the calling coroutine that
created the task is suspended.

This may happen if the calling coroutine chooses to sleep, chooses to await another
coroutine or task, or chooses to await the new task that was scheduled.

For example:

1 ...
2 # create a task from a coroutine
3 task = asyncio.create_task(task_coroutine())
4 # await the task, allowing it to run
5 await task

You can learn more about how to create asyncio tasks in the tutorial:

How to Create Asyncio Tasks in Python (https://superfastpython.com/asyncio-

create-task/)

Now that we know what a task is and how to schedule them, next, let’s look at how
we may use them in our programs.

Work With and Query Tasks

Tasks are the currency of asyncio programs.

In this section, we will take a closer look at how to interact with them in our
programs.

Task Life-Cycle
An asyncio Task has a life cycle.

https://superfastpython.com/python-asyncio/ 55/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Firstly, a task is created from a coroutine.

It is then scheduled for independent execution within the event loop.

At some point, it will run.

While running it may be suspended, such as awaiting another coroutine or task. It

may finish normally and return a result or fail with an exception.

Another coroutine may intervene and cancel the task.

Eventually, it will be done and cannot be executed again.

We can summarize this life-cycle as follows:

1. Created
2. Scheduled
2a Canceled
3. Running
3a. Suspended
3b. Result
3c. Exception
3d. Canceled
4. Done

Note that Suspended, Result, Exception, and Canceled are not states per se, they are
important points of transition for a running task.

The diagram below summarizes this life cycle showing the transitions between each
phase.

https://superfastpython.com/python-asyncio/ 56/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

ASYNCIO TASK LIFE-CYCLE

You can learn more about the asyncio task life-cycle in the tutorial:

Asyncio Task Life-Cycle (https://superfastpython.com/asyncio-task-life-cycle/)

Now that we are familiar with the life cycle of a task from a high level, let’s take a
closer look at each phase.

How to Check Task Status

After a Task is created, we can check the status of the task.

https://superfastpython.com/python-asyncio/ 57/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

There are two statuses we might want to check, they are:

Whether the task is done.

Whether the task was canceled.

Let’s take a closer look at each in turn.

Check if a Task is Done

We can check if a task is done via the done() method

(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.done).

The method returns True if the task is done, or False otherwise.

For example:

1 ...
2 # check if a task is done
3 if task.done():
4 # ...

A task is done if it has had the opportunity to run and is now no longer running.

A task that has been scheduled is not done.

Similarly, a task that is running is not done.

A task is done if:

The coroutine finishes normally.

The coroutine returns explicitly.
An unexpected error or exception is raised in the coroutine
The task is canceled.

https://superfastpython.com/python-asyncio/ 58/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Check if a Task is Canceled

We can check if a task is canceled via the cancelled() method

(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancelled).

The method returns True if the task was canceled, or False otherwise.

For example:

1 ...
2 # check if a task was canceled
3 if task.cancelled():
4 # ...

A task is canceled if the cancel() method was called on the task and completed
successfully, e..g cancel() returned True.

A task is not canceled if the cancel() method was not called, or if the cancel() method
was called but failed to cancel the task.

How to Get Task Result

We can get the result of a task via the result() method
(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.result).

This returns the return value of the coroutine wrapped by the Task or None if the
wrapped coroutine does not explicitly return a value.

For example:

1 ...
2 # get the return value from the wrapped coroutine
3 value = task.result()

If the coroutine raises an unhandled error or exception, it is re-raised when calling

the result() method and may need to be handled.

For example:

https://superfastpython.com/python-asyncio/ 59/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 try:
3 # get the return value from the wrapped coroutine
4 value = task.result()
5 except Exception:
6 # task failed and there is no result

If the task was canceled, then a CancelledError exception is raised when calling the
result() method and may need to be handled.

For example:

1 ...
2 try:
3 # get the return value from the wrapped coroutine
4 value = task.result()
5 except asyncio.CancelledError:
6 # task was canceled

As such, it is a good idea to check if the task was canceled first.

For example:

1 ...
2 # check if the task was not canceled
3 if not task.cancelled():
4 # get the return value from the wrapped coroutine
5 value = task.result()
6 else:
7 # task was canceled

If the task is not yet done, then an InvalidStateError exception is raised when calling
the result() method and may need to be handled.

For example:

1 ...
2 try:
3 # get the return value from the wrapped coroutine
4 value = task.result()
5 except asyncio.InvalidStateError:
6 # task is not yet done

As such, it is a good idea to check if the task is done first.

For example:

https://superfastpython.com/python-asyncio/ 60/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # check if the task is not done
3 if not task.done():
4 await task
5 # get the return value from the wrapped coroutine
6 value = task.result()

How to Get Task Exception

A coroutine wrapped by a task may raise an exception that is not handled.

This will cancel the task, in effect.

We can retrieve an unhandled exception in the coroutine wrapped by a task via the
exception() method (https://docs.python.org/3/library/asyncio-
task.html#asyncio.Task.exception).

For example:

1 ...
2 # get the exception raised by a task
3 exception = task.exception()

If an unhandled exception was not raised in the wrapped coroutine, then a value of
None is returned.

If the task was canceled, then a CancelledError exception is raised when calling the
exception() method and may need to be handled.

For example:

1 ...
2 try:
3 # get the exception raised by a task
4 exception = task.exception()
5 except asyncio.CancelledError:
6 # task was canceled

As such, it is a good idea to check if the task was canceled first.

For example:

https://superfastpython.com/python-asyncio/ 61/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # check if the task was not canceled
3 if not task.cancelled():
4 # get the exception raised by a task
5 exception = task.exception()
6 else:
7 # task was canceled

If the task is not yet done, then an InvalidStateError exception is raised when calling
the exception() method and may need to be handled.

For example:

1 ...
2 try:
3 # get the exception raised by a task
4 exception = task.exception()
5 except asyncio.InvalidStateError:
6 # task is not yet done

As such, it is a good idea to check if the task is done first.

For example:

1 ...
2 # check if the task is not done
3 if not task.done():
4 await task
5 # get the exception raised by a task
6 exception = task.exception()

How to Cancel a Task

We can cancel a scheduled task via the cancel() method
(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel).

The cancel method returns True if the task was canceled, or False otherwise.

For example:

1 ...
2 # cancel the task
3 was_cancelled = task.cancel()

If the task is already done, it cannot be canceled and the cancel() method will return
False and the task will not have the status of canceled.

https://superfastpython.com/python-asyncio/ 62/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The next time the task is given an opportunity to run, it will raise a CancelledError
exception.

If the CancelledError exception is not handled within the wrapped coroutine, the
task will be canceled.

Otherwise, if the CancelledError exception is handled within the wrapped coroutine,

the task will not be canceled.

The cancel() method can also take a message argument which will be used in the
content of the CancelledError.

How to Use Callback With a Task

We can add a done callback function to a task via the add_done_callback() method
(https://docs.python.org/3/library/asyncio-
task.html#asyncio.Task.add_done_callback).

This method takes the name of a function to call when the task is done.

The callback function must take the Task instance as an argument.

For example:

1 # done callback function

2 def handle(task):
3 print(task)
4
5 ...
6 # register a done callback function
7 task.add_done_callback(handle)

Recall that a task may be done when the wrapped coroutine finishes normally when it
returns, when an unhandled exception is raised or when the task is canceled.

The add_done_callback() method can be used to add or register as many done

callback functions as we like.

https://superfastpython.com/python-asyncio/ 63/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We can also remove or de-register a callback function via the

remove_done_callback() function (https://docs.python.org/3/library/asyncio-
task.html#asyncio.Task.remove_done_callback).

For example:

1 ...
2 # remove a done callback function
3 task.remove_done_callback(handle)

How to Set the Task Name

A task may have a name.

This name can be helpful if multiple tasks are created from the same coroutine and
we need some way to tell them apart programmatically.

The name can be set when the task is created from a coroutine via the “name”
argument.

For example:

1 ...
2 # create a task from a coroutine
3 task = asyncio.create_task(task_coroutine(), name='MyTask')

The name for the task can also be set via the set_name() method
(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.set_name).

For example:

1 ...
2 # set the name of the task
3 task.set_name('MyTask')

We can retrieve the name of a task via the get_name() method.

For example:

https://superfastpython.com/python-asyncio/ 64/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # get the name of a task
3 name = task.get_name()

You can learn more about checking the status of tasks in the tutorial:

How to Check Asyncio Task Status (https://superfastpython.com/asyncio-task-

status/)

Current and Running Tasks

We can introspect tasks running in the asyncio event loop.

This can be achieved by getting an asyncio.Task object for the currently running task
and for all tasks that are running.

How to Get the Current Task

We can get the current task via the asyncio.current_task() function
(https://docs.python.org/3/library/asyncio-task.html#asyncio.current_task).

This function will return a Task object for the task that is currently running.

For example:

1 ...
2 # get the current task
3 task = asyncio.current_task()

This will return a Task object for the currently running task.

This may be:

The main coroutine passed to asyncio.run().

A task created and scheduled within the asyncio program via
asyncio.create_task().

https://superfastpython.com/python-asyncio/ 65/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

A task may create and run another coroutine (e.g. not wrapped in a task). Getting the
current task from within a coroutine will return a Task object for the running task, but
not the coroutine that is currently running.

Getting the current task can be helpful if a coroutine or task requires details about
itself, such as the task name for logging.

We can explore how to get a Task instance for the main coroutine used to start an
asyncio program.

The example below defines a coroutine used as the entry point into the program. It
reports a message, then gets the current task and reports its details.

This is an important first example, as it highlights that all coroutines can be accessed
as tasks within the asyncio event loop.

The complete example is listed below.

1 # SuperFastPython.com
2 # example of getting the current task from the main coroutine
3 import asyncio
4
5 # define a main coroutine
6 async def main():
7 # report a message
8 print('main coroutine started')
9 # get the current task
10 task = asyncio.current_task()
11 # report its details
12 print(task)
13
14 # start the asyncio program
15 asyncio.run(main())

Running the example first creates the main coroutine and uses it to start the asyncio
program.

The main() coroutine runs and first reports a message.

It then retrieves the current task, which is a Task object that represents itself, the
currently running coroutine.

https://superfastpython.com/python-asyncio/ 66/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

It then reports the details of the currently running task.

We can see that the task has the default name for the first task, ‘Task-1‘ and is
executing the main() coroutine, the currently running coroutine.

This highlights that we can use the asyncio.current_task() function to access a Task
object for the currently running coroutine, that is automatically wrapped in a Task
object.

1 main coroutine started

2 <Task pending name='Task-1' coro=<main() running at ...> cb=[_run_until_complete_cb() at ...]>

You can learn more about getting the current task in the tutorial:

How to Get the Current Asyncio Task in Python

(https://superfastpython.com/asyncio-current-task/)

How to Get All Tasks

We may need to get access to all tasks in an asyncio program.

This may be for many reasons, such as:

To introspect the current status or complexity of the program.

To log the details of all running tasks.
To find a task that can be queried or canceled.

We can get a set of all scheduled and running (not yet done) tasks in an asyncio
program via the asyncio.all_tasks() function.

For example:

1 ...
2 # get all tasks
3 tasks = asyncio.all_tasks()

This will return a set of all tasks in the asyncio program.

https://superfastpython.com/python-asyncio/ 67/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

It is a set so that each task is only represented once.

A task will be included if:

The task has been scheduled but is not yet running.

The task is currently running (e.g. but is currently suspended)

The set will also include a task for the currently running task, e.g. the task that is
executing the coroutine that calls the asyncio.all_tasks() function.

Also, recall that the asyncio.run() method that is used to start an asyncio program
will wrap the provided coroutine in a task. This means that the set of all tasks will
include the task for the entry point of the program.

We can explore the case where we have many tasks within an asyncio program and
then get a set of all tasks.

In this example, we first create 10 tasks, each wrapping and running the same
coroutine.

The main coroutine then gets a set of all tasks scheduled or running in the program
and reports their details.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 68/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of starting many tasks and getting access to all tasks
3 import asyncio
4
5 # coroutine for a task
6 async def task_coroutine(value):
7 # report a message
8 print(f'task {value} is running')
9 # block for a moment
10 await asyncio.sleep(1)
11
12 # define a main coroutine
13 async def main():
14 # report a message
15 print('main coroutine started')
16 # start many tasks
17 started_tasks = [asyncio.create_task(task_coroutine(i)) for i in range(10)]
18 # allow some of the tasks time to start
19 await asyncio.sleep(0.1)
20 # get all tasks
21 tasks = asyncio.all_tasks()
22 # report all tasks
23 for task in tasks:
24 print(f'> {task.get_name()}, {task.get_coro()}')
25 # wait for all tasks to complete
26 for task in started_tasks:
27 await task
28
29 # start the asyncio program
30 asyncio.run(main())

Running the example first creates the main coroutine and uses it to start the asyncio
program.

The main() coroutine runs and first reports a message.

It then creates and schedules 10 tasks that wrap the custom coroutine,

The main() coroutine then blocks for a moment to allow the tasks to begin running.

The tasks start running and each reports a message and then sleeps.

The main() coroutine resumes and gets a list of all tasks in the program.

It then reports the name and coroutine of each.

Finally, it enumerates the list of tasks that were created and awaits each, allowing
them to be completed.

https://superfastpython.com/python-asyncio/ 69/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This highlights that we can get a set of all tasks in an asyncio program that includes
both the tasks that were created as well as the task that represents the entry point
into the program.

1 main coroutine started

2 task 0 is running
3 task 1 is running
4 task 2 is running
5 task 3 is running
6 task 4 is running
7 task 5 is running
8 task 6 is running
9 task 7 is running
10 task 8 is running
11 task 9 is running
12 > Task-9, <coroutine object task_coroutine at 0x10e186e30>
13 > Task-2, <coroutine object task_coroutine at 0x10e184e40>
14 > Task-11, <coroutine object task_coroutine at 0x10e186f10>
15 > Task-7, <coroutine object task_coroutine at 0x10e186d50>
16 > Task-4, <coroutine object task_coroutine at 0x10e185700>
17 > Task-10, <coroutine object task_coroutine at 0x10e186ea0>
18 > Task-8, <coroutine object task_coroutine at 0x10e186dc0>
19 > Task-5, <coroutine object task_coroutine at 0x10e186ab0>
20 > Task-1, <coroutine object main at 0x10e1847b0>
21 > Task-3, <coroutine object task_coroutine at 0x10e184f90>
22 > Task-6, <coroutine object task_coroutine at 0x10e186ce0>

You can learn more about getting all tasks. in the tutorial:

How to Get All Asyncio Tasks in Python (https://superfastpython.com/asyncio-all-

tasks/)

Next, we will explore how to run many coroutines concurrently.

Run Many Coroutines Concurrently

A benefit of asyncio is that we can run many coroutines concurrently.

These coroutines can be created in a group and stored, then executed all together at
the same time.

This can be achieved using the asyncio.gather() function.

Let’s take a closer look.

https://superfastpython.com/python-asyncio/ 70/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

What is Asyncio gather()

The asyncio.gather() module (https://docs.python.org/3/library/asyncio-
task.html#asyncio.gather) function allows the caller to group multiple awaitables
together.

Once grouped, the awaitables can be executed concurrently, awaited, and canceled.

“ Run awaitable objects in the aws sequence concurrently.

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
TASK.HTML)

It is a helpful utility function for both grouping and executing multiple coroutines or
multiple tasks.

For example:

1 ...
2 # run a collection of awaitables
3 results = await asyncio.gather(coro1(), asyncio.create_task(coro2()))

We may use the asyncio.gather() function in situations where we may create many
tasks or coroutines up-front and then wish to execute them all at once and wait for
them all to complete before continuing on.

This is a likely situation where the result is required from many like-tasks, e.g. same
task or coroutine with different data.

The awaitables can be executed concurrently, results returned, and the main
program can resume by making use of the results on which it is dependent.

The gather() function is more powerful than simply waiting for tasks to complete.

It allows a group of awaitables to be treated as a single awaitable.

https://superfastpython.com/python-asyncio/ 71/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This allows:

Executing and waiting for all awaitables in the group to be done via an await
expression.
Getting results from all grouped awaitables to be retrieved later via the result()
method.
The group of awaitables to be canceled via the cancel() method.
Checking if all awaitables in the group are done via the done() method.
Executing callback functions only when all tasks in the group are done.

And more.

How to use Asyncio gather()

In this section, we will take a closer look at how we might use the asyncio.gather()
function.

The asyncio.gather() function takes one or more awaitables as arguments.

Recall an awaitable may be a coroutine, a Future or a Task.

Therefore, we can call the gather() function with:

Multiple tasks
Multiple coroutines
Mixture of tasks and coroutines

For example:

1 ...
2 # execute multiple coroutines
3 asyncio.gather(coro1(), coro2())

If Task objects are provided to gather(), they will already be running because Tasks
are scheduled as part of being created.

https://superfastpython.com/python-asyncio/ 72/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The asyncio.gather() function takes awaitables as position arguments.

We cannot create a list or collection of awaitables and provide it to gather, as this will
result in an error.

For example:

1 ...
2 # cannot provide a list of awaitables directly
3 asyncio.gather([coro1(), coro2()])

A list of awaitables can be provided if it is first unpacked into separate expressions

using the star operator (*).

For example:

1 ...
2 # gather with an unpacked list of awaitables
3 asyncio.gather(*[coro1(), coro2()])

If coroutines are provided to gather(), they are wrapped in Task objects

automatically.

The gather() function does not block.

Instead, it returns an asyncio.Future (https://docs.python.org/3/library/asyncio-

future.html#asyncio.Future) object that represents the group of awaitables.

For example:

1 ...
2 # get a future that represents multiple awaitables
3 group = asyncio.gather(coro1(), coro2())

Once the Future object is created it is scheduled automatically within the event loop.

The awaitable represents the group, and all awaitables in the group will execute as
soon as they are able.

https://superfastpython.com/python-asyncio/ 73/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This means that if the caller did nothing else, the scheduled group of awaitables will
run (assuming the caller suspends).

It also means that you do not have to await the Future that is returned from
gather().

For example:

1 ...
2 # get a future that represents multiple awaitables
3 group = asyncio.gather(coro1(), coro2())
4 # suspend and wait a while, the group may be executing..
5 await asyncio.sleep(10)

The returned Future object can be awaited which will wait for all awaitables in the
group to be done.

For example:

1 ...
2 # run the group of awaitables
3 await group

Awaiting the Future returned from gather() will return a list of return values from the
awaitables.

If the awaitables do not return a value, then this list will contain the default “None”
return value.

For example:

1 ...
2 # run the group of awaitables and get return values
3 results = await group

This is more commonly performed in one line.

For example:

1 ...
2 # run tasks and get results on one line
3 results = await asyncio.gather(coro1(), coro2())

https://superfastpython.com/python-asyncio/ 74/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Example of gather() For Many Coroutines in a List

It is common to create multiple coroutines beforehand and then gather them later.

This allows a program to prepare the tasks that are to be executed concurrently and
then trigger their concurrent execution all at once and wait for them to complete.

We can collect many coroutines together into a list either manually or using a list
comprehension.

For example:

1 ...
2 # create many coroutines
3 coros = [task_coro(i) for i in range(10)]

We can then call gather() with all coroutines in the list.

The list of coroutines cannot be provided directly to the gather() function as this will
result in an error.

Instead, the gather() function requires each awaitable to be provided as a separate

positional argument.

This can be achieved by unwrapping the list into separate expressions and passing
them to the gather() function. The star operator (*) will perform this operation for us.

For example:

1 ...
2 # run the tasks
3 await asyncio.gather(*coros)

Tying this together, the complete example of running a list of pre-prepared

coroutines with gather() is listed below.

https://superfastpython.com/python-asyncio/ 75/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of gather for many coroutines in a list
3 import asyncio
4
5 # coroutine used for a task
6 async def task_coro(value):
7 # report a message
8 print(f'>task {value} executing')
9 # sleep for a moment
10 await asyncio.sleep(1)
11
12 # coroutine used for the entry point
13 async def main():
14 # report a message
15 print('main starting')
16 # create many coroutines
17 coros = [task_coro(i) for i in range(10)]
18 # run the tasks
19 await asyncio.gather(*coros)
20 # report a message
21 print('main done')
22
23 # start the asyncio program
24 asyncio.run(main())

Running the example executes the main() coroutine as the entry point to the
program.

The main() coroutine then creates a list of 10 coroutine objects using a list
comprehension.

This list is then provided to the gather() function and unpacked into 10 separate
expressions using the star operator.

The main() coroutine then awaits the Future object returned from the call to gather(),
suspending and waiting for all scheduled coroutines to complete their execution.

The coroutines run as soon as they are able, reporting their unique messages and
sleeping before terminating.

Only after all coroutines in the group are complete does the main() coroutine resume
and report its final message.

This highlights how we might prepare a collection of coroutines and provide them as
separate expressions to the gather() function.

https://superfastpython.com/python-asyncio/ 76/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 main starting
2 >task 0 executing
3 >task 1 executing
4 >task 2 executing
5 >task 3 executing
6 >task 4 executing
7 >task 5 executing
8 >task 6 executing
9 >task 7 executing
10 >task 8 executing
11 >task 9 executing
12 main done

You can learn more about how to use the gather() function in the tutorial:

How to Use asyncio.gather() in Python (https://superfastpython.com/asyncio-

gather/)

Next, we will explore how to wait on a group of asyncio tasks.

Wait for A Collection of Tasks

We can wait for asyncio tasks to complete via the asyncio.wait() function.

Different conditions can be waited for, such as all tasks to complete, the first task to
complete, and the first task to fail with an exception.

Let’s take a closer look.

What is asyncio.wait()
The asyncio.wait() function (https://docs.python.org/3/library/asyncio-
task.html#asyncio.wait) can be used to wait for a collection of asyncio tasks to
complete.

Recall that an asyncio task is an instance of the asyncio.Task class that wraps a
coroutine. It allows a coroutine to be scheduled and executed independently, and the
Task instance provides a handle on the task for querying status and getting results.

https://superfastpython.com/python-asyncio/ 77/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

You can learn more about asyncio tasks in the tutorial:

What is an Asyncio Task (/asyncio-task)

The wait() function allows us to wait for a collection of tasks to be done.

The call to wait can be configured to wait for different conditions, such as all tasks
being completed, the first task completed and the first task failing with an error.

Next, let’s look at how we might use the wait() function.

How to Use asyncio.wait()

The asyncio.wait() function takes a collection of awaitables, typically Task objects.

This could be a list, dict, or set of task objects that we have created, such as via calls
to the asyncio.create_task() function in a list comprehension.

For example:

1 ...
2 # create many tasks
3 tasks = [asyncio.create_task(task_coro(i)) for i in range(10)]

The asyncio.wait() will not return until some condition on the collection of tasks is
met.

By default, the condition is that all tasks are completed.

The wait() function returns a tuple of two sets. The first set contains all task objects
that meet the condition, and the second contains all other task objects that do not yet
meet the condition.

These sets are referred to as the “done” set and the “pending” set.

For example:

https://superfastpython.com/python-asyncio/ 78/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # wait for all tasks to complete
3 done, pending = await asyncio.wait(tasks)

Technically, the asyncio.wait() is a coroutine function that returns a coroutine.

We can then await this coroutine which will return the tuple of sets.

For example:

1 ...
2 # create the wait coroutine
3 wait_coro = asyncio.wait(tasks)
4 # await the wait coroutine
5 tuple = await wait_coro

The condition waited for can be specified by the “return_when” argument which is
set to asyncio.ALL_COMPLETED by default.

For example:

1 ...
2 # wait for all tasks to complete
3 done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)

We can wait for the first task to be completed by setting return_when to

FIRST_COMPLETED.

For example:

1 ...
2 # wait for the first task to be completed
3 done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)

When the first task is complete and returned in the done set, the remaining tasks are
not canceled and continue to execute concurrently.

We can wait for the first task to fail with an exception by setting return_when to
FIRST_EXCEPTION.

For example:

1 ...
2 # wait for the first task to fail
3 done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)

https://superfastpython.com/python-asyncio/ 79/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

In this case, the done set will contain the first task that failed with an exception. If no
task fails with an exception, the done set will contain all tasks and wait() will return
only after all tasks are completed.

We can specify how long we are willing to wait for the given condition via a “timeout”
argument in seconds.

If the timeout expires before the condition is met, the tuple of tasks is returned with
whatever subset of tasks do meet the condition at that time, e.g. the subset of tasks
that are completed if waiting for all tasks to complete.

For example:

1 ...
2 # wait for all tasks to complete with a timeout
3 done, pending = await asyncio.wait(tasks, timeout=3)

If the timeout is reached before the condition is met, an exception is not raised and
the remaining tasks are not canceled.

Now that we know how to use the asyncio.wait() function, let’s look at some worked
examples.

Example of Waiting for All Tasks

We can explore how to wait for all tasks using asyncio.wait().

In this example, we will define a simple task coroutine that generates a random value,
sleeps for a fraction of a second, then reports a message with the generated value.

The main coroutine will then create many tasks in a list comprehension with the
coroutine and then wait for all tasks to be completed.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 80/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of waiting for all tasks to complete
3 from random import random
4 import asyncio
5
6 # coroutine to execute in a new task
7 async def task_coro(arg):
8 # generate a random value between 0 and 1
9 value = random()
10 # block for a moment
11 await asyncio.sleep(value)
12 # report the value
13 print(f'>task {arg} done with {value}')
14
15 # main coroutine
16 async def main():
17 # create many tasks
18 tasks = [asyncio.create_task(task_coro(i)) for i in range(10)]
19 # wait for all tasks to complete
20 done,pending = await asyncio.wait(tasks)
21 # report results
22 print('All done')
23
24 # start the asyncio program
25 asyncio.run(main())

Running the example first creates the main() coroutine and uses it as the entry point
into the asyncio program.

The main() coroutine then creates a list of ten tasks in a list comprehension, each
providing a unique integer argument from 0 to 9.

The main() coroutine is then suspended and waits for all tasks to complete.

The tasks execute. Each generates a random value, sleeps for a moment, then reports
its generated value.

After all tasks have been completed, the main() coroutine resumes and reports a
final message.

This example highlights how we can use the wait() function to wait for a collection of
tasks to be completed.

This is perhaps the most common usage of the function.

Note, that the results will differ each time the program is run given the use of random
numbers.

https://superfastpython.com/python-asyncio/ 81/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 >task 5 done with 0.0591009105682192
2 >task 8 done with 0.10453715687017351
3 >task 0 done with 0.15462838864295925
4 >task 6 done with 0.4103492027393125
5 >task 9 done with 0.45567100006991623
6 >task 2 done with 0.6984682905809402
7 >task 7 done with 0.7785363531316224
8 >task 3 done with 0.827386088873161
9 >task 4 done with 0.9481344994700972
10 >task 1 done with 0.9577302665040541
11 All done

You can learn more about the wait() function in the tutorial:

How to Use Asyncio wait() in Python (https://superfastpython.com/asyncio-wait/)

Next, let’s look at how we can manage multiple tasks together with a TaskGroup.

Manage Multiple Coroutines in a Group

We can manage multiple coroutines together in a group using the asyncio.TaskGroup.

If a task in the group fails with an exception, all tasks in the group are cancelled
automatically.

What is an asyncio.TaskGroup
Python 3.11 introduced the asyncio.TaskGroup task for managing a group of
associated asyncio task.

“ Added the TaskGroup class, an asynchronous context manager holding a

group of tasks that will wait for all of them upon exit. For new code this is
recommended over using create_task() and gather() directly.

— WHAT’S NEW IN PYTHON 3.11

(HTTPS://DOCS.PYTHON.ORG/3/WHATSNEW/3.11.HTML)

https://superfastpython.com/python-asyncio/ 82/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The asyncio.TaskGroup class is intended as a replacement for the

asyncio.create_task() function for creating tasks and the asyncio.gather() function
for waiting on a group of tasks.

Historically, we create and issue a coroutine as an asyncio.Task using the

asyncio.create_task() function.

For example:

1 ...
2 # create and issue coroutine as task
3 task = asyncio.create_task(coro())

This creates a new asyncio.Task object and issues it to the asyncio event loop for
execution as soon as it is able.

We can then choose to await the task and wait for it to be completed.

For example:

1 ...
2 # wait for task to complete
3 result = await task

You can learn more about executing coroutines as asyncio.Task objects in the
tutorial:

How to Create Asyncio Tasks in Python (https://superfastpython.com/asyncio-

create-task/)

As we have seen, the asyncio.gather() function is used to create and issue many
coroutines simultaneously as asyncio.Task objects to the event loop, allowing the
caller to treat them all as a group.

The most common usage is to wait for all issued tasks to complete.

For example:

https://superfastpython.com/python-asyncio/ 83/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # issue coroutines as tasks and wait for them to complete
3 results = await asyncio.gather(coro1(), coro2(), coro2)

The asyncio.TaskGroup can perform both of these activities and is the preferred
approach.

“ An asynchronous context manager holding a group of tasks. Tasks can be

added to the group using create_task(). All tasks are awaited when the
context manager exits.

— ASYNCIO TASK GROUPS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

TASK.HTML#ASYNCIO.TASKGROUP)

How to Create an asyncio.TaskGroup

An asyncio.TaskGroup object implements the asynchronous context manager
interface, and this is the preferred usage of the class.

This means that an instance of the class is created and is used via the “async with”
expression.

For example:

1 ...
2 # create a taskgroup
3 async with asyncio.TaskGroup() as group:
4 # ...

If you are new to the “async with” expression, see the tutorial:

How to Use the “async with” Expression in Python

(https://superfastpython.com/asyncio-async-with/)

Recall that an asynchronous context manager implements the __aenter__() and

__aexit__() methods which can be awaited.

https://superfastpython.com/python-asyncio/ 84/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

In the case of the asyncio.TaskGroup, the __aexit__() method which is called

automatically when the context manager block is exited will await all tasks created by
the asyncio.TaskGroup.

This means that exiting the TaskGroup object’s block normally or via an exception
will automatically await until all group tasks are done.

1 ...
2 # create a taskgroup
3 async with asyncio.TaskGroup() as group:
4 # ...
5 # wait for all group tasks are done

You can learn more about asynchronous context managers in the tutorial:

Asynchronous Context Managers in Python

(https://superfastpython.com/asynchronous-context-manager/)

How to Create Tasks Using asyncio.TaskGroup

We can create a task in the task group via the create_task() method on the
asyncio.TaskGroup object.

For example:

1 ...
2 # create a taskgroup
3 async with asyncio.TaskGroup() as group:
4 # create and issue a task
5 task = group.create_task(coro())

This will create an asyncio.Task object and issue it to the asyncio event loop for
execution, just like the asyncio.create_task() function, except that the task is
associated with the group.

We can await the task directly if we choose and get results.

For example:

https://superfastpython.com/python-asyncio/ 85/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # create a taskgroup
3 async with asyncio.TaskGroup() as group:
4 # create and issue a task
5 result = await group.create_task(coro())

The benefit of using the asyncio.TaskGroup is that we can issue multiple tasks in the
group and execute code in between. such as checking results or gathering more data.

How to Wait on Tasks Using asyncio.TaskGroup

We can wait on all tasks in the group by exiting the asynchronous context manager
block.

As such, the tasks are awaited automatically and nothing additional is required.

For example:

1 ...
2 # create a taskgroup
3 async with asyncio.TaskGroup() as group:
4 # ...
5 # wait for all group tasks are done

If this behavior is not preferred, then we must ensure all tasks are “done” (finished,
canceled, or failed) before exiting the context manager.

How to Cancel All Tasks If One Task Fails Using

asyncio.TaskGroup
If one task in the group fails with an exception, then all non-done tasks remaining in
the group will be canceled.

This is performed automatically and does not require any additional code.

For example:

https://superfastpython.com/python-asyncio/ 86/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # handle the failure of any tasks in the group
2 try:
3 ...
4 # create a taskgroup
5 async with asyncio.TaskGroup() as group:
6 # create and issue a task
7 task1 = group.create_task(coro1())
8 # create and issue a task
9 task2 = group.create_task(coro2())
10 # create and issue a task
11 task3 = group.create_task(coro3())
12 # wait for all group tasks are done
13 except:
14 # all non-done tasks are cancelled
15 pass

If this behavior is not preferred, then the failure of each task must be managed within
the tasks themselves, e.g. by a try-except block within the coroutine.

Now that we know how to use the asyncio.TaskGroup, let’s look at some worked
examples.

Example of Waiting on Multiple Tasks with a TaskGroup

We can explore the case of creating multiple tasks within an asyncio.TaskGroup and
then waiting for all tasks to complete.

This can be achieved by first defining a suite of different coroutines that represent the
tasks we want to complete.

In this case, we will define 3 coroutines that each report a different message and then
sleep for one second.

https://superfastpython.com/python-asyncio/ 87/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # coroutine task
2 async def task1():
3 # report a message
4 print('Hello from coroutine 1')
5 # sleep to simulate waiting
6 await asyncio.sleep(1)
7
8 # coroutine task
9 async def task2():
10 # report a message
11 print('Hello from coroutine 2')
12 # sleep to simulate waiting
13 await asyncio.sleep(1)
14
15 # coroutine task
16 async def task3():
17 # report a message
18 print('Hello from coroutine 3')
19 # sleep to simulate waiting
20 await asyncio.sleep(1)

Next, we can define a main() coroutine that creates the asyncio.TaskGroup via the
context manager interface.

1 # asyncio entry point

2 async def main():
3 # create task group
4 async with asyncio.TaskGroup() as group:
5 # ...

We can then create and issue each coroutine as a task into the event loop, although
collected together as part of the group.

1 ...
2 # run first task
3 group.create_task(task1())
4 # run second task
5 group.create_task(task2())
6 # run third task
7 group.create_task(task3())

Notice that we don’t need to keep a reference to the asyncio.Task objects as the
asyncio.TaskGroup will keep track of them for us.

Also, notice that we don’t need to await the tasks because when we exit the context
manager block for the asyncio.TaskGroup we will await all tasks in the group.

Tying this together, the complete example is listed below.

https://superfastpython.com/python-asyncio/ 88/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # example of asyncio task group
2 import asyncio
3
4 # coroutine task
5 async def task1():
6 # report a message
7 print('Hello from coroutine 1')
8 # sleep to simulate waiting
9 await asyncio.sleep(1)
10
11 # coroutine task
12 async def task2():
13 # report a message
14 print('Hello from coroutine 2')
15 # sleep to simulate waiting
16 await asyncio.sleep(1)
17
18 # coroutine task
19 async def task3():
20 # report a message
21 print('Hello from coroutine 3')
22 # sleep to simulate waiting
23 await asyncio.sleep(1)
24
25 # asyncio entry point
26 async def main():
27 # create task group
28 async with asyncio.TaskGroup() as group:
29 # run first task
30 group.create_task(task1())
31 # run second task
32 group.create_task(task2())
33 # run third task
34 group.create_task(task3())
35 # wait for all tasks to complete...
36 print('Done')
37
38 # entry point
39 asyncio.run(main())

Running the example first executes the main() coroutine, starting a new event loop
for us.

The main() coroutine runs and creates an asyncio.TaskGroup.

All three coroutines are then created as asyncio.Task objects and issued to the event
loop via the asyncio.TaskGroup.

The context manager block for the asyncio.TaskGroup is exited which automatically
awaits all three tasks.

The tasks report their message and sleep.

Once all tasks are completed the main() coroutine reports a final message.

https://superfastpython.com/python-asyncio/ 89/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 Hello from coroutine 1
2 Hello from coroutine 2
3 Hello from coroutine 3
4 Done

Next, let’s explore how we might use an asyncio.TaskGroup with tasks that take
arguments and return values.

You can learn more about how to use the TaskGroup in the tutorial:

How to use asyncio.TaskGroup (https://superfastpython.com/asyncio-taskgroup/)

Next, we will explore how to wait for a single coroutine with a time limit.

Wait for a Coroutine with a Time Limit

We can wait for an asyncio task or coroutine to complete with a timeout using the
asyncio.wait_for() function.

If the timeout elapses before the task completes, the task is canceled.

Let’s take a closer look.

What is Asyncio wait_for()

The asyncio.wait_for() function allows the caller to wait for an asyncio task or
coroutine to complete with a timeout.

If no timeout is specified, the wait_for() function will wait until the task is completed.

If a timeout is specified and elapses before the task is complete, then the task is
canceled.

https://superfastpython.com/python-asyncio/ 90/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Wait for the aw awaitable to complete with a timeout.

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
TASK.HTML)

This allows the caller to both set an expectation about how long they are willing to
wait for a task to complete, and to enforce the timeout by canceling the task if the
timeout elapses.

Now that we know what the asyncio.wait_for() function is, let’s look at how to use it.

How to Use Asyncio wait_for()

The asyncio.wait_for() function (https://docs.python.org/3/library/asyncio-
task.html#asyncio.wait_for) takes an awaitable and a timeout.

The awaitable may be a coroutine or a task.

A timeout must be specified and may be None for no timeout, an integer or floating
point number of seconds.

The wait_for() function returns a coroutine that is not executed until it is explicitly
awaited or scheduled as a task.

For example:

1 ...
2 # wait for a task to complete
3 await asyncio.wait_for(coro, timeout=10)

If a coroutine is provided, it will be converted to the task when the wait_for()

coroutine is executed.

If the timeout elapses before the task is completed, the task is canceled, and an
asyncio.TimeoutError is raised, which may need to be handled.

https://superfastpython.com/python-asyncio/ 91/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # execute a task with a timeout
3 try:
4 # wait for a task to complete
5 await asyncio.wait_for(coro, timeout=1)
6 except asyncio.TimeoutError:
7 # ...

If the waited-for task fails with an unhandled exception, the exception will be
propagated back to the caller that is awaiting on the wait_for() coroutine, in which
case it may need to be handled.

For example

1 ...
2 # execute a task that may fail
3 try:
4 # wait for a task to complete
5 await asyncio.wait_for(coro, timeout=1)
6 except asyncio.TimeoutError:
7 # ...
8 except Exception:
9 # ...

Next, let’s look at how we can call wait_for() with a timeout.

Example of Asyncio wait_for() With a Timeout

We can explore how to wait for a coroutine with a timeout that elapses before the
task is completed.

In this example, we execute a coroutine as above, except the caller waits a fixed
timeout of 0.2 seconds or 200 milliseconds.

Recall that one second is equal to 1,000 milliseconds.

The task coroutine is modified so that it sleeps for more than one second, ensuring
that the timeout always expires before the task is complete.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 92/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of waiting for a coroutine with a timeout
3 from random import random
4 import asyncio
5
6 # coroutine to execute in a new task
7 async def task_coro(arg):
8 # generate a random value between 0 and 1
9 value = 1 + random()
10 # report message
11 print(f'>task got {value}')
12 # block for a moment
13 await asyncio.sleep(value)
14 # report all done
15 print('>task done')
16
17 # main coroutine
18 async def main():
19 # create a task
20 task = task_coro(1)
21 # execute and wait for the task without a timeout
22 try:
23 await asyncio.wait_for(task, timeout=0.2)
24 except asyncio.TimeoutError:
25 print('Gave up waiting, task canceled')
26
27 # start the asyncio program
28 asyncio.run(main())

Running the example first creates the main() coroutine and uses it as the entry point
into the asyncio program.

The main() coroutine creates the task coroutine. It then calls wait_for() and passes
the task coroutine and sets the timeout to 0.2 seconds.

The main() coroutine is suspended and the task_coro() is executed. It reports a

message and sleeps for a moment.

The main() coroutine resumes after the timeout has elapsed. The wait_for()
coroutine cancels the task_coro() coroutine and the main() coroutine is suspended.

The task_coro() runs again and responds to the request to be terminated. It raises a
TimeoutError exception and terminates.

The main() coroutine resumes and handles the TimeoutError raised by the
task_coro().

This highlights how we can call the wait_for() function with a timeout and to cancel a
task if it is not completed within a timeout.
https://superfastpython.com/python-asyncio/ 93/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The output from the program will differ each time it is run given the use of random
numbers.

1 >task got 0.685375224799321

2 Gave up waiting, task canceled

You can learn more about the wait_for() function in the tutorial:

Asyncio wait_for() to Wait With a Timeout (https://superfastpython.com/asyncio-

wait_for/)

Next, we will explore how we might protect an asyncio task from being canceled.

Shield Tasks from Cancellation

Asyncio tasks can be canceled by calling their cancel() method.

We can protect a task from being canceled by wrapping it in a call to asyncio.shield().

Let’s take a closer look.

What is Asyncio shield()

The asyncio.shield() function wraps an awaitable in Future that will absorb requests to
be canceled.

“ Protect an awaitable object from being cancelled.

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
TASK.HTML)

This means the shielded future can be passed around to tasks that may try to cancel
it and the cancellation request will look like it was successful, except that the Task or
coroutine that is being shielded will continue to run.
https://superfastpython.com/python-asyncio/ 94/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

It may be useful in asyncio programs where some tasks can be canceled, but others,
perhaps with a higher priority cannot.

It may also be useful in programs where some tasks can safely be canceled, such as
those that were designed with asyncio in mind, whereas others cannot be safely
terminated and therefore must be shielded from cancellation.

Now that we know what asyncio.shield() is, let’s look at how to use it.

How to Use Asyncio shield()

The asyncio.shield() function (https://docs.python.org/3/library/asyncio-
task.html#asyncio.shield) will protect another Task or coroutine from being canceled.

It takes an awaitable as an argument and returns an asyncio.Future object.

The Future object can then be awaited directly or passed to another task or
coroutine.

For example:

1 ...
2 # shield a task from cancellation
3 shielded = asyncio.shield(task)
4 # await the shielded task
5 await shielded

The returned Future can be canceled by calling the cancel() method.

If the inner task is running, the request will be reported as successful.

For example:

1 ...
2 # cancel a shielded task
3 was_canceld = shielded.cancel()

Any coroutines awaiting the Future object will raise an asyncio.CancelledError,

which may need to be handled.
https://superfastpython.com/python-asyncio/ 95/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 try:
3 # await the shielded task
4 await asyncio.shield(task)
5 except asyncio.CancelledError:
6 # ...

Importantly, the request for cancellation made on the Future object is not
propagated to the inner task.

This means that the request for cancellation is absorbed by the shield.

For example:

1 ...
2 # create a task
3 task = asyncio.create_task(coro())
4 # create a shield
5 shield = asyncio.shield(task)
6 # cancel the shield (does not cancel the task)
7 shield.cancel()

If a coroutine is provided to the asyncio.shield() function it is wrapped in an

asyncio.Task() and scheduled immediately.

This means that the shield does not need to be awaited for the inner coroutine to
run.

“ If aw is a coroutine it is automatically scheduled as a Task.

— COROUTINES AND TASKS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
TASK.HTML)

If the task that is being shielded is canceled, the cancellation request will be
propagated up to the shield, which will also be canceled.

For example:

https://superfastpython.com/python-asyncio/ 96/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # create a task
3 task = asyncio.create_task(coro())
4 # create a shield
5 shield = asyncio.shield(task)
6 # cancel the task (also cancels the shield)
7 task.cancel()

Now that we know how to use the asyncio.shield() function, let’s look at some worked
examples.

Example of Asyncio shield() for a Task

We can explore how to protect a task from cancellation using asyncio.shield().

In this example, we define a simple coroutine task that takes an integer argument,
sleeps for a second, then returns the argument. The coroutine can then be created
and scheduled as a Task.

We can define a second coroutine that takes a task, sleeps for a fraction of a second,
then cancels the provided task.

In the main coroutine, we can then shield the first task and pass it to the second task,
then await the shielded task.

The expectation is that the shield will be canceled and leave the inner task intact. The
cancellation will disrupt the main coroutine. We can check the status of the inner task
at the end of the program and we expect it to have been completed normally,
regardless of the request to cancel made on the shield.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 97/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of using asyncio shield to protect a task from cancellation
3 import asyncio
4
5 # define a simple asynchronous
6 async def simple_task(number):
7 # block for a moment
8 await asyncio.sleep(1)
9 # return the argument
10 return number
11
12 # cancel the given task after a moment
13 async def cancel_task(task):
14 # block for a moment
15 await asyncio.sleep(0.2)
16 # cancel the task
17 was_cancelled = task.cancel()
18 print(f'cancelled: {was_cancelled}')
19
20 # define a simple coroutine
21 async def main():
22 # create the coroutine
23 coro = simple_task(1)
24 # create a task
25 task = asyncio.create_task(coro)
26 # created the shielded task
27 shielded = asyncio.shield(task)
28 # create the task to cancel the previous task
29 asyncio.create_task(cancel_task(shielded))
30 # handle cancellation
31 try:
32 # await the shielded task
33 result = await shielded
34 # report the result
35 print(f'>got: {result}')
36 except asyncio.CancelledError:
37 print('shielded was cancelled')
38 # wait a moment
39 await asyncio.sleep(1)
40 # report the details of the tasks
41 print(f'shielded: {shielded}')
42 print(f'task: {task}')
43
44 # start
45 asyncio.run(main())

Running the example first creates the main() coroutine and uses it as the entry point
into the application.

The task coroutine is created, then it is wrapped and scheduled in a Task.

The task is then shielded from cancellation.

The shielded task is then passed to the cancel_task() coroutine which is wrapped in a
task and scheduled.

The main coroutine then awaits the shielded task, which expects a CancelledError
exception.
https://superfastpython.com/python-asyncio/ 98/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The task runs for a moment then sleeps. The cancellation task runs for a moment,
sleeps, resumes then cancels the shielded task. The request to cancel reports that it
was successful.

This raises a CancelledError exception in the shielded Future, although not in the
inner task.

The main() coroutine resumes and responds to the CancelledError exception,

reporting a message. It then sleeps for a while longer.

The task resumes, finishes, and returns a value.

Finally, the main() coroutine resumes, and reports the status of the shielded future
and the inner task. We can see that the shielded future is marked as canceled and yet
the inner task is marked as finished normally and provides a return value.

This example highlights how a shield can be used to successfully protect an inner task
from cancellation.

1 cancelled: True
2 shielded was cancelled
3 shielded: <Future cancelled>
4 task: <Task finished name='Task-2' coro=<simple_task() done, defined at ...> result=1>

You can learn more about the shield() function in the tutorial:

Asyncio Shield From Cancellation (https://superfastpython.com/asyncio-shield/)

Next, we will explore how to run a blocking task from an asyncio program.

Run a Blocking Task in Asyncio

A blocking task is a task that stops the current thread from progressing.

If a blocking task is executed in an asyncio program it stops the entire event loop,
preventing any other coroutines from progressing.
https://superfastpython.com/python-asyncio/ 99/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We can run blocking calls asynchronously in an asyncio program via the

asyncio.to_thread() and loop.run_in_executor() functions.

Need to Run Blocking Tasks in Asyncio

The focus of asyncio is asynchronous programming and non-blocking IO.

Nevertheless, we often need to execute a blocking function call within an asyncio

application.

This could be for many reasons, such as:

To execute a CPU-bound task like calculating something.

To execute a blocking IO-bound task like reading or writing from a file.
To call into a third-party library that does not support asyncio yet.

Making a blocking call directly in an asyncio program will cause the event loop to stop
while the blocking call is executing. It will not allow other coroutines to run in the
background.

How can we execute a blocking call in an asyncio program asynchronously?

How to Run Blocking Tasks

The asyncio module provides two approaches for executing blocking calls in asyncio
programs.

The first is to use the asyncio.to_thread() function

(https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread).

This is in the high-level API and is intended for application developers.

The asyncio.to_thread() function takes a function name to execute and any

arguments.
https://superfastpython.com/python-asyncio/ 100/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The function is executed in a separate thread. It returns a coroutine that can be

awaited or scheduled as an independent task.

For example:

1 ...
2 # execute a function in a separate thread
3 await asyncio.to_thread(task)

The task will not begin executing until the returned coroutine is given an opportunity
to run in the event loop.

The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes

to execute blocking calls.

As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks.

An alternative approach is to use the loop.run_in_executor() function

(https://docs.python.org/3/library/asyncio-
eventloop.html#asyncio.loop.run_in_executor).

This is in the low-level asyncio API and first requires access to the event loop, such as
via the asyncio.get_running_loop() function
(https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_running_loop).

The loop.run_in_executor() function takes an executor and a function to execute.

If None is provided for the executor, then the default executor is used, which is a
ThreadPoolExecutor.

The loop.run_in_executor() function returns an awaitable that can be awaited if

needed. The task will begin executing immediately, so the returned awaitable does
not need to be awaited or scheduled for the blocking call to start executing.

For example:

https://superfastpython.com/python-asyncio/ 101/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # get the event loop
3 loop = asyncio.get_running_loop()
4 # execute a function in a separate thread
5 await loop.run_in_executor(None, task)

Alternatively, an executor can be created and passed to the loop.run_in_executor()

function, which will execute the asynchronous call in the executor.

The caller must manage the executor in this case, shutting it down once the caller is
finished with it.

For example:

1 ...
2 # create a process pool
3 with ProcessPoolExecutor as exe:
4 # get the event loop
5 loop = asyncio.get_running_loop()
6 # execute a function in a separate thread
7 await loop.run_in_executor(exe, task)
8 # process pool is shutdown automatically...

These two approaches allow a blocking call to be executed as an asynchronous task

in an asyncio program.

Now that we know how to execute blocking calls in an asyncio program, let’s look at
some worked examples.

Example of Running I/O-Bound Task in Asyncio with

to_thread()
We can explore how to execute a blocking IO-bound call in an asyncio program using
asyncio.to_thread().

In this example, we will define a function that blocks the caller for a few seconds. We
will then execute this function asynchronously in a thread pool from asyncio using
the asyncio.to_thread() function.

This will free the caller to continue with other activities.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 102/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of running a blocking io-bound task in asyncio
3 import asyncio
4 import time
5
6 # a blocking io-bound task
7 def blocking_task():
8 # report a message
9 print('Task starting')
10 # block for a while
11 time.sleep(2)
12 # report a message
13 print('Task done')
14
15 # main coroutine
16 async def main():
17 # report a message
18 print('Main running the blocking task')
19 # create a coroutine for the blocking task
20 coro = asyncio.to_thread(blocking_task)
21 # schedule the task
22 task = asyncio.create_task(coro)
23 # report a message
24 print('Main doing other things')
25 # allow the scheduled task to start
26 await asyncio.sleep(0)
27 # await the task
28 await task
29
30 # run the asyncio program
31 asyncio.run(main())

Running the example first creates the main() coroutine and runs it as the entry point
into the asyncio program.

The main() coroutine runs and reports a message. It then issues a call to the blocking
function call to the thread pool. This returns a coroutine,

The coroutine is then wrapped in a Task and executed independently.

The main() coroutine is free to continue with other activities. In this case, it sleeps for
a moment to allow the scheduled task to start executing. This allows the target
function to be issued to the ThreadPoolExecutor behind the scenes and start
running.

The main() coroutine then suspends and waits for the task to complete.

The blocking function reports a message, sleeps for 2 seconds, then reports a final
message.

https://superfastpython.com/python-asyncio/ 103/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This highlights how we can execute a blocking IO-bound task in a separate thread
asynchronously from an asyncio program.

1 Main running the blocking task

2 Main doing other things
3 Task starting
4 Task done

You can learn more about the to_thread() function in the tutorial:

How to Use Asyncio to_thread() (https://superfastpython.com/asyncio-to_thread/)

Next, we will explore how to develop and use asynchronous iterators.

Asynchronous Iterators
Iteration is a basic operation in Python.

We can iterate lists, strings, and all manner of other structures.

Asyncio allows us to develop asynchronous iterators.

We can create and use asynchronous iterators in asyncio programs by defining an

object that implements the __aiter__() and __anext__() methods.

Let’s take a closer look.

What Are Asynchronous Iterators

An asynchronous iterator is an object that implements the __aiter__() and __anext__()
methods.

Before we take a close look at asynchronous iterators, let’s review classical iterators.

https://superfastpython.com/python-asyncio/ 104/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Iterators

An iterator is a Python object that implements a specific interface.

Specifically, the __iter__() method that returns an instance of the iterator and the
__next__() method that steps the iterator one cycle and returns a value.

“ iterator: An object representing a stream of data. Repeated calls to the

iterator’s __next__() method (or passing it to the built-in function next())
return successive items in the stream. When no more data are available a
StopIteration exception is raised instead.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

An iterator can be stepped using the next() built-in function or traversed using a for
loop.

Many Python objects are iterable, most notable are containers such as lists.

Asynchronous Iterators

An asynchronous iterator is a Python object that implements a specific interface.

“ asynchronous iterator: An object that implements the __aiter__() and

__anext__() methods.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

An asynchronous iterator must implement the __aiter__() and __anext__() methods.

The __aiter__() method must return an instance of the iterator.

The __anext__() method must return an awaitable that steps the iterator.
https://superfastpython.com/python-asyncio/ 105/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

An asynchronous iterator may only be stepped or traversed in an asyncio program,

such as within a coroutine.

Asynchronous iterators were introduced in PEP 492 – Coroutines with async and
await syntax (https://peps.python.org/pep-0492/).

An asynchronous iterator can be stepped using the anext() built-in function

(https://docs.python.org/3/library/functions.html#anext) that returns an awaitable
that executes one step of the iterator, e.g. one call to the __anext__() method.

An asynchronous iterator can be traversed using the “async for” expression that will
automatically call anext() each iteration and await the returned awaitable in order to
retrieve the return value.

“ An asynchronous iterable is able to call asynchronous code in its iter

implementation, and asynchronous iterator can call asynchronous code in
its next method.

— PEP 492 – COROUTINES WITH ASYNC AND AWAIT SYNTAX

(HTTPS://PEPS.PYTHON.ORG/PEP-0492/)

What is the “async for” loop?

The async for expression is used to traverse an asynchronous iterator.

It is an asynchronous for-loop statement.

An asynchronous iterator is an iterator that yields awaitables.

You may recall that an awaitable is an object that can be waited for, such as a
coroutine or a task.

https://superfastpython.com/python-asyncio/ 106/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ awaitable: An object that can be used in an await expression.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

An asynchronous generator will automatically implement the asynchronous iterator

methods, allowing it to be iterated like an asynchronous iterator.

The await for expression allows the caller to traverse an asynchronous iterator of
awaitables and retrieve the result from each.

This is not the same as traversing a collection or list of awaitables (e.g. coroutine
objects), instead, the awaitables returned must be provided using the expected
asynchronous iterator methods.

Internally, the async for loop will automatically resolve or await each awaitable,
scheduling coroutines as needed.

Because it is a for-loop, it assumes, although does not require, that each awaitable
being traversed yields a return value.

The async for loop must be used within a coroutine because internally it will use the
await expression, which can only be used within coroutines.

The async for expression can be used to traverse an asynchronous iterator within a
coroutine.

For example:

1 ...
2 # traverse an asynchronous iterator
3 async for item in async_iterator:
4 print(item)

This does not execute the for-loop in parallel. The asyncio is unable to execute more
than one coroutine at a time within a Python thread.

https://superfastpython.com/python-asyncio/ 107/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Instead, this is an asynchronous for-loop.

The difference is that the coroutine that executes the for loop will suspend and
internally await for each awaitable.

Behind the scenes, this may require coroutines to be scheduled and awaited, or tasks
to be awaited.

We may also use the async for expression in a list comprehension.

For example:

1 ...
2 # build a list of results
3 results = [item async for item async_iterator]

This would construct a list of return values from the asynchronous iterator.

Next, let’s look at how to define, create and use asynchronous iterators.

How to Use Asynchronous Iterators

In this section, we will take a close look at how to define, create, step, and traverse an
asynchronous iterator in asyncio programs.

Let’s start with how to define an asynchronous iterator.

Define an Asynchronous Iterator

We can define an asynchronous iterator by defining a class that implements the

__aiter__() and __anext__() methods.

These methods are defined on a Python object as per normal.

Importantly, because the __anext__() function must return an awaitable, it must be

defined using the “async def” expression.

https://superfastpython.com/python-asyncio/ 108/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ object.__anext__(self): Must return an awaitable resulting in a next value of

the iterator. Should raise a StopAsyncIteration error when the iteration is
over.

— ASYNCHRONOUS ITERATORS
(HTTPS://DOCS.PYTHON.ORG/3/REFERENCE/DATAMODEL.HTML#ASYNCHRONOUS -
ITERATORS)

When the iteration is complete, the __anext__() method must raise a

StopAsyncIteration exception.

For example:

1 # define an asynchronous iterator

2 class AsyncIterator():
3 # constructor, define some state
4 def __init__(self):
5 self.counter = 0
6
7 # create an instance of the iterator
8 def __aiter__(self):
9 return self
10
11 # return the next awaitable
12 async def __anext__(self):
13 # check for no further items
14 if self.counter >= 10:
15 raise StopAsyncIteration
16 # increment the counter
17 self.counter += 1
18 # return the counter value
19 return self.counter

Because the asynchronous iterator is a coroutine and each iterator returns an

awaitable that is scheduled and executed in the asyncio event loop, we can execute
and await awaitables within the body of the iterator.

For example:

https://superfastpython.com/python-asyncio/ 109/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # return the next awaitable
3 async def __anext__(self):
4 # check for no further items
5 if self.counter >= 10:
6 raise StopAsyncIteration
7 # increment the counter
8 self.counter += 1
9 # simulate work
10 await asyncio.sleep(1)
11 # return the counter value
12 return self.counter

Next, let’s look at how we might use an asynchronous iterator.

Create Asynchronous Iterator

To use an asynchronous iterator we must create the iterator.

This involves creating the Python object as per normal.

For example:

1 ...
2 # create the iterator
3 it = AsyncIterator()

This returns an “asynchronous iterable“, which is an instance of an “asynchronous

iterator“.

Step an Asynchronous Iterator

One step of the iterator can be traversed using the anext() built-in function
(https://docs.python.org/3/library/functions.html#anext), just like a classical iterator
using the next() function.

The result is an awaitable that is awaited.

For example:

1 ...
2 # get an awaitable for one step of the iterator
3 awaitable = anext(it)
4 # execute the one step of the iterator and get the result
5 result = await awaitable

https://superfastpython.com/python-asyncio/ 110/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This can be achieved in one step.

For example:

1 ...
2 # step the async iterator
3 result = await anext(it)

Traverse an Asynchronous Iterator

The asynchronous iterator can also be traversed in a loop using the “async for”
expression that will await each iteration of the loop automatically.

For example:

1 ...
2 # traverse an asynchronous iterator
3 async for result in AsyncIterator():
4 print(result)

You can learn more about the “async for” expression in the tutorial:

Asyncio async for loop (/asyncio-async-for/)

We may also use an asynchronous list comprehension with the “async for”
expression to collect the results of the iterator.

For example:

1 ...
2 # async list comprehension with async iterator
3 results = [item async for item in AsyncIterator()]

Example of an Asynchronous Iterator

We can explore how to traverse an asynchronous iterator using the “async for”
expression.

In this example, we will update the previous example to traverse the iterator to
completion using an “async for” loop.
https://superfastpython.com/python-asyncio/ 111/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This loop will automatically await each awaitable returned from the iterator, retrieve
the returned value, and make it available within the loop body so that in this case it
can be reported.

This is perhaps the most common usage pattern for asynchronous iterators.

The complete example is listed below.

1 # SuperFastPython.com
2 # example of an asynchronous iterator with async for loop
3 import asyncio
4
5 # define an asynchronous iterator
6 class AsyncIterator():
7 # constructor, define some state
8 def __init__(self):
9 self.counter = 0
10
11 # create an instance of the iterator
12 def __aiter__(self):
13 return self
14
15 # return the next awaitable
16 async def __anext__(self):
17 # check for no further items
18 if self.counter >= 10:
19 raise StopAsyncIteration
20 # increment the counter
21 self.counter += 1
22 # simulate work
23 await asyncio.sleep(1)
24 # return the counter value
25 return self.counter
26
27 # main coroutine
28 async def main():
29 # loop over async iterator with async for loop
30 async for item in AsyncIterator():
31 print(item)
32
33 # execute the asyncio program
34 asyncio.run(main())

Running the example first creates the main() coroutine and uses it as the entry point
into the asyncio program.

The main() coroutine runs and starts the for loop.

An instance of the asynchronous iterator is created and the loop automatically steps
it using the anext() function to return an awaitable. The loop then awaits the
awaitable and retrieves a value which is made available to the body of the loop where
it is reported.
https://superfastpython.com/python-asyncio/ 112/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This process is then repeated, suspending the main() coroutine, executing a step of
the iterator and suspending, and resuming the main() coroutine until the iterator is
exhausted.

Once the internal counter of the iterator reaches 10, a StopAsyncIteration is raised.
This does not terminate the program. Instead, it is expected and handled by the
“async for” expression and breaks the loop.

This highlights how an asynchronous iterator can be traversed using an async for
expression.

1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 10

You can learn more about async iterators in the tutorial:

Asynchronous Iterators in Python (https://superfastpython.com/asynchronous-

iterators/)

Next, we will explore asynchronous generators.

Asynchronous Generators
Generators are a fundamental part of Python.

A generator is a function that has at least one “yield” expression. They are functions
that can be suspended and resumed, just like coroutines.

In fact, Python coroutines are an extension of Python generators.

Asyncio allows us to develop asynchronous generators.

https://superfastpython.com/python-asyncio/ 113/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We can create an asynchronous generator by defining a coroutine that makes use of

the “yield” expression.

Let’s take a closer look.

What Are Asynchronous Generators

An asynchronous generator is a coroutine that uses the yield expression.

Before we dive into the details of asynchronous generators, let’s first review classical
Python generators.

Generators

A generator is a Python function that returns a value via a yield expression.

For example:

1 # define a generator
2 def generator():
3 for i in range(10):
4 yield i

The generator is executed to the yield expression, after which a value is returned.
This suspends the generator at that point. The next time the generator is executed it
is resumed from the point it was resumed and runs until the next yield expression.

“ generator: A function which returns a generator iterator. It looks like a

normal function except that it contains yield expressions for producing a
series of values usable in a for-loop or that can be retrieved one at a time
with the next() function.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

https://superfastpython.com/python-asyncio/ 114/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Technically, a generator function creates and returns a generator iterator. The

generator iterator executes the content of the generator function, yielding and
resuming as needed.

“ generator iterator: An object created by a generator function. Each yield

temporarily suspends processing, remembering the location execution state
[…] When the generator iterator resumes, it picks up where it left off …

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

A generator can be executed in steps by using the next() built-in function

(https://docs.python.org/3/library/functions.html#next).

For example:

1 ...
2 # create the generator
3 gen = generator()
4 # step the generator
5 result = next(gen)

Although, it is more common to iterate the generator to completion, such as using a

for-loop or a list comprehension.

For example:

1 ...
2 # traverse the generator and collect results
3 results = [item for item in generator()]

Next, let’s take a closer look at asynchronous generators.

Asynchronous Generators

An asynchronous generator is a coroutine that uses the yield expression.

Unlike a function generator, the coroutine can schedule and await other coroutines
and tasks.
https://superfastpython.com/python-asyncio/ 115/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ asynchronous generator: A function which returns an asynchronous

generator iterator. It looks like a coroutine function defined with async def
except that it contains yield expressions for producing a series of values
usable in an async for loop.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

Like a classical generator, an asynchronous generator function can be used to create

an asynchronous generator iterator that can be traversed using the built-in anext()
function, instead of the next() function.

“ asynchronous generator iterator: An object created by a asynchronous

generator function. This is an asynchronous iterator which when called using
the __anext__() method returns an awaitable object which will execute the
body of the asynchronous generator function until the next yield expression.

— PYTHON GLOSSARY (HTTPS://DOCS.PYTHON.ORG/3/GLOSSARY.HTML)

This means that the asynchronous generator iterator implements the __anext__()
method and can be used with the async for expression.

This means that each iteration of the generator is scheduled and executed as
awaitable. The “async for” expression will schedule and execute each iteration of the
generator, suspending the calling coroutine and awaiting the result.

You can learn more about the “async for” expression in the tutorial:

Asyncio async for loop (/asyncio-async-for/)

https://superfastpython.com/python-asyncio/ 116/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

How to Use an Asynchronous Generator

In this section, we will take a close look at how to define, create, step, and traverse an
asynchronous generator in asyncio programs.

Let’s start with how to define an asynchronous generator.

Define an Asynchronous Generator

We can define an asynchronous generator by defining a coroutine that has at least

one yield expression.

This means that the function is defined using the “async def” expression.

For example:

1 # define an asynchronous generator

2 async def async_generator():
3 for i in range(10)
4 yield i

Because the asynchronous generator is a coroutine and each iterator returns an

awaitable that is scheduled and executed in the asyncio event loop, we can execute
and await awaitables within the body of the generator.

For example:

1 # define an asynchronous generator that awaits

2 async def async_generator():
3 for i in range(10)
4 # suspend and sleep a moment
5 await asyncio.sleep(1)
6 # yield a value to the caller
7 yield i

Next, let’s look at how we might use an asynchronous generator.

Create Asynchronous Generator

To use an asynchronous generator we must create the generator.

https://superfastpython.com/python-asyncio/ 117/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This looks like calling it, but instead creates and returns an iterator object.

For example:

1 ...
2 # create the iterator
3 it = async_generator()

This returns a type of asynchronous iterator called an asynchronous generator

iterator.

Step an Asynchronous Generator

One step of the generator can be traversed using the anext() built-in function
(https://docs.python.org/3/library/functions.html#anext), just like a classical
generator using the next() function.

The result is an awaitable that is awaited.

For example:

1 ...
2 # get an awaitable for one step of the generator
3 awaitable = anext(gen)
4 # execute the one step of the generator and get the result
5 result = await awaitable

This can be achieved in one step.

For example:

1 ...
2 # step the async generator
3 result = await anext(gen)

Traverse an Asynchronous Generator

The asynchronous generator can also be traversed in a loop using the “async for”
expression that will await each iteration of the loop automatically.

For example:
https://superfastpython.com/python-asyncio/ 118/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # traverse an asynchronous generator
3 async for result in async_generator():
4 print(result)

You can learn more about the “async for” expression in the tutorial:

We may also use an asynchronous list comprehension with the “async for”
expression to collect the results of the generator.

For example:

1 ...
2 # async list comprehension with async generator
3 results = [item async for item in async_generator()]

Example of an Asynchronous Generator

We can explore how to traverse an asynchronous generator using the “async for”
expression.

In this example, we will update the previous example to traverse the generator to
completion using an “async for” loop.

This loop will automatically await each awaitable returned from the generator,
retrieve the yielded value, and make it available within the loop body so that in this
case it can be reported.

This is perhaps the most common usage pattern for asynchronous generators.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 119/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of asynchronous generator with async for loop
3 import asyncio
4
5 # define an asynchronous generator
6 async def async_generator():
7 # normal loop
8 for i in range(10):
9 # block to simulate doing work
10 await asyncio.sleep(1)
11 # yield the result
12 yield i
13
14 # main coroutine
15 async def main():
16 # loop over async generator with async for loop
17 async for item in async_generator():
18 print(item)
19
20 # execute the asyncio program
21 asyncio.run(main())

Running the example first creates the main() coroutine and uses it as the entry point
into the asyncio program.

The main() coroutine runs and starts the for loop.

An instance of the asynchronous generator is created and the loop automatically

steps it using the anext() function to return an awaitable. The loop then awaits the
awaitable and retrieves a value which is made available to the body of the loop where
it is reported.

This process is then repeated, suspending the main() coroutine, executing an

iteration of the generator, and suspending, and resuming the main() coroutine until
the generator is exhausted.

This highlights how an asynchronous generator can be traversed using an async for
expression.

1 0
2 1
3 2
4 3
5 4
6 5
7 6
8 7
9 8
10 9

https://superfastpython.com/python-asyncio/ 120/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

You can learn more about async generators in the tutorial:

Asynchronous Generators in Python

(https://superfastpython.com/asynchronous-generators-in-python/)

Next, we will explore asynchronous context managers.

Asynchronous Context Managers

A context manager is a Python construct that provides a try-finally like environment
with a consistent interface and handy syntax, e.g. via the “with” expression.

It is commonly used with resources, ensuring the resource is always closed or

released after we are finished with it, regardless of whether the usage of the
resources was successful or failed with an exception.

Asyncio allows us to develop asynchronous context managers.

We can create and use asynchronous context managers in asyncio programs by

defining an object that implements the __aenter__() and __aexit__() methods as
coroutines.

Let’s take a closer look.

What is an Asynchronous Context Manager

An asynchronous context manager is a Python object that implements the
__aenter__() and __aexit__() methods.

Before we dive into the details of asynchronous context managers, let’s review
classical context managers.

https://superfastpython.com/python-asyncio/ 121/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Context Manager

A context manager is a Python object that implements the __enter__() and __exit__()
methods.

“ A context manager is an object that defines the runtime context to be

established when executing a with statement. The context manager handles
the entry into, and the exit from, the desired runtime context for the
execution of the block of code.

— WITH STATEMENT CONTEXT MANAGERS

(HTTPS://DOCS.PYTHON.ORG/3/REFERENCE/DATAMODEL.HTML#CONTEXT-
MANAGERS)

The __enter__() method defines what happens at the beginning of a block, such
as opening or preparing resources, like a file, socket or thread pool.
The __exit__() method defines what happens when the block is exited, such as
closing a prepared resource.

“ Typical uses of context managers include saving and restoring various kinds
of global state, locking and unlocking resources, closing opened files, etc.

— WITH STATEMENT CONTEXT MANAGERS

(HTTPS://DOCS.PYTHON.ORG/3/REFERENCE/DATAMODEL.HTML#CONTEXT-
MANAGERS)

A context manager is used via the “with” expression.

Typically the context manager object is created in the beginning of the “with”
expression and the __enter__() method is called automatically. The body of the
content makes use of the resource via the named context manager object, then the
__aexit__() method is called automatically when the block is exited, normally or via an
exception.
https://superfastpython.com/python-asyncio/ 122/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # open a context manager
3 with ContextManager() as manager:
4 # ...
5 # closed automatically

This mirrors a try-finally expression.

For example:

1 ...
2 # create the object
3 manager = ContextManager()
4 try:
5 manager.__enter__()
6 # ...
7 finally:
8 manager.__exit__()

Next, let’s take a look at asynchronous context managers.

Asynchronous Context Manager

Asynchronous context managers were introduced in “PEP 492 – Coroutines with

async and await syntax (https://peps.python.org/pep-0492/)“.

They provide a context manager that can be suspended when entering and exiting.

“ An asynchronous context manager is a context manager that is able to

suspend execution in its __aenter__ and __aexit__ methods.

— ASYNCHRONOUS CONTEXT MANAGERS

(HTTPS://DOCS.PYTHON.ORG/3/REFERENCE/DATAMODEL.HTML#ASYNCHRONOUS -
CONTEXT-MANAGERS)

The __aenter__ and __aexit__ methods are defined as coroutines and are awaited by
the caller.

This is achieved using the “async with” expression.

https://superfastpython.com/python-asyncio/ 123/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

You can learn more about the “async with” expression in the tutorial:

What is Asyncio async with (/asyncio-async-with/)

As such, asynchronous context managers can only be used within asyncio programs,
such as within calling coroutines.

What is “async with”

The “async with” expression is for creating and using asynchronous context
managers.

It is an extension of the “with” expression for use in coroutines within asyncio

programs.

The “async with” expression is just like the “with” expression used for context
managers, except it allows asynchronous context managers to be used within
coroutines.

In order to better understand “async with“, let’s take a closer look at asynchronous
context managers.

The async with expression allows a coroutine to create and use an asynchronous
version of a context manager.

For example:

1 ...
2 # create and use an asynchronous context manager
3 async with AsyncContextManager() as manager:
4 # ...

This is equivalent to something like:

https://superfastpython.com/python-asyncio/ 124/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # create or enter the async context manager
3 manager = await AsyncContextManager()
4 try:
5 # ...
6 finally:
7 # close or exit the context manager
8 await manager.close()

Notice that we are implementing much the same pattern as a traditional context
manager, except that creating and closing the context manager involve awaiting
coroutines.

This suspends the execution of the current coroutine, schedules a new coroutine and
waits for it to complete.

As such an asynchronous context manager must implement the __aenter__() and

__aexit__() methods that must be defined via the async def expression. This makes
them coroutines themselves which may also await.

How to Use Asynchronous Context Managers

In this section, we will explore how we can define, create, and use asynchronous
context managers in our asyncio programs.

Define an Asynchronous Context Manager

We can define an asynchronous context manager as a Python object that implements

the __aenter__() and __aexit__() methods.

Importantly, both methods must be defined as coroutines using the “async def” and
therefore must return awaitables.

For example:

https://superfastpython.com/python-asyncio/ 125/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # define an asynchronous context manager
2 class AsyncContextManager:
3 # enter the async context manager
4 async def __aenter__(self):
5 # report a message
6 print('>entering the context manager')
7
8 # exit the async context manager
9 async def __aexit__(self, exc_type, exc, tb):
10 # report a message
11 print('>exiting the context manager')

Because each of the methods are coroutines, they may themselves await coroutines
or tasks.

For example:

1 # define an asynchronous context manager

2 class AsyncContextManager:
3 # enter the async context manager
4 async def __aenter__(self):
5 # report a message
6 print('>entering the context manager')
7 # block for a moment
8 await asyncio.sleep(0.5)
9
10 # exit the async context manager
11 async def __aexit__(self, exc_type, exc, tb):
12 # report a message
13 print('>exiting the context manager')
14 # block for a moment
15 await asyncio.sleep(0.5)

Use an Asynchronous Context Manager

An asynchronous context manager is used via the “async with” expression.

This will automatically await the enter and exit coroutines, suspending the calling
coroutine as needed.

For example:

1 ...
2 # use an asynchronous context manager
3 async with AsyncContextManager() as manager:
4 # ...

As such, the “async with” expression and asynchronous context managers more
generally can only be used within asyncio programs, such as within coroutines.

https://superfastpython.com/python-asyncio/ 126/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Now that we know how to use asynchronous context managers, let’s look at a worked
example.

Example of an Asynchronous Context Manager and “async

with”
We can explore how to use an asynchronous context manager via the “async with”
expression.

In this example, we will update the above example to use the context manager in a
normal manner.

We will use an “async with” expression and on one line, create and enter the context
manager. This will automatically await the enter method.

We can then make use of the manager within the inner block. In this case, we will just
report a message.

Exiting the inner block will automatically await the exit method of the context
manager.

Contrasting this example with the previous example shows how much heavy lifting
the “async with” expression does for us in an asyncio program.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 127/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of an asynchronous context manager via async with
3 import asyncio
4
5 # define an asynchronous context manager
6 class AsyncContextManager:
7 # enter the async context manager
8 async def __aenter__(self):
9 # report a message
10 print('>entering the context manager')
11 # block for a moment
12 await asyncio.sleep(0.5)
13
14 # exit the async context manager
15 async def __aexit__(self, exc_type, exc, tb):
16 # report a message
17 print('>exiting the context manager')
18 # block for a moment
19 await asyncio.sleep(0.5)
20
21 # define a simple coroutine
22 async def custom_coroutine():
23 # create and use the asynchronous context manager
24 async with AsyncContextManager() as manager:
25 # report the result
26 print(f'within the manager')
27
28 # start the asyncio program
29 asyncio.run(custom_coroutine())

Running the example first creates the main() coroutine and uses it as the entry point
into the asyncio program.

The main() coroutine runs and creates an instance of our AsyncContextManager

class in an “async with” expression.

This expression automatically calls the enter method and awaits the coroutine. A
message is reported and the coroutine blocks for a moment.

The main() coroutine resumes and executes the body of the context manager,
printing a message.

The block is exited and the exit method of the context manager is awaited
automatically, reporting a message and sleeping a moment.

This highlights the normal usage pattern for an asynchronous context manager in an
asyncio program.

1 >entering the context manager

2 within the manager
3 >exiting the context manager

https://superfastpython.com/python-asyncio/ 128/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

You can learn more about async context managers in the tutorial:

Asynchronous Context Managers in Python

(https://superfastpython.com/asynchronous-context-manager/)

Next, we will explore asynchronous comprehensions.

Asynchronous Comprehensions
Comprehensions, like list and dict comprehensions are one feature of Python when
we think of “pythonic“.

It is a way we do loops that is different to many other languages.

Asyncio allows us to use asynchronous comprehensions.

We can traverse an asynchronous generators and asynchronous iterators using an

asynchronous comprehension via the “async for” expression.

Let’s take a closer look.

What are Asynchronous Comprehensions

An async comprehension is an asynchronous version of a classical comprehension.

Asyncio supports two types of asynchronous comprehensions, they are the “async
for” comprehension and the “await” comprehension.

https://superfastpython.com/python-asyncio/ 129/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ PEP 530 adds support for using async for in list, set, dict comprehensions
and generator expressions

— PEP 530: ASYNCHRONOUS COMPREHENSIONS, WHAT’S NEW IN PYTHON 3.6

(HTTPS://DOCS.PYTHON.ORG/3/WHATSNEW/3.6.HTML#PEP-530-ASYNCHRONOUS -
COMPREHENSIONS).

Before we look at each, let’s first recall classical comprehensions.

Comprehensions
Comprehensions allow data collections like lists, dicts, and sets to be created in a
concise way.

“ List comprehensions provide a concise way to create lists.

— LIST COMPREHENSIONS
(HTTPS://DOCS.PYTHON.ORG/3/TUTORIAL/DATASTRUCTURES.HTML#LIST-
COMPREHENSIONS)

A list comprehension allows a list to be created from a for expression within the new
list expression.

For example:

1 ...
2 # create a list using a list comprehension
3 result = [a*2 for a in range(100)]

Comprehensions are also supported for creating dicts and sets.

For example:

https://superfastpython.com/python-asyncio/ 130/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # create a dict using a comprehension
3 result = {a:i for a,i in zip(['a','b','c'],range(3))}
4 # create a set using a comprehension
5 result = {a for a in [1, 2, 3, 2, 3, 1, 5, 4]}

Asynchronous Comprehensions
An asynchronous comprehension allows a list, set, or dict to be created using the
“async for” expression with an asynchronous iterable.

“ We propose to allow using async for inside list, set and dict comprehensions.
— PEP 530 – ASYNCHRONOUS COMPREHENSIONS (HTTPS://PEPS.PYTHON.ORG/PEP-
0530/)

For example:

1 ...
2 # async list comprehension with an async iterator
3 result = [a async for a in aiterable]

This will create and schedule coroutines or tasks as needed and yield their results
into a list.

Recall that the “async for” expression (/asyncio-async-for/) may only be used within
coroutines and tasks.

Also, recall that an asynchronous iterator is an iterator that yields awaitables.

The “async for” expression allows the caller to traverse an asynchronous iterator of
awaitables and retrieve the result from each.

Internally, the async for loop will automatically resolve or await each awaitable,
scheduling coroutines as needed.

An async generator automatically implements the methods for the async iterator and
may also be used in an asynchronous comprehension.
https://superfastpython.com/python-asyncio/ 131/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # async list comprehension with an async generator
3 result = [a async for a in agenerator]

Await Comprehensions
The “await” expression may also be used within a list, set, or dict comprehension,
referred to as an await comprehension.

“ We propose to allow the use of await expressions in both asynchronous and

synchronous comprehensions

— PEP 530 – ASYNCHRONOUS COMPREHENSIONS (HTTPS://PEPS.PYTHON.ORG/PEP-

0530/)

Like an async comprehension, it may only be used within an asyncio coroutine or

task.

This allows a data structure, like a list, to be created by suspending and awaiting a
series of awaitables.

For example:

1 ...
2 # await list compression with a collection of awaitables
3 results = [await a for a in awaitables]

This will create a list of results by awaiting each awaitable in turn.

The current coroutine will be suspended to execute awaitables sequentially, which is

different and perhaps slower than executing them concurrently using
asyncio.gather().

https://superfastpython.com/python-asyncio/ 132/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

You can learn more about async comprehensions in the tutorial:

Asynchronous Comprehensions in Python

(https://superfastpython.com/asynchronous-comprehensions/)

Next, we will explore how to run commands using subprocesses from asyncio.

Run Commands in Non-Blocking

Subprocesses
We can execute commands from asyncio.

The command will run in a subprocess that we can write to and read from using non-
blocking I/O.

Let’s take a closer look.

What is asyncio.subprocess.Process
The asyncio.subprocess.Process class (https://docs.python.org/3/library/asyncio-
subprocess.html#asyncio.subprocess.Process) provides a representation of a
subprocess run by asyncio.

It provides a handle on a subprocess in asyncio programs, allowing actions to be

performed on it, such as waiting and terminating it.

“ Process is a high-level wrapper that allows communicating with

subprocesses and watching for their completion.

— INTERACTING WITH SUBPROCESSES

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
SUBPROCESS.HTML#INTERACTING-WITH-SUBPROCESSES)

https://superfastpython.com/python-asyncio/ 133/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The API is very similar to the multiprocessing.Process class

(https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process)
and perhaps more so with the subprocess.Popen class
(https://docs.python.org/3/library/subprocess.html#subprocess.Popen).

Specifically, it shares methods such as wait(), communicate(), and send_signal() and

attributes such as stdin, stdout, and stderr with the subprocess.Popen.

Now that we know what the asyncio.subprocess.Process class is, let’s look at how
we might use it in our asyncio programs.

We do not create a asyncio.subprocess.Process directly.

Instead, an instance of the class is created for us when executing a subprocess in an

asyncio program.

“ An object that wraps OS processes created by the create_subprocess_exec()

and create_subprocess_shell() functions.

— INTERACTING WITH SUBPROCESSES

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
SUBPROCESS.HTML#INTERACTING-WITH-SUBPROCESSES)

There are two ways to execute an external program as a subprocess and acquire a
Process instance, they are:

asyncio.create_subprocess_exec() for running commands directly.

asyncio.create_subprocess_shell() for running commands via the shell.

Let’s look at examples of each in turn.

https://superfastpython.com/python-asyncio/ 134/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

How to Run a Command Directly

A command (https://en.wikipedia.org/wiki/Command-line_interface) is a program
executed on the command line (terminal or command prompt). It is another program
that is run directly.

Common examples on Linux and macOS might be:

‘ls‘ to list the contents of a directory

‘cat‘ to report the content of a file
‘date‘ to report the date
‘echo‘ to report back a string
‘sleep‘ to sleep for a number of seconds

And so on.

We can execute a command from an asyncio program via the

create_subprocess_exec() function.

The asyncio.create_subprocess_exec() function takes a command and executes it

directly.

This is helpful as it allows the command to be executed in a subprocess and for

asyncio coroutines to read, write, and wait for it.

“ Because all asyncio subprocess functions are asynchronous and asyncio

provides many tools to work with such functions, it is easy to execute and
monitor multiple subprocesses in parallel.

— ASYNCIO SUBPROCESSES (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

SUBPROCESS.HTML)

https://superfastpython.com/python-asyncio/ 135/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Unlike the asyncio.create_subprocess_shell() function, the

asyncio.create_subprocess_exec() will not execute the command using the shell.

This means that the capabilities provided by the shell, such as shell variables,
scripting, and wildcards are not available when executing the command.

It also means that executing the command may be more secure as there is no
opportunity for a shell injection
(https://en.wikipedia.org/wiki/Code_injection#Shell_injection).

Now that we know what asyncio.create_subprocess_exec() does, let’s look at how to

use it.

How to Use Asyncio create_subprocess_exec()

The asyncio.create_subprocess_exec() function

(https://docs.python.org/3/library/asyncio-
subprocess.html#asyncio.create_subprocess_exec) will execute a given string
command in a subprocess.

It returns a asyncio.subprocess.Process object

(https://docs.python.org/3/library/asyncio-
subprocess.html#asyncio.subprocess.Process) that represents the subprocess.

“ Process is a high-level wrapper that allows communicating with

subprocesses and watching for their completion.

— INTERACTING WITH SUBPROCESSES

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
SUBPROCESS.HTML#ASYNCIO.SUBPROCESS.PROCESS)

https://superfastpython.com/python-asyncio/ 136/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The create_subprocess_exec() function is a coroutine, which means we must await

it. It will return once the subprocess has been started, not when the subprocess is
finished.

For example:

1 ...
2 # execute a command in a subprocess
3 process = await asyncio.create_subprocess_exec('ls')

Arguments to the command being executed must be provided as subsequent

arguments to the create_subprocess_exec() function.

For example:

1 ...
2 # execute a command with arguments in a subprocess
3 process = await asyncio.create_subprocess_exec('ls', '-l')

We can wait for the subprocess to finish by awaiting the wait() method.

For example:

1 ...
2 # wait for the subprocess to terminate
3 await process.wait()

We can stop the subprocess directly by calling the terminate() or kill() methods,
which will raise a signal in the subprocess.

For example:

1 ...
2 # terminate the subprocess
3 process.terminate()

The input and output of the command will be handled by stdin, stderr, and stdout.

We can have the asyncio program handle the input or output for the subprocess.

This can be achieved by specifying the input or output stream and specifying a
constant to redirect, such as asyncio.subprocess.PIPE.

https://superfastpython.com/python-asyncio/ 137/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example, we can redirect the output of a command to the asyncio program:

1 ...
2 # start a subprocess and redirect output
3 process = await asyncio.create_subprocess_exec('ls', stdout=asyncio.subprocess.PIPE)

We can then read the output of the program via the asyncio.subprocess.Process
instance via the communicate() method.

This method is a coroutine and must be awaited. It is used to both send and receive
data with the subprocess.

For example:

1 ...
2 # read data from the subprocess
3 line = process.communicate()

We can also send data to the subprocess via the communicate() method by setting
the “input” argument in bytes.

For example:

1 ...
2 # start a subprocess and redirect input
3 process = await asyncio.create_subprocess_exec('ls', stdin=asyncio.subprocess.PIPE)
4 # send data to the subprocess
5 process.communicate(input=b'Hello\n')

Behind the scenes the asyncio.subprocess.PIPE configures the subprocess to point

to a StreamReader or StreamWriter for sending data to or from the subprocess,
and the communicate() method will read or write bytes from the configured reader.

“ If PIPE is passed to stdin argument, the Process.stdin attribute will point to a

StreamWriter instance. If PIPE is passed to stdout or stderr arguments, the
Process.stdout and Process.stderr attributes will point to StreamReader
instances.

— ASYNCIO SUBPROCESSES (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

SUBPROCESS.HTML)

https://superfastpython.com/python-asyncio/ 138/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We can interact with the StreamReader or StreamWriter directly via the subprocess
via the stdin, stdout, and stderr attributes.

For example:

1 ...
2 # read a line from the subprocess output stream
3 line = await process.stdout.readline()

Now that we know how to use the create_subprocess_exec() function, let’s look at
some worked examples.

Example of Asyncio create_subprocess_exec()

We can explore how to run a command in a subprocess from asyncio.

In this example, we will execute the “echo” command

(https://en.wikipedia.org/wiki/Echo_(command)) to report back a string.

The echo command will report the provided string on standard output directly.

The complete example is listed below.

Note, this example assumes you have access to the “echo” command, I’m not sure it
will work on Windows.

1 # SuperFastPython.com
2 # example of executing a command as a subprocess with asyncio
3 import asyncio
4
5 # main coroutine
6 async def main():
7 # start executing a command in a subprocess
8 process = await asyncio.create_subprocess_exec('echo', 'Hello World')
9 # report the details of the subprocess
10 print(f'subprocess: {process}')
11
12 # entry point
13 asyncio.run(main())

Running the example first creates the main() coroutine and executes it as the entry
point into the asyncio program.

https://superfastpython.com/python-asyncio/ 139/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The main() coroutine runs and calls the create_subprocess_exec() function to

execute a command.

The main() coroutine suspends while the subprocess is created. A Process instance is
returned.

The main() coroutine resumes and reports the details of the subprocess. The main()
process terminates and the asyncio program terminates.

The output of the echo command is reported on the command line.

This highlights how we can execute a command from an asyncio program.

1 Hello World
2 subprocess: <Process 50249>

How to Run a Command Via the Shell

We can execute commands using the shell
(https://en.wikipedia.org/wiki/Shell_(computing)).

The shell is a user interface for the command line, called a command line interpreter
(CLI).

It will interpret and execute commands on behalf of the user.

It also offers features such as a primitive programming language for scripting,

wildcards, piping, shell variables (e.g. PATH), and more.

For example, we can redirect the output of one command as input to another
command, such as the contents of the “/etc/services” file into the word count “wc”
command and count the number of lines:

1 cat /etc/services | wc -l

https://superfastpython.com/python-asyncio/ 140/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Examples of shells in the Unix based operating systems include:

‘sh’
‘bash’
‘zsh’
And so on.

On Windows, the shell is probably cmd.exe (https://en.wikipedia.org/wiki/Cmd.exe).

See this great list of command line shells:

List of command-line interpreters, Wikipedia

(https://en.wikipedia.org/wiki/List_of_command-line_interpreters)

The shell is already running, it was used to start the Python program.

You don’t need to do anything special to get or have access to the shell.

We can execute a command from an asyncio program via the

create_subprocess_shell() function.

The asyncio.create_subprocess_shell() function takes a command and executes it

using the current user shell.

This is helpful as it not only allows the command to be executed, but allows the
capabilities of the shell to be used, such as redirection, wildcards and more.

https://superfastpython.com/python-asyncio/ 141/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ … the specified command will be executed through the shell. This can be
useful if you are using Python primarily for the enhanced control flow it
offers over most system shells and still want convenient access to other shell
features such as shell pipes, filename wildcards, environment variable
expansion, and expansion of ~ to a user’s home directory.

— SUBPROCESS — SUBPROCESS MANAGEMENT

(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/SUBPROCESS.HTML)

The command will be executed in a subprocess of the process executing the asyncio
program.

Importantly, the asyncio program is able to interact with the subprocess

asynchronously, e.g. via coroutines.

“ Because all asyncio subprocess functions are asynchronous and asyncio

provides many tools to work with such functions, it is easy to execute and
monitor multiple subprocesses in parallel.

— ASYNCIO SUBPROCESSES (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

SUBPROCESS.HTML)

There can be security considerations when executing a command via the shell instead
of directly.

This is because there is at least one level of indirection and interpretation between
the request to execute the command and the command being executed, allowing
possible malicious injection.

https://superfastpython.com/python-asyncio/ 142/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Important It is the application’s responsibility to ensure that all whitespace

and special characters are quoted appropriately to avoid shell injection
vulnerabilities.

— ASYNCIO SUBPROCESSES (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

SUBPROCESS.HTML)

Now that we know what asyncio.create_subprocess_shell() does, let’s look at how to

use it.

How to Use Asyncio create_subprocess_shell()

The asyncio.create_subprocess_shell() function

(https://docs.python.org/3/library/asyncio-
subprocess.html#asyncio.create_subprocess_shell) will execute a given string
command via the current shell.

It returns a asyncio.subprocess.Process object

(https://docs.python.org/3/library/asyncio-
subprocess.html#asyncio.subprocess.Process) that represents the process.

It is very similar to the create_subprocess_shell() function we saw in a previous

section. Nevertheless, we will review how to use the function and interact with the
process via the Process instance (in case you skipped straight to this section).

The create_subprocess_shell() function is a coroutine, which means we must await

it. It will return once the subprocess has been started, not when the subprocess is
finished.

For example:

1 ...
2 # start a subprocess
3 process = await asyncio.create_subprocess_shell('ls')

https://superfastpython.com/python-asyncio/ 143/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We can wait for the subprocess to finish by awaiting the wait() method.

For example:

1 ...
2 # wait for the subprocess to terminate
3 await process.wait()

We can stop the subprocess directly by calling the terminate() or kill() methods,
which will raise a signal in the subprocess.

The input and output of the command will be handled by the shell, e.g. stdin, stderr,
and stdout.

We can have the asyncio program handle the input or output for the subprocess.

This can be achieved by specifying the input or output stream and specifying a
constant to redirect, such as asyncio.subprocess.PIPE.

For example, we can redirect the output of a command to the asyncio program:

1 ...
2 # start a subprocess and redirect output
3 process = await asyncio.create_subprocess_shell('ls', stdout=asyncio.subprocess.PIPE)

We can then read the output of the program via the asyncio.subprocess.Process
instance via the communicate() method.

This method is a coroutine and must be awaited. It is used to both send and receive
data with the subprocess.

For example:

1 ...
2 # read data from the subprocess
3 line = process.communicate()

We can also send data to the subprocess via the communicate() method by setting
the “input” argument in bytes.

https://superfastpython.com/python-asyncio/ 144/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # start a subprocess and redirect input
3 process = await asyncio.create_subprocess_shell('ls', stdin=asyncio.subprocess.PIPE)
4 # send data to the subprocess
5 process.communicate(input=b'Hello\n')

Behind the scenes the asyncio.subprocess.PIPE configures the subprocess to point

to a StreamReader or StreamWriter for sending data to or from the subprocess,
and the communicate() method will read or write bytes from the configured reader.

“ If PIPE is passed to stdin argument, the Process.stdin attribute will point to a

StreamWriter instance. If PIPE is passed to stdout or stderr arguments, the
Process.stdout and Process.stderr attributes will point to StreamReader
instances.

— ASYNCIO SUBPROCESSES (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

SUBPROCESS.HTML)

We can interact with the StreamReader or StreamWriter directly via the subprocess
via the stdin, stdout, and stderr attributes.

For example:

1 ...
2 # read a line from the subprocess output stream
3 line = await process.stdout.readline()

Now that we know how to use the create_subprocess_shell() function, let’s look at
some worked examples.

Example of Asyncio create_subprocess_shell()

We can explore how to run a command in a subprocess from asyncio using the shell.

In this example, we will execute the “echo” command

(https://en.wikipedia.org/wiki/Echo_(command)) to report back a string.
https://superfastpython.com/python-asyncio/ 145/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The echo command will report the provided string on standard output directly.

The complete example is listed below.

Note, this example assumes you have access to the “echo” command, I’m not sure it
will work on Windows.

1 # SuperFastPython.com
2 # example of executing a shell command as a subprocess with asyncio
3 import asyncio
4
5 # main coroutine
6 async def main():
7 # start executing a shell command in a subprocess
8 process = await asyncio.create_subprocess_shell('echo Hello World')
9 # report the details of the subprocess
10 print(f'subprocess: {process}')
11
12 # entry point
13 asyncio.run(main())

Running the example first creates the main() coroutine and executes it as the entry
point into the asyncio program.

The main() coroutine runs and calls the create_subprocess_shell() function to

execute a command.

The main() coroutine suspends while the subprocess is created. A Process instance is
returned.

The main() coroutine resumes and reports the details of the subprocess. The main()
process terminates and the asyncio program terminates.

The output of the echo command is reported on the command line.

This highlights how we can execute a command using the shell from an asyncio
program.

1 subprocess: <Process 43916>

2 Hello World

https://superfastpython.com/python-asyncio/ 146/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Non-Blocking Streams
A major benefit of asyncio is the ability to use non-blocking streams.

Let’s take a closer look.

Asyncio Streams
Asyncio provides non-blocking I/O socket programming.

This is provided via streams.

“ Streams are high-level async/await-ready primitives to work with network

connections. Streams allow sending and receiving data without using
callbacks or low-level protocols and transports.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

Sockets can be opened that provide access to a stream writer and a stream writer.

Data can then be written and read from the stream using coroutines, suspending
when appropriate.

Once finished, the socket can be closed.

The asyncio streams capability is low-level meaning that any protocols required must
be implemented manually.

This might include common web protocols, such as:

HTTP or HTTPS for interacting with web servers

SMTP for interacting with email servers
https://superfastpython.com/python-asyncio/ 147/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

FTP for interacting with file servers.

The streams can also be used to create a server to handle requests using a standard
protocol, or to develop your own application-specific protocol.

Now that we know what asyncio streams are, let’s look at how to use them.

How to Open a Connection

An asyncio TCP client socket connection can be opened using the
asyncio.open_connection() function (https://docs.python.org/3/library/asyncio-
stream.html#asyncio.open_connection).

“ Establish a network connection and return a pair of (reader, writer) objects.

The returned reader and writer objects are instances of StreamReader and
StreamWriter classes.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

This is a coroutine that must be awaited and will return once the socket connection is
open.

The function returns a StreamReader and StreamWriter object for interacting with
the socket.

For example:

1 ...
2 # open a connection
3 reader, writer = await asyncio.open_connection(...)

The asyncio.open_connection() function takes many arguments in order to

configure the socket connection.

https://superfastpython.com/python-asyncio/ 148/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The two required arguments are the host and the port.

The host is a string that specifies the server to connect to, such as a domain name or
an IP address.

The port is the socket port number, such as 80 for HTTP servers, 443 for HTTPS
servers, 23 for SMTP and so on.

For example:

1 ...
2 # open a connection to an http server
3 reader, writer = await asyncio.open_connection('www.google.com', 80)

Encrypted socket connections are supported over the SSL protocol.

The most common example might be HTTPS which is replacing HTTP.

This can be achieved by setting the “ssl” argument to True.

For example:

1 ...
2 # open a connection to an https server
3 reader, writer = await asyncio.open_connection('www.google.com', 443, ssl=True)

How to Start a Server

An asyncio TCP server socket can be opened using the asyncio.start_server()
function (https://docs.python.org/3/library/asyncio-
stream.html#asyncio.start_server).

“ Create a TCP server (socket type SOCK_STREAM) listening on port of the host
address.

— ASYNCIO EVENT LOOP (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

EVENTLOOP.HTML)

https://superfastpython.com/python-asyncio/ 149/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This is a coroutine that must be awaited.

The function returns an asyncio.Server object

(https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server) that
represents the running server.

For example:

1 ...
2 # start a tcp server
3 server = await asyncio.start_server(...)

The three required arguments are the callback function, the host, and the port.

The callback function is a custom function specified by name that will be called each
time a client connects to the server.

“ The client_connected_cb callback is called whenever a new client connection

is established. It receives a (reader, writer) pair as two arguments, instances
of the StreamReader and StreamWriter classes.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

The host is the domain name or IP address that clients will specify to connect. The
port is the socket port number on which to receive connections, such as 21 for FTP or
80 for HTTP.

For example:

1 # handle connections
2 async def handler(reader, writer):
3 # ...
4
5 ...
6 # start a server to receive http connections
7 server = await asyncio.start_server(handler, '127.0.0.1', 80)

https://superfastpython.com/python-asyncio/ 150/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

How to Write Data with the StreamWriter

We can write data to the socket using an asyncio.StreamWriter
(https://docs.python.org/3/library/asyncio-stream.html#streamwriter).

“ Represents a writer object that provides APIs to write data to the IO stream.
— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
STREAM.HTML)

Data is written as bytes.

Byte data can be written to the socket using the write() method
(https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamWriter.write).

“ The method attempts to write the data to the underlying socket immediately.
If that fails, the data is queued in an internal write buffer until it can be sent.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

For example:

1 ...
2 # write byte data
3 writer.write(byte_data)

Alternatively, multiple “lines” of byte data organized into a list or iterable can be
written using the writelines() method (https://docs.python.org/3/library/asyncio-
stream.html#asyncio.StreamWriter.writelines).

For example:

1 ...
2 # write lines of byte data
3 writer.writelines(byte_lines)

https://superfastpython.com/python-asyncio/ 151/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Neither method for writing data blocks or suspends the calling coroutine.

After writing byte data it is a good idea to drain the socket via the drain() method
(https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamWriter.drain).

“ Wait until it is appropriate to resume writing to the stream.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
STREAM.HTML)

This is a coroutine and will suspend the caller until the bytes have been transmitted
and the socket is ready.

For example:

1 ...
2 # write byte data
3 writer.write(byte_data)
4 # wait for data to be transmitted
5 await writer.drain()

How to Read Data with the StreamReader

We can read data from the socket using an asyncio.StreamReader
(https://docs.python.org/3/library/asyncio-stream.html#streamreader).

“ Represents a reader object that provides APIs to read data from the IO
stream.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

Data is read in byte format, therefore strings may need to be encoded before being
used.

https://superfastpython.com/python-asyncio/ 152/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

All read methods are coroutines that must be awaited.

An arbitrary number of bytes can be read via the read() method, which will read until
the end of file (EOF).

1 ...
2 # read byte data
3 byte_data = await reader.read()

Additionally, the number of bytes to read can be specified via the “n” argument.

“ Read up to n bytes. If n is not provided, or set to -1, read until EOF and
return all read bytes.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

This may be helpful if you know the number of bytes expected from the next
response.

For example:

1 ...
2 # read byte data
3 byte_data = await reader.read(n=100)

A single line of data can be read using the readline() method.

This will return bytes until a new line character ‘\n’ is encountered, or EOF.

https://superfastpython.com/python-asyncio/ 153/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

“ Read one line, where “line” is a sequence of bytes ending with \n. If EOF is
received and \n was not found, the method returns partially read data. If
EOF is received and the internal buffer is empty, return an empty bytes
object.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

This is helpful when reading standard protocols that operate with lines of text.

1 ...
2 # read a line data
3 byte_line = await reader.readline()

Additionally, there is a readexactly() method to read an exact number of bytes

otherwise raise an exception, and a readuntil() that will read bytes until a specified
character in byte form is read.

How to Close Connection

The socket can be closed via the asyncio.StreamWriter.

The close() method can be called which will close the socket.

“ The method closes the stream and the underlying socket.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-
STREAM.HTML)

This method does not block.

For example:

1 ...
2 # close the socket
3 writer.close()

https://superfastpython.com/python-asyncio/ 154/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Although the close() method does not block, we can wait for the socket to close
completely before continuing on.

This can be achieved via the wait_closed() method.

“ Wait until the stream is closed. Should be called after close() to wait until the
underlying connection is closed.

— ASYNCIO STREAMS (HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO-

STREAM.HTML)

This is a coroutine that can be awaited.

For example:

1 ...
2 # close the socket
3 writer.close()
4 # wait for the socket to close
5 await writer.wait_closed()

We can check if the socket has been closed or is in the process of being closed via the
is_closing() method.

For example:

1 ...
2 # check if the socket is closed or closing
3 if writer.is_closing():
4 # ...

Now that we know how to use asyncio streams, let’s look at a worked example.

Example of Checking Website Status

We can query the HTTP status of websites using asyncio by opening a stream and
writing and reading HTTP requests and responses.

https://superfastpython.com/python-asyncio/ 155/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

We can then use asyncio to query the status of many websites concurrently, and even
report the results dynamically.

Let’s get started.

How to Check HTTP Status with Asyncio

The asyncio module provides support for opening socket connections and reading
and writing data via streams.

We can use this capability to check the status of web pages.

This involves perhaps four steps, they are:

1. Open a connection
2. Write a request
3. Read a response
4. Close the connection

Let’s take a closer look at each part in turn.

Open HTTP Connection

A connection can be opened in asyncio using the asyncio.open_connection()
function (https://docs.python.org/3/library/asyncio-
stream.html#asyncio.open_connection).

Among many arguments, the function takes the string hostname and integer port
number

This is a coroutine that must be awaited and returns a StreamReader and a

StreamWriter for reading and writing with the socket.

This can be used to open an HTTP connection on port 80.

https://superfastpython.com/python-asyncio/ 156/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # open a socket connection
3 reader, writer = await asyncio.open_connection('www.google.com', 80)

We can also open an SSL connection using the ssl=True argument. This can be used
to open an HTTPS connection on port 443.

For example:

1 ...
2 # open a socket connection
3 reader, writer = await asyncio.open_connection('www.google.com', 443)

Write HTTP Request

Once open, we can write a query to the StreamWriter to make an HTTP request.

For example, an HTTP version 1.1 request

(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) is in plain text. We can
request the file path ‘/’, which may look as follows:

1 GET / HTTP/1.1
2 Host: www.google.com

Importantly, there must be a carriage return and a line feed (\r\n) at the end of each
line, and an empty line at the end.

As Python strings this may look as follows:

1 'GET / HTTP/1.1\r\n'
2 'Host: www.google.com\r\n'
3 '\r\n'

You can learn more about HTTP v1.1 request messages here:

HTTP/1.1 request messages

(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#HTTP/1.1_request_m
essages)

https://superfastpython.com/python-asyncio/ 157/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This string must be encoded as bytes before being written to the StreamWriter
(https://docs.python.org/3/library/asyncio-stream.html#streamwriter).

This can be achieved using the encode() method

(https://docs.python.org/3/library/stdtypes.html#str.encode) on the string itself.

The default ‘utf-8‘ encoding may be sufficient.

For example:

1 ...
2 # encode string as bytes
3 byte_data = string.encode()

You can see a listing of encodings here:

Python Standard Encodings

(https://docs.python.org/3/library/codecs.html#standard-encodings)

The bytes can then be written to the socket via the StreamWriter via the write()
method (https://docs.python.org/3/library/asyncio-
stream.html#asyncio.StreamWriter.write).

For example:

1 ...
2 # write query to socket
3 writer.write(byte_data)

After writing the request, it is a good idea to wait for the byte data to be sent and for
the socket to be ready.

This can be achieved by the drain() method.

This is a coroutine that must be awaited.

For example:

https://superfastpython.com/python-asyncio/ 158/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # wait for the socket to be ready.
3 await writer.drain()

Read HTTP Response

Once the HTTP request has been made, we can read the response.

This can be achieved via the StreamReader

(https://docs.python.org/3/library/asyncio-stream.html#streamreader) for the
socket.

The response can be read using the read() method which will read a chunk of bytes,
or the readline() method which will read one line of bytes.

We might prefer the readline() method because we are using the text-based HTTP
protocol which sends HTML data one line at a time.

The readline() method is a coroutine and must be awaited.

For example:

1 ...
2 # read one line of response
3 line_bytes = await reader.readline()

HTTP 1.1 responses

(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#HTTP/1.1_response_mess
ages) are composed of two parts, a header separated by an empty line, then the body
terminating with an empty line.

The header has information about whether the request was successful and what type
of file will be sent, and the body contains the content of the file, such as an HTML
webpage.

The first line of the HTTP header contains the HTTP status for the requested page on
the server.

https://superfastpython.com/python-asyncio/ 159/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

You can learn more about HTTP v1.1 responses here:

HTTP/1.1 response messages

(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#HTTP/1.1_response_
messages)

Each line must be decoded from bytes into a string.

This can be achieved using the decode() method

(https://docs.python.org/3/library/stdtypes.html#bytes.decode) on the byte data.
Again, the default encoding is ‘utf_8‘.

For example:

1 ...
2 # decode bytes into a string
3 line_data = line_bytes.decode()

Close HTTP Connection

We can close the socket connection by closing the StreamWriter.

This can be achieved by calling the close() method

(https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamWriter.close).

For example:

1 ...
2 # close the connection
3 writer.close()

This does not block and may not close the socket immediately.

Now that we know how to make HTTP requests and read responses using asyncio,
let’s look at some worked examples of checking web page statuses.

https://superfastpython.com/python-asyncio/ 160/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Example of Checking HTTP Status Sequentially

We can develop an example to check the HTTP status for multiple websites using
asyncio.

In this example, we will first develop a coroutine that will check the status of a given
URL. We will then call this coroutine once for each of the top 10 websites
(https://en.wikipedia.org/wiki/List_of_most_visited_websites).

Firstly, we can define a coroutine that will take a URL string and return the HTTP
status.

1 # get the HTTP/S status of a webpage

2 async def get_status(url):
3 # ...

The URL must be parsed into its constituent components.

We require the hostname and file path when making the HTTP request. We also need
to know the URL scheme (HTTP or HTTPS) in order to determine whether SSL is
required nor not.

This can be achieved using the urllib.parse.urlsplit() function

(https://docs.python.org/3/library/urllib.parse.html) that takes a URL string and
returns a named tuple of all the URL elements.

1 ...
2 # split the url into components
3 url_parsed = urlsplit(url)

We can then open the HTTP connection based on the URL scheme and use the URL
hostname.

1 ...
2 # open the connection
3 if url_parsed.scheme == 'https':
4 reader, writer = await asyncio.open_connection(url_parsed.hostname, 443, ssl=True)
5 else:
6 reader, writer = await asyncio.open_connection(url_parsed.hostname, 80)

Next, we can create the HTTP GET request using the hostname and file path and write
the encoded bytes to the socket using the StreamWriter.
https://superfastpython.com/python-asyncio/ 161/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # send GET request
3 query = f'GET {url_parsed.path} HTTP/1.1\r\nHost: {url_parsed.hostname}\r\n\r\n'
4 # write query to socket
5 writer.write(query.encode())
6 # wait for the bytes to be written to the socket
7 await writer.drain()

Next, we can read the HTTP response.

We only require the first line of the response that contains the HTTP status.

1 ...
2 # read the single line response
3 response = await reader.readline()

The connection can then be closed.

1 ...
2 # close the connection
3 writer.close()

Finally, we can decode the bytes read from the server, remote trailing white space,
and return the HTTP status.

1 ...
2 # decode and strip white space
3 status = response.decode().strip()
4 # return the response
5 return status

Tying this together, the complete get_status() coroutine is listed below.

It does not have any error handling, such as the case where the host cannot be
reached or is slow to respond.

These additions would make a nice extension for the reader.

https://superfastpython.com/python-asyncio/ 162/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # get the HTTP/S status of a webpage
2 async def get_status(url):
3 # split the url into components
4 url_parsed = urlsplit(url)
5 # open the connection
6 if url_parsed.scheme == 'https':
7 reader, writer = await asyncio.open_connection(url_parsed.hostname, 443, ssl=True)
8 else:
9 reader, writer = await asyncio.open_connection(url_parsed.hostname, 80)
10 # send GET request
11 query = f'GET {url_parsed.path} HTTP/1.1\r\nHost: {url_parsed.hostname}\r\n\r\n'
12 # write query to socket
13 writer.write(query.encode())
14 # wait for the bytes to be written to the socket
15 await writer.drain()
16 # read the single line response
17 response = await reader.readline()
18 # close the connection
19 writer.close()
20 # decode and strip white space
21 status = response.decode().strip()
22 # return the response
23 return status

Next, we can call the get_status() coroutine for multiple web pages or websites we
want to check.

In this case, we will define a list of the top 10 web pages in the world.

1 ...
2 # list of top 10 websites to check
3 sites = ['https://www.google.com/',
4 'https://www.youtube.com/',
5 'https://www.facebook.com/',
6 'https://twitter.com/',
7 'https://www.instagram.com/',
8 'https://www.baidu.com/',
9 'https://www.wikipedia.org/',
10 'https://yandex.ru/',
11 'https://yahoo.com/',
12 'https://www.whatsapp.com/'
13 ]

We can then query each, in turn, using our get_status() coroutine.

In this case, we will do so sequentially in a loop, and report the status of each in turn.

1 ...
2 # check the status of all websites
3 for url in sites:
4 # get the status for the url
5 status = await get_status(url)
6 # report the url and its status
7 print(f'{url:30}:\t{status}')

We can do better than sequential when using asyncio, but this provides a good
starting point that we can improve upon later.

https://superfastpython.com/python-asyncio/ 163/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Tying this together, the main() coroutine queries the status of the top 10 websites.

1 # main coroutine
2 async def main():
3 # list of top 10 websites to check
4 sites = ['https://www.google.com/',
5 'https://www.youtube.com/',
6 'https://www.facebook.com/',
7 'https://twitter.com/',
8 'https://www.instagram.com/',
9 'https://www.baidu.com/',
10 'https://www.wikipedia.org/',
11 'https://yandex.ru/',
12 'https://yahoo.com/',
13 'https://www.whatsapp.com/'
14 ]
15 # check the status of all websites
16 for url in sites:
17 # get the status for the url
18 status = await get_status(url)
19 # report the url and its status
20 print(f'{url:30}:\t{status}')

Finally, we can create the main() coroutine and use it as the entry point to the
asyncio program.

1 ...
2 # run the asyncio program
3 asyncio.run(main())

Tying this together, the complete example is listed below.

https://superfastpython.com/python-asyncio/ 164/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # check the status of many webpages
3 import asyncio
4 from urllib.parse import urlsplit
5
6 # get the HTTP/S status of a webpage
7 async def get_status(url):
8 # split the url into components
9 url_parsed = urlsplit(url)
10 # open the connection
11 if url_parsed.scheme == 'https':
12 reader, writer = await asyncio.open_connection(url_parsed.hostname, 443, ssl=True)
13 else:
14 reader, writer = await asyncio.open_connection(url_parsed.hostname, 80)
15 # send GET request
16 query = f'GET {url_parsed.path} HTTP/1.1\r\nHost: {url_parsed.hostname}\r\n\r\n'
17 # write query to socket
18 writer.write(query.encode())
19 # wait for the bytes to be written to the socket
20 await writer.drain()
21 # read the single line response
22 response = await reader.readline()
23 # close the connection
24 writer.close()
25 # decode and strip white space
26 status = response.decode().strip()
27 # return the response
28 return status
29
30 # main coroutine
31 async def main():
32 # list of top 10 websites to check
33 sites = ['https://www.google.com/',
34 'https://www.youtube.com/',
35 'https://www.facebook.com/',
36 'https://twitter.com/',
37 'https://www.instagram.com/',
38 'https://www.baidu.com/',
39 'https://www.wikipedia.org/',
40 'https://yandex.ru/',
41 'https://yahoo.com/',
42 'https://www.whatsapp.com/'
43 ]
44 # check the status of all websites
45 for url in sites:
46 # get the status for the url
47 status = await get_status(url)
48 # report the url and its status
49 print(f'{url:30}:\t{status}')
50
51 # run the asyncio program
52 asyncio.run(main())

Running the example first creates the main() coroutine and uses it as the entry point
into the program.

The main() coroutine runs, defining a list of the top 10 websites.

The list of websites is then traversed sequentially. The main() coroutine suspends
and calls the get_status() coroutine to query the status of one website.

https://superfastpython.com/python-asyncio/ 165/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The get_status() coroutine runs, parses the URL, and opens a connection. It
constructs an HTTP GET query and writes it to the host. A response is read, decoded,
and returned.

The main() coroutine resumes and reports the HTTP status of the URL.

This is repeated for each URL in the list.

The program takes about 5.6 seconds to complete, or about half a second per URL on
average.

This highlights how we can use asyncio to query the HTTP status of webpages.

Nevertheless, it does not take full advantage of the asyncio to execute tasks
concurrently.

1 https://www.google.com/ : HTTP/1.1 200 OK

2 https://www.youtube.com/ : HTTP/1.1 200 OK
3 https://www.facebook.com/ : HTTP/1.1 302 Found
4 https://twitter.com/ : HTTP/1.1 200 OK
5 https://www.instagram.com/ : HTTP/1.1 200 OK
6 https://www.baidu.com/ : HTTP/1.1 200 OK
7 https://www.wikipedia.org/ : HTTP/1.1 200 OK
8 https://yandex.ru/ : HTTP/1.1 302 Moved temporarily
9 https://yahoo.com/ : HTTP/1.1 301 Moved Permanently
10 https://www.whatsapp.com/ : HTTP/1.1 302 Found

Next, let’s look at how we might update the example to execute the coroutines
concurrently.

Example of Checking Website Status Concurrently

A benefit of asyncio is that we can execute many coroutines concurrently.

We can query the status of websites concurrently in asyncio using the

asyncio.gather() function (https://docs.python.org/3/library/asyncio-
task.html#asyncio.gather).

https://superfastpython.com/python-asyncio/ 166/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This function takes one or more coroutines, suspends executing the provided
coroutines, and returns the results from each as an iterable. We can then traverse the
list of URLs and iterable of return values from the coroutines and report results.

This may be a simpler approach than the above.

First, we can create a list of coroutines.

1 ...
2 # create all coroutine requests
3 coros = [get_status(url) for url in sites]

Next, we can execute the coroutines and get the iterable of results using
asyncio.gather().

Note that we cannot provide the list of coroutines directly, but instead must unpack
the list into separate expressions that are provided as positional arguments to the
function.

1 ...
2 # execute all coroutines and wait
3 results = await asyncio.gather(*coros)

This will execute all of the coroutines concurrently and retrieve their results.

We can then traverse the list of URLs and returned status and report each in turn.

1 ...
2 # process all results
3 for url, status in zip(sites, results):
4 # report status
5 print(f'{url:30}:\t{status}')

Tying this together, the complete example is listed below.

https://superfastpython.com/python-asyncio/ 167/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # check the status of many webpages
3 import asyncio
4 from urllib.parse import urlsplit
5
6 # get the HTTP/S status of a webpage
7 async def get_status(url):
8 # split the url into components
9 url_parsed = urlsplit(url)
10 # open the connection
11 if url_parsed.scheme == 'https':
12 reader, writer = await asyncio.open_connection(url_parsed.hostname, 443, ssl=True)
13 else:
14 reader, writer = await asyncio.open_connection(url_parsed.hostname, 80)
15 # send GET request
16 query = f'GET {url_parsed.path} HTTP/1.1\r\nHost: {url_parsed.hostname}\r\n\r\n'
17 # write query to socket
18 writer.write(query.encode())
19 # wait for the bytes to be written to the socket
20 await writer.drain()
21 # read the single line response
22 response = await reader.readline()
23 # close the connection
24 writer.close()
25 # decode and strip white space
26 status = response.decode().strip()
27 # return the response
28 return status
29
30 # main coroutine
31 async def main():
32 # list of top 10 websites to check
33 sites = ['https://www.google.com/',
34 'https://www.youtube.com/',
35 'https://www.facebook.com/',
36 'https://twitter.com/',
37 'https://www.instagram.com/',
38 'https://www.baidu.com/',
39 'https://www.wikipedia.org/',
40 'https://yandex.ru/',
41 'https://yahoo.com/',
42 'https://www.whatsapp.com/'
43 ]
44 # create all coroutine requests
45 coros = [get_status(url) for url in sites]
46 # execute all coroutines and wait
47 results = await asyncio.gather(*coros)
48 # process all results
49 for url, status in zip(sites, results):
50 # report status
51 print(f'{url:30}:\t{status}')
52
53 # run the asyncio program
54 asyncio.run(main())

Running the example executes the main() coroutine as before.

In this case, a list of coroutines is created in a list comprehension.

The asyncio.gather() function is then called, passing the coroutines and suspending
the main() coroutine until they are all complete.

https://superfastpython.com/python-asyncio/ 168/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The coroutines execute, querying each website concurrently and returning their
status.

The main() coroutine resumes and receives an iterable of status values. This iterable
along with the list of URLs is then traversed using the zip() built-in function and the
statuses are reported.

This highlights a simpler approach to executing the coroutines concurrently and

reporting the results after all tasks are completed.

It is also faster than the sequential version above, completing in about 1.4 seconds on
my system.

1 https://www.google.com/ : HTTP/1.1 200 OK

2 https://www.youtube.com/ : HTTP/1.1 200 OK
3 https://www.facebook.com/ : HTTP/1.1 302 Found
4 https://twitter.com/ : HTTP/1.1 200 OK
5 https://www.instagram.com/ : HTTP/1.1 200 OK
6 https://www.baidu.com/ : HTTP/1.1 200 OK
7 https://www.wikipedia.org/ : HTTP/1.1 200 OK
8 https://yandex.ru/ : HTTP/1.1 302 Moved temporarily
9 https://yahoo.com/ : HTTP/1.1 301 Moved Permanently
10 https://www.whatsapp.com/ : HTTP/1.1 302 Found

Next, let’s explore common errors when getting started with asyncio.

Python Asyncio Common Errors

This section gives examples of general errors encountered by developers when using
asyncio in Python.

The 5 most common asyncio errors are:

1. Trying to run coroutines by calling them.

2. Not letting coroutines run in the event loop.
3. Using the asyncio low-level API.
4. Exiting the main coroutine too early.
5. Assuming race conditions and deadlocks are not possible.

https://superfastpython.com/python-asyncio/ 169/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Let’s take a closer look at each in turn.

Error 1: Trying to Run Coroutines by Calling Them

The most common error encountered by beginners to asyncio is calling a coroutine
like a function.

For example, we can define a coroutine using the “async def” expression:

1 # custom coroutine
2 async def custom_coro():
3 print('hi there')

The beginner will then attempt to call this coroutine like a function and expect the
print message to be reported.

For example:

1 ...
2 # error attempt at calling a coroutine like a function
3 custom_coro()

Calling a coroutine like a function will not execute the body of the coroutine.

Instead, it will create a coroutine object.

This object can then be awaited within the asyncio runtime, e.g. the event loop.

We can start the event loop to run the coroutine using the asyncio.run() function.

For example:

1 ...
2 # run a coroutine
3 asyncio.run(custom_coro())

Alternatively, we can suspend the current coroutine and schedule the other coroutine
using the “await” expression.

For example:

https://superfastpython.com/python-asyncio/ 170/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # schedule a coroutine
3 await custom_coro()

You can learn more about running coroutines in the tutorial:

How to Run an Asyncio Coroutine in Python (/asyncio-run-coroutine)

Error 2: Not Letting Coroutines Run in the Event Loop

If a coroutine is not run, you will get a runtime warning as follows:

1 sys:1: RuntimeWarning: coroutine 'custom_coro' was never awaited

This will happen if you create a coroutine object but do not schedule it for execution
within the asyncio event loop.

For example, you may attempt to call a coroutine from a regular Python program:

1 ...
2 # attempt to call the coroutine
3 custom_coro()

This will not call the coroutine.

Instead, it will create a coroutine object.

For example:

1 ...
2 # create a coroutine object
3 coro = custom_coro()

If you do not allow this coroutine to run, you will get a runtime error.

You can let the coroutine run, as we saw in the previous section, by starting the
asyncio event loop and passing it the coroutine object.

For example:

https://superfastpython.com/python-asyncio/ 171/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # create a coroutine object
3 coro = custom_coro()
4 # run a coroutine
5 asyncio.run(coro)

Or, on one line in a compound statement:

1 ...
2 # run a coroutine
3 asyncio.run(custom_coro())

You can learn more about running coroutines in the tutorial:

How to Run an Asyncio Coroutine in Python (/asyncio-run-coroutine)

If you get this error within an asyncio program, it is because you have created a
coroutine and have not scheduled it for execution.

This can be achieved using the await expression.

For example:

1 ...
2 # create a coroutine object
3 coro = custom_coro()
4 # suspend and allow the other coroutine to run
5 await coro

Or, you can schedule it to run independently as a task.

For example:

1 ...
2 # create a coroutine object
3 coro = custom_coro()
4 # schedule the coro to run as a task interdependently
5 task = asyncio.create_task(coro)

You can learn more about creating tasks in the tutorial:

How to Create an Asyncio Task in Python (/asyncio-create-task)

Error 3: Using the Low-Level Asyncio API

A big problem with beginners is that they use the wrong asyncio API.
https://superfastpython.com/python-asyncio/ 172/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This is common for a number of reasons.

The API has changed a lot with recent versions of Python.

The API docs page makes things confusing, showing both APIs.
Examples elsewhere on the web mix up using the different APIs.

Using the wrong API makes things more verbose (e.g. more code), more difficult, and
way less understandable.

Asyncio offers two APIs (https://docs.python.org/3/library/asyncio.html).

1. High-level API for application developers (us)

2. Low-level API for framework and library developers (not us)

The lower-level API provides the foundation for the high-level API and includes the
internals of the event loop, transport protocols, policies, and more.

“ … there are low-level APIs for library and framework developers

— ASYNCIO — ASYNCHRONOUS I/O
(HTTPS://DOCS.PYTHON.ORG/3/LIBRARY/ASYNCIO.HTML)

We should almost always stick to the high-level API.

We absolutely must stick to the high-level API when getting started.

We may dip into the low-level API to achieve specific outcomes on occasion.

If you start getting a handle on the event loop or use a “loop” variable to do things,
you are doing it wrong.

I am not saying don’t learn the low-level API.

https://superfastpython.com/python-asyncio/ 173/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Go for it. It’s great.

Just don’t start there.

Drive asyncio via the high-level API for a while. Develop some programs. Get
comfortable with asynchronous programming and running coroutines at will.

Then later, dip in and have a look around.

Error 4: Exiting the Main Coroutine Too Early

A major point of confusion in asyncio programs is not giving tasks enough time to
complete.

We can schedule many coroutines to run independently within an asyncio program

via the asyncio.create_task() method.

The main coroutine, the entry point for the asyncio program, can then carry on with
other activities.

If the main coroutine exits, then the asyncio program will terminate.

The program will terminate even if there are one or many coroutines running
independently as tasks.

This can catch you off guard.

You may issue many tasks and then allow the main coroutine to resume, expecting all
issued tasks to complete in their own time.

Instead, if the main coroutine has nothing else to do, it should wait on the remaining
tasks.

https://superfastpython.com/python-asyncio/ 174/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This can be achieved by first getting a set of all running tasks via the
asyncio.all_tasks() function, removing itself from this set, then waiting on the
remaining tasks via the asyncio.wait() function.

For example:

1 ...
2 # get a set of all running tasks
3 all_tasks = asyncio.all_tasks()
4 # get the current tasks
5 current_task = asyncio.current_task()
6 # remove the current task from the list of all tasks
7 all_tasks.remove(current_task)
8 # suspend until all tasks are completed
9 await asyncio.wait(all_tasks)

Error 5: Assuming Race Conditions and Deadlocks are

Impossible
Concurrent programming has the hazard of concurrency-specific failure modes.

This includes problems such as race conditions and deadlocks.

A race condition involves two or more units of concurrency executing the same
critical section at the same time and leaving a resource or data in an inconsistent or
unexpected state. This can lead to data corruption and data loss.

A deadlock is when a unit of concurrency waits for a condition that can never occur,
such as for a resource to become available.

Many Python developers believe these problems are not possible with coroutines in
asyncio.

The reason being that only one coroutine can run within the event loop at any one
time.

It is true that only one coroutine can run at a time.

https://superfastpython.com/python-asyncio/ 175/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The problem is, coroutines can suspend and resume and may do so while using a
shared resource or shared variable.

Without protecting critical sections, race conditions can occur in asyncio programs.

Without careful management of synchronization primitives, deadlocks can occur

As such, it is important that asyncio programs are created ensuring coroutine-safety,

a concept similar to thread-safety and process-safety, applied to coroutines.

Python Asyncio Common Questions

This section answers common questions asked by developers when using asyncio in
Python.

Do you have a question about asyncio?

Ask your question in the comments below and I will do my best to answer it and
perhaps add it to this list of questions.

How to Stop a Task?

We can cancel a task via the cancel() method on an asyncio.Task object
(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel).

The cancel() method returns True if the task was canceled, or False otherwise.

For example:

1 ...
2 # cancel the task
3 was_cancelled = task.cancel()

If the task is already done, it cannot be canceled and the cancel() method will return
False and the task will not have the status of canceled.

https://superfastpython.com/python-asyncio/ 176/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The next time the task is given an opportunity to run, it will raise a CancelledError
exception.

If the CancelledError exception is not handled within the wrapped coroutine, the
task will be canceled.

Otherwise, if the CancelledError exception is handled within the wrapped coroutine,

the task will not be canceled.

The cancel() method can also take a message argument which will be used in the
content of the CancelledError.

We can explore how to cancel a running task.

In this example, we define a task coroutine that reports a message and then blocks
for a moment.

We then define the main coroutine that is used as the entry point into the asyncio
program. It reports a message, creates and schedules the task, then waits a moment.

The main coroutine then resumes and cancels the task while it is running. It waits a
moment more to allow the task to respond to the request to cancel. The main
coroutine then reports whether the request to cancel the task was successful.

The task is canceled and is then done.

The main coroutine then reports whether the status of the task is canceled before
closing the program.

The complete example is listed below.

https://superfastpython.com/python-asyncio/ 177/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # SuperFastPython.com
2 # example of canceling a running task
3 import asyncio
4
5 # define a coroutine for a task
6 async def task_coroutine():
7 # report a message
8 print('executing the task')
9 # block for a moment
10 await asyncio.sleep(1)
11
12 # custom coroutine
13 async def main():
14 # report a message
15 print('main coroutine started')
16 # create and schedule the task
17 task = asyncio.create_task(task_coroutine())
18 # wait a moment
19 await asyncio.sleep(0.1)
20 # cancel the task
21 was_cancelled = task.cancel()
22 # report whether the cancel request was successful
23 print(f'was canceled: {was_cancelled}')
24 # wait a moment
25 await asyncio.sleep(0.1)
26 # check the status of the task
27 print(f'canceled: {task.cancelled()}')
28 # report a final message
29 print('main coroutine done')
30
31 # start the asyncio program
32 asyncio.run(main())

Running the example starts the asyncio event loop and executes the main()
coroutine.

The main() coroutine reports a message, then creates and schedules the task
coroutine.

It then suspends and awaits a moment to allow the task coroutine to begin running.

The task runs, reports a message and sleeps for a while.

The main() coroutine resumes and cancels the task. It reports that the request to
cancel the task was successful.

It then sleeps for a moment to allow the task to respond to the request to be
canceled.

The task_coroutine() resumes and a CancelledError exception is raised that causes

the task to fail and be done.

https://superfastpython.com/python-asyncio/ 178/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The main() coroutine resumes and reports whether the task has the status of
canceled. In this case, it does.

This example highlights the normal case of canceling a running task.

1 main coroutine started

2 executing the task
3 was canceled: True
4 canceled: True
5 main coroutine done

How to Wait for a Task To Finish?

We can wait for a task to finish by awaiting the asyncio.Task object directly.

For example:

1 ...
2 # wait for the task to finish
3 await task

We may create and wait for the task in a single line.

For example:

1 ...
2 # create and wait for the task to finish
3 await asyncio.create_task(custom_coro())

How to Get a Return Value from a Task?

We may need to return values from coroutines to the caller.

We can retrieve a return value from a coroutine by awaiting it.

It assumes that the other coroutine being awaited returns a value.

For example:

https://superfastpython.com/python-asyncio/ 179/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 # coroutine that returns a value
2 async def other_coro():
3 return 100

Awaiting the other coroutine will suspend the calling coroutine and schedule the
other coroutine for execution. Once the other coroutine has been completed, the
calling coroutine will resume. The return value will be passed from the other
coroutine to the caller.

For example:

1 ...
2 # execute coroutine and retrieve return value
3 value = await other_coro()

A coroutine can be wrapped in an asyncio.Task object.

This is helpful for independently executing the coroutine without having the current
coroutine await it.

This can be achieved using the asyncio.create_task() function

(https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task).

For example:

1 ...
2 # wrap coroutine in a task and schedule it for execution
3 task = asyncio.create_task(other_coro())

You can learn more about how to create tasks in the tutorial:

How to Create an Asyncio Task in Python (/asyncio-create-task)

There are two ways to retrieve the return value from an asyncio.Task, they are:

1. Await the task.

2. Call the result() method.

We can await the task to retrieve the return value.

https://superfastpython.com/python-asyncio/ 180/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

If the task is scheduled or running, then the caller will suspend until the task is
complete and the return value will be provided.

If the task is completed, the return value will be provided immediately.

For example:

1 ...
2 # get the return value from a task
3 value = await task

Unlike a coroutine, we can await a task more than once without raising an error.

For example:

1 ...
2 # get the return value from a task
3 value = await task
4 # get the return value from a task
5 value = await task

We can also get the return value from the task by calling the result() method
(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.result) on the
asyncio.Task object.

For example:

1 ...
2 # get the return value from a task
3 value = task.result()

This requires that the task is done. If not, an InvalidStateError exception will be
raised.

If the task was canceled a CancelledError exception will be raised.

You can learn more about getting the result from tasks in the tutorial:

How to Get Asyncio Task Results (/asyncio-task-result)

https://superfastpython.com/python-asyncio/ 181/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

How to Run a Task in the Background?

We can run a coroutine in the background by wrapping it in an asyncio.Task object.

This can be achieved by calling the asyncio.create_task() function and passing it the
coroutine.

The coroutine will be wrapped in a Task object and will be scheduled for execution.
The task object will be returned and the caller will not suspend.

For example:

1 ...
2 # schedule the task for execution
3 task = asyncio.create_task(other_coroutine())

The task will not begin executing until at least the current coroutine is suspended, for
any reason.

We can help things along by suspending for a moment to allow the task to start
running.

This can be achieved by sleeping for zero seconds.

For example:

1 ...
2 # suspend for a moment to allow the task to start running
3 await asyncio.sleep(0)

This will suspend the caller only for a brief moment and allow the ask an opportunity
to run.

This is not required as the caller may suspend at some future time or terminate as
part of normal execution.

We may also await the task directly once the caller has run out of things to do.

https://superfastpython.com/python-asyncio/ 182/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example:

1 ...
2 # wait for the task to complete
3 await task

How to Wait for All Background Tasks?

We can wait for all independent tasks in an asyncio program.

This can be achieved by first getting a set of all currently running tasks via the
asyncio.all_tasks() function.

For example:

1 ...
2 # get a set of all running tasks
3 all_tasks = asyncio.all_tasks()

This will return a set that contains one asyncio.Task object for each task that is
currently running, including the main() coroutine.

We cannot wait on this set directly, as it will block forever as it includes the task that is
the current task.

Therefore we can get the asyncio.Task object for the currently running task and
remove it from the set.

This can be achieved by first calling the asyncio.current_task() method to get the
task for the current coroutine and then remove it from the set via the remove()
method.

For example:

1 ...
2 # get the current tasks
3 current_task = asyncio.current_task()
4 # remove the current task from the list of all tasks
5 all_tasks.remove(current_task)

https://superfastpython.com/python-asyncio/ 183/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Finally, we can wait on the set of remaining tasks.

This will suspend the caller until all tasks in the set are complete.

For example:

1 ...
2 # suspend until all tasks are completed
3 await asyncio.wait(all_tasks)

Tying this together, the snippet below added to the end of the main() coroutine will
wait for all background tasks to complete.

1 ...
2 # get a set of all running tasks
3 all_tasks = asyncio.all_tasks()
4 # get the current tasks
5 current_task = asyncio.current_task()
6 # remove the current task from the list of all tasks
7 all_tasks.remove(current_task)
8 # suspend until all tasks are completed
9 await asyncio.wait(all_tasks)

Does a Running Task Stop the Event Loop from Exiting?

No.

A task that is scheduled and run independently will not stop the event loop from
exiting.

If your main coroutine has no other activities to complete and there are independent
tasks running in the background, you should retrieve the running tasks and wait on
them

The previous question/answer shows exactly how to do this.

How to Show Progress of Running Tasks?

We can show progress using a done callback function on each task.

https://superfastpython.com/python-asyncio/ 184/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

A done callback is a function that we can register on an asyncio.Task.

It is called once the task is done, either normally or if it fails.

The done callback function is a regular function, not a coroutine, and takes the
asyncio.Task that it is associated with as an argument.

We can use the same callback function for all tasks and report progress in a general
way, such as by reporting a message.

For example:

1 # callback function to show progress of tasks

2 def progress(task):
3 # report progress of the task
4 print('.', end='')

We can register a callback function on each asyncio.Task that we issue.

This can be achieved using the add_done_callback() method

(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.add_done_callback)
on each task and passing it the name of the callback function.

For example:

1 ...
2 # add a done callback to a task
3 task.add_done_callback(progress)

How to Run a Task After a Delay?

We can develop a custom wrapper coroutine to execute a target coroutine after a
delay.

The wrapper coroutine may take two arguments, a coroutine and a time in seconds.

It will sleep for the given delay interval in seconds, then await the provided coroutine.

https://superfastpython.com/python-asyncio/ 185/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The delay() coroutine below implements this.

1 # coroutine that will start another coroutine after a delay in seconds

2 async def delay(coro, seconds):
3 # suspend for a time limit in seconds
4 await asyncio.sleep(seconds)
5 # execute the other coroutine
6 await coro

To use the wrapper coroutine, a coroutine object can be created and either awaited
directly or executed independently as a task.

For example, the caller may suspend and schedule the delayed coroutine and wait for
it to be done:

1 ...
2 # execute a coroutine after a delay
3 await delay(coro, 10)

Alternatively, the caller may schedule the delayed coroutine to run independently:

1 ...
2 # execute a coroutine after a delay independently
3 _ = asyncio.create_task(delay(coro, 10))

How to Run a Follow-Up Task?

There are three main ways to issue follow-up tasks in asyncio.

They are:

1. Schedule the follow-up task from the completed task itself.

2. Schedule the follow-up task from the caller.
3. Schedule the follow-up task automatically using a done callback.

Let’s take a closer look at each approach.

The task that is completed can issue its own follow-up task.

This may require checking some state in order to determine whether the follow-up
task should be issued or not.
https://superfastpython.com/python-asyncio/ 186/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The task can then be scheduled via a call to asyncio.create_task().

For example:

1 ...
2 # schedule a follow-up task
3 task = asyncio.create_task(followup_task())

The task itself may choose to await the follow-up task or let it complete in the
background independently.

For example:

1 ...
2 # wait for the follow-up task to complete
3 await task

The caller that issued the task can choose to issue a follow-up task.

For example, when the caller issues the first task, it may keep the asyncio.Task
object.

It can then check the result of the task or whether the task was completed
successfully or not.

The caller can then decide to issue a follow-up task.

It may or may not await the follow-up task directly.

For example:

1 ...
2 # issue and await the first task
3 task = await asyncio.create_task(task())
4 # check the result of the task
5 if task.result():
6 # issue the follow-up task
7 followup = await asyncio.create_task(followup_task())

We can execute a follow-up task automatically using a done callback function.

https://superfastpython.com/python-asyncio/ 187/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

For example, the caller that issues the task can register a done callback function on
the task itself.

The done callback function must take the asyncio.Task object as an argument and
will be called only after the task is done. It can then choose to issue a follow-up task.

The done callback function is a regular Python function, not a coroutine, so it cannot
await the follow-up task

For example, the callback function may look as follows:

1 # callback function
2 def callback(task):
3 # schedule and await the follow-up task
4 _ = asyncio.create_task(followup())

The caller can issue the first task and register the done callback function.

For example:

1 ...
2 # schedule and the task
3 task = asyncio.create_task(work())
4 # add the done callback function
5 task.add_done_callback(callback)

How to Execute a Blocking I/O or CPU-bound Function in

Asyncio?
The asyncio module provides two approaches for executing blocking calls in asyncio
programs.

The first is to use the asyncio.to_thread() function

(https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread).

This is in the high-level API and is intended for application developers.

The asyncio.to_thread() function takes a function name to execute and any

arguments.
https://superfastpython.com/python-asyncio/ 188/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The function is executed in a separate thread. It returns a coroutine that can be

awaited or scheduled as an independent task.

For example:

1 ...
2 # execute a function in a separate thread
3 await asyncio.to_thread(task)

The task will not begin executing until the returned coroutine is given an opportunity
to run in the event loop.

The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes

to execute blocking calls.

As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks.

An alternative approach is to use the loop.run_in_executor() function

(https://docs.python.org/3/library/asyncio-
eventloop.html#asyncio.loop.run_in_executor).

This is in the low-level asyncio API and first requires access to the event loop, such as
via the asyncio.get_running_loop() function
(https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_running_loop).

The loop.run_in_executor() function takes an executor and a function to execute.

If None is provided for the executor, then the default executor is used, which is a
ThreadPoolExecutor.

The loop.run_in_executor() function returns an awaitable that can be awaited if

needed. The task will begin executing immediately, so the returned awaitable does
not need to be awaited or scheduled for the blocking call to start executing.

For example:

https://superfastpython.com/python-asyncio/ 189/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
1 ...
2 # get the event loop
3 loop = asyncio.get_running_loop()
4 # execute a function in a separate thread
5 await loop.run_in_executor(None, task)

Alternatively, an executor can be created and passed to the loop.run_in_executor()

function, which will execute the asynchronous call in the executor.

The caller must manage the executor in this case, shutting it down once the caller is
finished with it.

For example:

1 ...
2 # create a process pool
3 with ProcessPoolExecutor as exe:
4 # get the event loop
5 loop = asyncio.get_running_loop()
6 # execute a function in a separate thread
7 await loop.run_in_executor(exe, task)
8 # process pool is shutdown automatically...

These two approaches allow a blocking call to be executed as an asynchronous task

in an asyncio program.

Common Objections to Using Asyncio

Asyncio and coroutines may not be the best solution for all concurrency problems in
your program.

That being said, there may also be some misunderstandings that are preventing you
from making full and best use of the capabilities of the asyncio in Python.

In this section, we review some of the common objections seen by developers when
considering using the asyncio.

What About the Global Interpreter Lock (GIL)?

The GIL protects the internals of the Python interpreter from concurrent access and
modification from multiple threads.

https://superfastpython.com/python-asyncio/ 190/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

The asyncio event loop runs in one thread.

This means that all coroutines run in a single thread.

As such the GIL is not an issue when using asyncio and coroutine.

Are Python Coroutines “Real“?

Coroutines are managed in software.

Coroutines run and are managed (switched) within the asyncio event loop in the
Python runtime.

They are not a software representation of a capability provided by the underlying

operating system, like threads and processes.

In this sense, Python does not have support for “native coroutines”, but I’m not sure
such things exist in modern operating systems.

Isn’t Python Concurrency Buggy?

No.

Python provides first-class concurrency with coroutines, threads, and processes.

It has for a long time now and it is widely used in open source and commercial
projects.

Isn’t Python a Bad Choice for Concurrency?

Developers love python for many reasons, most commonly because it is easy to use
and fast for development.

https://superfastpython.com/python-asyncio/ 191/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Python is commonly used for glue code, one-off scripts, but more and more for large-
scale software systems.

If you are using Python and then you need concurrency, then you work with what you
have. The question is moot.

If you need concurrency and you have not chosen a language, perhaps another
language would be more appropriate, or perhaps not. Consider the full scope of
functional and non-functional requirements (or user needs, wants, and desires) for
your project and the capabilities of different development platforms.

Why Not Use Threads Instead?

You can use threads instead of asyncio.

Any program developed using threads can be rewritten to use asyncio and
coroutines.

Any program developed using coroutines and asyncio can be rewritten to use
threads.

Adopting asyncio in a project is a choice, the rationale is yours.

For the most part, they are functionally equivalent.

Many use cases will execute faster using threads and may be more familiar to a wider
array of Python developers.

Some use cases in the areas of network programming and executing system
commands may be simpler (less code) when using asyncio, and significantly more
scalable than using threads.

https://superfastpython.com/python-asyncio/ 192/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Further Reading
This section provides additional resources that you may find helpful.

Python Asyncio Books

Python Asyncio Mastery (https://superfastpython.com/pam-further-reading),

Jason Brownlee (my book!)
Python Asyncio Jump-Start (https://superfastpython.com/paj-further-reading),
Jason Brownlee.
Python Asyncio Interview Questions (https://superfastpython.com/python-
asyncio-interview-questions/), Jason Brownlee.
Asyncio Module API Cheat Sheet (https://marvelous-writer-
6152.ck.page/d29b7d8dfb)

I also recommend the following books:

Python Concurrency with asyncio (https://amzn.to/3LZvxNn), Matthew Fowler,

2022.
Using Asyncio in Python (https://amzn.to/3lNp2ml), Caleb Hattingh, 2020.
asyncio Recipes (https://amzn.to/47oN8dk), Mohamed Mustapha Tahrioui, 2019.

Guides

Python Asyncio: The Complete Guide (https://superfastpython.com/python-

asyncio/)
Python Asynchronous Programming (https://superfastpython.com/python-
asynchronous-programming/)

APIs

asyncio — Asynchronous I/O (https://docs.python.org/3/library/asyncio.html)

https://superfastpython.com/python-asyncio/ 193/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Asyncio Coroutines and Tasks (https://docs.python.org/3/library/asyncio-

task.html)
Asyncio Streams (https://docs.python.org/3/library/asyncio-stream.html)
Asyncio Subprocesses (https://docs.python.org/3/library/asyncio-
subprocess.html)
Asyncio Queues (https://docs.python.org/3/library/asyncio-queue.html)
Asyncio Synchronization Primitives (https://docs.python.org/3/library/asyncio-
sync.html)

References

Asynchronous I/O, Wikipedia (https://en.wikipedia.org/wiki/Asynchronous_I/O).

Coroutine, Wikipedia (https://en.wikipedia.org/wiki/Coroutine).

Conclusions
This is a large guide, and you have discovered in great detail how asyncio and
coroutines work in Python and how to best use them in your project.

Did you find this guide useful?

I’d love to know, please share a kind word in the comments below.

Have you used asyncio on a project?

I’d love to hear about it, please let me know in the comments.

Do you have any questions?

Leave your question in a comment below and I will reply fast with my best advice.

Join the discussion on reddit

(https://www.reddit.com/r/Python/comments/yqrr94/python_asyncio_the_complete_g
uide/) and hackernews (https://news.ycombinator.com/item?id=33547323).

https://superfastpython.com/python-asyncio/ 194/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

About Jason Brownlee

Hi, my name is Jason Brownlee, Ph.D. and I’m the guy behind this website. I
am obsessed with Python Concurrency.

I help python developers learn concurrency, super fast.

Learn more (/about).

Parallel Loops in Python

Discover how to run your loops in parallel, download your free book
(https://marvelous-writer-6152.ck.page/99ee689b9b) now:

(https://marvelous-writer-
6152.ck.page/99ee689b9b)

Your free book "Parallel Loops in Python" includes complete and working code
templates that you can modify and use right now in your own projects.

Download Your FREE Book (https://marvelous-writer-

6152.ck.page/99ee689b9b)

https://superfastpython.com/python-asyncio/ 195/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Comments
Alexey says
DECEMBER 4, 2022 AT 10:41 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-364)

Thank you! It was wonderful. Now I would like to see a complete guide about low-level asyncio api. It would be
great, thank you))

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

DECEMBER 5, 2022 AT 7:27 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-373)

Thank you, I’m happy it helped.

Great idea!

REPLY

Vahidin Bajrić says

DECEMBER 11, 2022 AT 1:47 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-488)

One who masters Asyncio has mastered the Python.

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

DECEMBER 11, 2022 AT 5:34 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-493)

You might be right. It’s challenging for sure.

REPLY

Mark Tompkins says

JANUARY 31, 2023 AT 11:23 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1202)

Great read! Thank you! You did an excellent job!

I want to develop a scheduler, that emulates a commercial timer, such as an omron h5s digital time switch. I
want to create two versions, python, and micropython.

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

FEBRUARY 1, 2023 AT 5:21 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1208)

https://superfastpython.com/python-asyncio/ 196/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Thank you!

That sounds like a great project!

REPLY

Paul Rautenbach says

FEBRUARY 5, 2023 AT 9:43 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1255)

Thank you for this guide. I learnt a lot. In particular, I learned the reason why my MicroPython program fails to
work.
It tries to use uasyncio (a cut down version of asyncio for MicroPython run on microprocesors). My program
also uses lv_micropython which is the LVGL graphical user interface framework running within MicroPython.
My program implements a voice reminder application in MicroPython on an M5Stack Core2.
I’ve succeeded in using uasyncio Stream (StreamReader wrapping the microphone device) to capture audio
from the microphone into a buffer and then write it to a file. At the moment this only works as a stand alone
program. Also, as another stand alone program, I can play the recorded audio file through the speaker using
uasyncio.StreamWriter to wrap the speaker device.
My problem comes when trying to drive this code from the LVGL user interface. Specifically, I need to record
audio as long as a record button remains pressed and listen to the recorded audio as long as a listen button
remains pressed (or the whole audio file if the listen button is pressed for as long as the audio reminder lasts).
To implement the listen button I tried to call uasyncio.create_task(speaker.play_wav(“/recorded.wav”)) when
the listen button is pressed and stop the playing when the listen button is released. The stopping is done by
setting a stopped flag tested by a ‘while not stopped’ loop that plays successive chunks of the audio file to the
speaker.
The problem seems to be that the button event callback function, registered with the listen button and called
by LVGL in response to the button being pressed or released, is not a coroutine. It therefore can’t successfully
call uasyncio.create_task(). The same problem will arise for recording while a record button is pressed, when I
come to implement that.
I would appreciate any help and advice you can give on how I might work around this problem or approach
things in a different way that avoids creating the playing (or recording) task from the non-coroutine callback.
I realise this is a long comment but I tried to keep it succinct.
Thanks in advance for your help.
Paul

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

FEBRUARY 6, 2023 AT 5:27 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1262)

I’m happy that the guide is useful Paul!

That sounds like a great app.

https://superfastpython.com/python-asyncio/ 197/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Perhaps you can check of the audio API you are using supports asyncio or not. If not, you will need to
issue the blocking calls to a thread pool via asyncio.run_in_executor() or asyncio.to_thread().

This tutorial will help you get started:

https://superfastpython.com/asyncio-blocking-tasks/ (https://superfastpython.com/asyncio-blocking-
tasks/)
REPLY

Tomas says
FEBRUARY 11, 2023 AT 1:10 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1345)

Thanks you for guide. He allowed to streamline knowledge about asyncio. Is it possible to control the number
of coroutines running at the same time? For example, the simultaneous work of only 10 out of 1000
coroutines and the addition of new tasks to replace the completed ones. I was able to implement this using
the threading module, but the possibility of implementing such logic using asyncio is interesting.

REPLY

Tomas says
FEBRUARY 11, 2023 AT 2:07 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1346)

Found the answer to my question here: https://superfastpython.com/asyncio-semaphore/

(https://superfastpython.com/asyncio-semaphore/) ))) It’s worth adding a reference to
asyncio.Semaphore() to this guide.

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

FEBRUARY 11, 2023 AT 5:30 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-
1351)

Great suggestion, thanks Tomas.

REPLY

Paul Harley says

MARCH 4, 2023 AT 11:06 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1702)

Thank you for writing such a clear guide. I will absolutely be book-marking this and coming back to it regularly.
There is a clarification that I would like to ask you for. We see it is possible to execute a coroutine (or rather,
schedule it for execution) using await but it is also possible to execute a coroutine using event loop methods
such as run_until_complete()
What is the difference between them? Is one preferable over the other? My guess is that await is more friendly

https://superfastpython.com/python-asyncio/ 198/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

since it could allow other tasks to run as well as the desired one, whereas run_until_complete() is immediately
blocking i.e all other running tasks are suspended until the desired one completes. Anyway that’s my guess
but I’d really love to get your insight about this. Thanks again.
REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

MARCH 5, 2023 AT 5:42 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-1723)

Thank you Paul.

Yes, generally we would schedule coroutines using await and not interact with the event loop directly.
The API on the event loop is intended for library developers, not application developers. You can learn
more about this here:
https://superfastpython.com/asyncio-apis/ (https://superfastpython.com/asyncio-apis/)

REPLY

Chris says
MAY 5, 2023 AT 12:41 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-3688)

Hi Jason,
This is the best guide for asyncio I have read. There are many conceptual foundations you have laid for the
reader to really understand the big picture. Without those, all the functions are really just a mystery, and
prone to be misused.
If possible, it would be great if you could point us to some references that explain more about how the event
loop interacts with different types of tasks, some of which are IO tasks, and some are Python coroutines.

Is it the case that the async IO tasks run in a separate process, and the operating system will inform the event
loop of its completion through some kind of callback?
What about asyncio.sleep()? It seems it does not actually take up CPU time: if t1 involves asyncio.sleep(1) and
t2 involves actual work that takes 1 sec, and the caller coroutine uses await asyncio.gather(t1, t2), both t1 and
t2 will finish in 1 sec.
How is event loop implemented? If there is only a single task asyncio.sleep(3) on the loop, does it periodically
check whether the task finishes using poll, or it waits for an interrupt?

Thank you so much!Chris

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

MAY 5, 2023 AT 6:36 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-3701)

Thank you for your kind words!

https://superfastpython.com/python-asyncio/ 199/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

I don’t have a ton on the event loop, this might be a good start:
https://superfastpython.com/asyncio-event-loop/ (https://superfastpython.com/asyncio-event-loop/)

Implementation details differ, e.g. there are different types of event loops.

After that, you might want to dig into the docs here:
https://docs.python.org/3/library/asyncio-eventloop.html (https://docs.python.org/3/library/asyncio-
eventloop.html)
REPLY

Chris says
MAY 7, 2023 AT 6:12 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-3766)

Thanks a lot for your references Jason!

REPLY

Mathieu says
MAY 17, 2023 AT 7:26 PM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-4173)

Hello !
This website looks very interesting and well made ! But before spending hours reading and learning, I want to
know if it would help me in my situation, let me explain.
I have to develop a GUI in Python but I have to use asyncio in order to make calculations fatser and overall : I
need to make a restart button for the user if he wants to stop the process.
Would I learn how to do theses things by reading our work ?

Thank you in advance, sincerly.

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

MAY 18, 2023 AT 6:04 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-4185)

Sorry, I don’t have any tutorials on developing GUIs and adding asyncio. I may cover the topic in the
future.

REPLY

Tim says
MAY 18, 2023 AT 1:37 PM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-4190)

https://superfastpython.com/python-asyncio/ 200/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

This is the best guide for asyncio that I’ve stumbled across. There are other guides that suffice, but your
explanation of each component in asyncio is quite helpful.

You are an excellent instructor

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

MAY 18, 2023 AT 4:54 PM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-4193)

Thank you deeply Tim, your support means a lot!

REPLY

Olimjon Nematov says

JULY 19, 2023 AT 4:14 PM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-6466)

Thanks for detailed explanation of asyncio.

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

JULY 20, 2023 AT 5:11 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-6497)

You’re very welcome!

REPLY

Timothy Beamish says

SEPTEMBER 20, 2023 AT 2:35 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-9301)

On this page (https://superfastpython.com/python-asyncio/#When_to_Not_Use_Asyncio

(https://superfastpython.com/python-asyncio/#When_to_Not_Use_Asyncio)), you claim that the statement
“Asyncio is faster than threads” is false. Asyncio uses coroutines to achieve concurrency.

Yet on this page (https://superfastpython.com/coroutines-faster-threads/

(https://superfastpython.com/coroutines-faster-threads/)), you claim coroutines are faster to start than
threads.

These two concepts appear to be in conflict with each other. Can explain this further and resolve this conflict?

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

SEPTEMBER 20, 2023 AT 5:32 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-9309)

https://superfastpython.com/python-asyncio/ 201/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

Hi Timothy, great question. I can see why you say that.

There are two different issues here:

1) Speed to start.
2) Speed of program overall.

Starting a thread is slower than starting a coroutine. The rationale is that a thread is a heavier object
(e.g. native thread in the OS) whereas a coroutine is just a type of routine (e.g. a function). The
benchmark results bare this out:
https://superfastpython.com/coroutines-faster-threads/ (https://superfastpython.com/coroutines-
faster-threads/)

Now consider a program that does a ton of socket IO. It’s slow. We can make it faster by doing the IO
concurrently with threads. WIN! We can also make it faster by doing the IO concurrently with
coroutines. WIN! They are different, e.g. blocking IO vs non-blocking IO and threads vs coroutines, but
they both offer a benefit. The comment in this guide that “Asyncio is faster than threads is false” holds
and does not conflict.

Although we can make a slow sequential program faster with threads or with asyncio, the asyncio
version will not be faster than the threaded version. This has to be stated because many developers
wrongly believe that using asyncio will be faster than threads.

This is a general statement, e.g. it is expected to hold across programs, across tasks.

Nevertheless, if we focus our microbenchmark on startup time, we will see that coroutines will indeed
start faster than the threads. But if we expand our benchmark to the entire program, this benefit is
expected to be washed out by the variability of running the program while other programs are
running on the same OS/CPU, of the variability of socket IO, etc. Put another way, they will perform
about the same when the whole program is considered.

You can see examples of this when doing file IO here, the asyncio versions are about as fast as the
threaded versions (admitted the async fileio is simulated using a thread pool under the covers):
https://superfastpython.com/learning-paths/#File_IO_Concurrency_Learning_Path
(https://superfastpython.com/learning-paths/#File_IO_Concurrency_Learning_Path)

I’ll develop a socket io example to make this point clearer. Thank you kindly for the prompt and
question!

Does that help?

REPLY

Timothy Beamish says

https://superfastpython.com/python-asyncio/ 202/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python
SEPTEMBER 21, 2023 AT 1:50 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-9352)

Thanks for the prompt reply.

Consider an async python web server (eg. Uvicorn, implementing ASGI) connected to a
synchronous application (Django). In this setup, Uvicorn creates a coroutine for each request,
sends it to Django, which must create a new thread per request because it needs to flip its
context from sync to async. This thread can remain synchronous and you achieve a level of
concurrency because as the thread makes blocking I/O calls, it relinquishes it’s GIL and
another thread takes over.

An alternative setup is to convert Django to fully asynchronous and thus there would be only
one thread in the system (the Uvicorn event-loop thread). It would handle each request via a
coroutine. In Django, each request’s call flow would need to become async to avoid blocking
the event-loop when it makes an I/O call (to a database or another server).

Based on your logic, the second setup starts the coroutines faster but the first setup might
complete the full request-to-response cycle faster. This is surprising because we’ve been led to
believe that single-threaded event-loop-based web servers are generally faster than multi-
threaded servers.

What’s your take on these two scenarios and can you incorporate server load into the answer
(i.e. do threads win at higher loads or vice versa)?

REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

SEPTEMBER 21, 2023 AT 8:46 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-
ASYNCIO/#COMMENT-9363)

Great example!

Yes, they are functionally equilivient.

From a performance persective, requests can be served as fast with threads as with
corutines. No differences. Neither is faster.

There is a fixed startup time. Corutines are faster to start than threads, threads are
faster to start than processes. We overcome this fixed startup time by starting each
worker once and maintaining a pool of live request handlers (of some size…).

Technically, you’re correct. There is a fraction of a second additional overhead to start

a thread and this can be added to request serving time if one thread is started per
request, or amortized over all requests that a worker thread serves. Because it’s fixed,

https://superfastpython.com/python-asyncio/ 203/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

it’s written off and most concurrency people say that coroutines are no faster than
threads. This holds for fixed-duration and variable-duration tasks.

In fact, some say that threads are faster than coroutines:

“ Asyncio will make my code blazing fast. Unfortunately,

no. In fact, most benchmarks seem to show that
threading solutions are slightly faster than their
comparable Asyncio solutions.

— Page 6, Using Asyncio in Python, 2020.

The rationale here that threads faster than coroutines, I guess, is that there is a cost to
suspend/resume in one thread that adds up to more than context switching between
OS-level threads, perhaps. Or failing to go “async all the way down” adds up many
micro blocking calls that impact overall performance. Not seen this myself yet. But
maybe I’ve not scaled enough.

Another key aspect we have not touched on in scalability. We can run more coroutines
in a system than threads simply because we need less memory to maintain a
coroutine than an os thread. See this tutorial:

– Coroutines Use Less Memory Than Threads in Python

https://superfastpython.com/coroutines-less-memory-threads/
(https://superfastpython.com/coroutines-less-memory-threads/)

Does this help? Am I making it worse?

UPDATE: I have run some tests comparing coroutines to threads, you can see the
results here:

– Asyncio Coroutines Faster Than Threads!?

https://superfastpython.com/asyncio-coroutines-faster-than-threads/
(https://superfastpython.com/asyncio-coroutines-faster-than-threads/)
REPLY

Mudit says
NOVEMBER 24, 2023 AT 4:26 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-10061)

In the section titled “When does a task run”, it says:

https://superfastpython.com/python-asyncio/ 204/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

In fact, the task will not execute until the event loop has an opportunity to run.

This will not happen until all other coroutines are not running and it is the task’s turn to
run.

I think you meant: This will not happen until all other coroutines ARE running ?
REPLY

JASON BROWNLEE (HTTPS://SUPERFASTPYTHON.COM) says

NOVEMBER 24, 2023 AT 6:21 AM (HTTPS://SUPERFASTPYTHON.COM/PYTHON-ASYNCIO/#COMMENT-10062)

No, it is is correct.

The event loop will only execute one coroutine at a time, even though multiple coroutines are
“running”.

For a new task to execute, all other coroutines must be suspended.

Does that help?

REPLY

Learn Asyncio Fast

(without the frustration)
(https://superfastpython.com/paj-footer)
What if you could develop Python programs that
were asynchronous from the start?

The asyncio module provides easy-to-use coroutine-

based concurrency for asynchronous programming.

Introducing: "Python Asyncio Jump-Start

(https://superfastpython.com/paj-footer)".

A new book designed to teach you the asyncio module step-by-step, super
fast!

https://superfastpython.com/python-asyncio/ 205/206
2/21/24, 6:59 PM Python Asyncio: The Complete Guide - Super Fast Python

COPYRIGHT © 2024 SUPER FAST PYTHON

LINKEDIN (HTTPS://WWW.LINKEDIN.COM/COMPANY/SUPER-FAST-PYTHON/) | TWITTER
(HTTPS://TWITTER.COM/SUPERFASTPYTHON) | FACEBOOK (HTTPS://WWW.FACEBOOK.COM/SUPERFASTPYTHON) | RSS
(HTTPS://SUPERFASTPYTHON.COM/FEED/)

SUPER FAST PYTHON PTY. LTD.

PO BOX 206, VERMONT, VICTORIA, 3133, AUSTRALIA
ABN: 65 654 877 433, ACN: 654 877 433
EMAIL: [email protected]

https://superfastpython.com/python-asyncio/ 206/206