Concurrency in Python

In Python, early concurrency methods were dominated by traditional multi-processing and multi-threading, similar to Java. At the same time, there were many third-party asynchronous solutions (gevent/tornado/twisted, etc.).
In the Python 3 era, the official asyncio library and async/await syntax were introduced as Python’s official coroutine implementation, which gradually became popular.

Processes

Example of multi-processing programming:

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

The API of multiprocessing is close to threading, making it relatively easy to create multi-process programs. It is a solution recommended by Python officially to bypass the GIL restriction of multi-threading.
However, it should be noted that the parameters for creating a process need to be pickle-serializable. It is best to use process-safe data structures like Pipe and Queue (Official Programming guidelines).

Threads

Example of multi-threading code:

from threading import Thread

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Thread(target=f, args=('bob',))
    p.start()
    p.join()

# Thread pool approach
with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(pow, 323, 1235)
    print(future.result())

Drawback of CPython Threads: GIL (Global Interpreter Lock)
The GIL is a global lock used by CPython when executing Python bytecodes, preventing the interpreter from fully utilizing multi-cores for CPU-bound tasks, while IO-bound tasks release the GIL.
To bypass the GIL, one must switch to multi-processing or use C extensions.

Coroutines

asyncio example:

import asyncio
import time

async def say_after(delay, what):
    await asyncio.sleep(delay)
    print(what)

async def main():
    print(f"started at {time.strftime('%X')}")
    await asyncio.gather(say_after(1, 'hello'), say_after(2,'world'))
    print(f"finished at {time.strftime('%X')}")

asyncio.run(main())
# started at 22:32:23
# hello
# world
# finished at 22:32:25

async syntax and asyncio

Starting from version 3.4, Python’s standard library includes the asyncio module, and from 3.5 onwards, it supports async/await syntax.
The implementation of Python coroutines can be traced back to the yield keyword and the generator structure introduced in the Python 2 era:

  • Generators pause via yield and can return values.
  • The caller resumes the generator via next() or send() methods, and can send data to the generator via send().
  • The yield from syntactic sugar facilitates iterating over every value in a generator.
  • With the introduction of async/await syntax, the coroutine type was officially established.
  • The asyncio library provides an official event loop implementation and supports IO multiplexing on different operating systems (select/epoll/iocp, etc.), or can be replaced by third-party implementations (like uvloop) via configuration.
  • With the help of the concurrent.futures thread pool/process pool module, it supports multi-threading/multi-processing, but the event loop itself remains single-threaded.

Concurrency in Go

goroutine and channel example:

package main

import "fmt"

func main() {
   messages := make(chan string)
   go func() { messages <- "ping" }()
   msg := <-messages
   fmt.Println(msg)
}

goroutine and channel

Golang implements user-space coroutines called goroutines, scheduling them via the GPM model.

It also supports network IO multiplexing via netpoller.
Communication between different goroutines is achieved through channels.

CSP

CSP (Communicating Sequential Processes) is a concurrency model that interacts via message passing rather than shared variables.

Comparison

Stackful vs Stackless Coroutines

Single-threaded vs Multi-threaded