{"id":4085,"date":"2022-07-14T09:52:13","date_gmt":"2022-07-14T09:52:13","guid":{"rendered":"https:\/\/www.pythontutorial.net\/?page_id=4085"},"modified":"2023-06-04T02:29:27","modified_gmt":"2023-06-04T02:29:27","slug":"python-thread-queue","status":"publish","type":"page","link":"https:\/\/www.pythontutorial.net\/python-concurrency\/python-thread-queue\/","title":{"rendered":"Python Thread-safe Queue"},"content":{"rendered":"\n

Summary<\/strong>: in this tutorial, you’ll learn how to use a Python thread-safe queue to exchange data safely between multiple threads.<\/p>\n\n\n\n

Introduction to the Python thread-safe queue #<\/a><\/h2>\n\n\n\n

The built-in queue<\/code> module allows you to exchange data safely between multiple threads<\/a>. The Queue<\/code> class in the queue<\/code> module implements all required locking semantics.<\/p>\n\n\n\n

Creating a new queue #<\/a><\/h3>\n\n\n\n

To create a new queue, you import the Queue class from the queue module: <\/p>\n\n\n

from<\/span> queue import<\/span> Queue<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
and use the Queue<\/code> constructor as follows:<\/p>\n\n\n
queue = Queue()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
To create a queue with a size limit, you can use the maxsize<\/code> parameter. For example, the following creates a queue that can store up to 10 items:<\/p>\n\n\n
queue = Queue(maxsize=10<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Adding an item to the queue #<\/a><\/h3>\n\n\n\n
To add an item to the queue, you use the put()<\/code> method like this:<\/p>\n\n\n
queue.add(item)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Once the queue is full, you won’t be able to add an item to it. Also, the call to the put()<\/code> method will block until the queue has space available.<\/p>\n\n\n\n
If you don’t want the put()<\/code> method to block if the queue is full, you can set the block<\/code> argument to False<\/code>:<\/p>\n\n\n
queue.put(item, block=False<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
In this case, the put()<\/code> method will raise the queue.Full<\/code> exception if the queue is full:<\/p>\n\n\n
try<\/span>:\n   queue.put(item, block=False<\/span>)\nexcept<\/span> queue.Full as<\/span> e:\n   # handle exceptoin<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
To add an item to a sized limited queue and block with a timeout, you can use the timeout<\/code> parameter like this:<\/p>\n\n\n
try<\/span>:\n   queue.put(item, timeout=3<\/span>)\nexcept<\/span> queue.Full as<\/span> e:\n   # handle exceptoin<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Getting an item from the queue #<\/a><\/h3>\n\n\n\n
To get an item from the queue, you can use the get()<\/code> method:<\/p>\n\n\n
item = queue.get()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
The get()<\/code> method will block until an item is available for retrieval from the queue.<\/p>\n\n\n\n
To get an item from the queue without blocking, you can set the block parameter to False<\/code>:<\/p>\n\n\n
try<\/span>:\n   queue.get(block=False<\/span>)\nexcept<\/span> queue.Empty:\n   # handle exception<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
To get an item from the queue and block it with a time limit, you can use the get()<\/code> method with a timeout:<\/p>\n\n\n
try<\/span>:\n   item = queue.get(timeout=10<\/span>)\nexcept<\/span> queue.Empty:\n   # ...<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Getting the size of the queue #<\/a><\/h3>\n\n\n\n
The qsize()<\/code> method returns the number of items in the queue:<\/p>\n\n\n
size = queue.size()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Also, the empty()<\/code> method returns True<\/code> if the queue is empty or False otherwise. On the other hand, the full()<\/code> method returns True if the queue is full or False<\/code> otherwise.<\/p>\n\n\n\n
Marking a task as completed #<\/a><\/h3>\n\n\n\n
An item that you add to the queue represents a unit of work or a task. <\/p>\n\n\n\n
When a thread calls the get()<\/code> method to get the item from the queue, it may need to process it before the task is considered completed.<\/p>\n\n\n\n
Once completed, the thread may call the task_done()<\/code> method of the queue to indicate that it has processed the task completely:<\/p>\n\n\n
item = queue.get()\n\n# process the item<\/span>\n# ...<\/span>\n\n# mark the item as completed<\/span>\nqueue.task_done()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Waiting for all tasks on the queue to be completed #<\/a><\/h3>\n\n\n\n
To wait for all tasks on the queue to be completed, you can call the join()<\/code> method on the queue object:<\/p>\n\n\n
queue.join()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Python thread-safe queue example #<\/a><\/h2>\n\n\n\n
The following example illustrates how to use the thread-safe queue to exchange data between two threads:<\/p>\n\n\n
import<\/span> time\nfrom<\/span> queue import<\/span> Empty, Queue\nfrom<\/span> threading import<\/span> Thread\n\n\ndef<\/span> producer<\/span>(queue)<\/span>:<\/span>\n    for<\/span> i in<\/span> range(1<\/span>, 6<\/span>):\n        print(f'Inserting item {i}<\/span> into the queue'<\/span>)\n        time.sleep(1<\/span>)\n        queue.put(i)\n\n\ndef<\/span> consumer<\/span>(queue)<\/span>:<\/span>\n    while<\/span> True<\/span>:\n        try<\/span>:\n            item = queue.get()\n        except<\/span> Empty:\n            continue<\/span>\n        else<\/span>:\n            print(f'Processing item {item}<\/span>'<\/span>)\n            time.sleep(2<\/span>)\n            queue.task_done()\n\n\ndef<\/span> main<\/span>()<\/span>:<\/span>\n    queue = Queue()\n\n    # create a producer thread and start it<\/span>\n    producer_thread = Thread(\n        target=producer,\n        args=(queue,)\n    )\n    producer_thread.start()\n\n    # create a consumer thread and start it<\/span>\n    consumer_thread = Thread(\n        target=consumer,\n        args=(queue,),\n        daemon=True<\/span>\n    )\n    consumer_thread.start()\n\n    # wait for all tasks to be added to the queue<\/span>\n    producer_thread.join()\n\n    # wait for all tasks on the queue to be completed<\/span>\n    queue.join()\n\n\nif<\/span> __name__ == '__main__'<\/span>:\n    main()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
How it works.<\/p>\n\n\n\n
First, define the producer()<\/code> function that adds numbers from 1 to 11 to the queue. It delays one second in each iteration:<\/p>\n\n\n
def<\/span> producer<\/span>(queue)<\/span>:<\/span>\n    for<\/span> i in<\/span> range(1<\/span>, 6<\/span>):\n        print(f'Inserting item {i}<\/span> into the queue'<\/span>)\n        time.sleep(1<\/span>)\n        queue.put(i)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Second, define the consumer()<\/code> function that gets an item from the queue and processes it. It delays two seconds after processing each item on the queue:<\/p>\n\n\n
def<\/span> consumer<\/span>(queue)<\/span>:<\/span>\n    while<\/span> True<\/span>:\n        try<\/span>:\n            item = queue.get()\n        except<\/span> Empty:\n            continue<\/span>\n        else<\/span>:\n            print(f'Processing item {item}<\/span>'<\/span>)\n            time.sleep(2<\/span>)\n            queue.task_done()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
The queue.task_done()<\/code> indicates that the function has processed the item on the queue.<\/p>\n\n\n\n
Third, define the main()<\/code> function that creates two threads, one thread adds a number to the queue every second while another thread processes an item on the queue every two seconds:<\/p>\n\n\n
def<\/span> main<\/span>()<\/span>:<\/span>\n    queue = Queue()\n\n    # create a producer thread and start it<\/span>\n    producer_thread = Thread(\n        target=producer,\n        args=(queue,)\n    )\n    producer_thread.start()\n\n    # create a consumer thread and start it<\/span>\n    consumer_thread = Thread(\n        target=consumer,\n        args=(queue,),\n        daemon=True<\/span>\n    )\n    consumer_thread.start()\n\n    # wait for all tasks to be added to the queue<\/span>\n    producer_thread.join()\n\n    # wait for all tasks on the queue to be completed<\/span>\n    queue.join()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Output:<\/p>\n\n\n
Inserting item 1<\/span> into the queue\nInserting item 2<\/span> into the queue\nProcessing item 1<\/span>\nInserting item 3<\/span> into the queue\nProcessing item 2<\/span>\nInserting item 4<\/span> into the queue\nInserting item 5<\/span> into the queue\nProcessing item 3<\/span>\nProcessing item 4<\/span>\nProcessing item 5<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
The following are steps in the main()<\/code> function:<\/p>\n\n\n\n
    \n
  1. Create a new queue by calling the Queue()<\/code> constructor<\/li>\n\n\n\n
  2. Create a new thread called producer_thread<\/code> and start it immediately<\/li>\n\n\n\n
  3. Create a daemon thread<\/a> called consumer_thread<\/code> and start it immediately.<\/li>\n\n\n\n
  4. Wait for all the numbers to be added to the queue using the join()<\/code> method of the thread.<\/li>\n\n\n\n
  5. Wait for all the tasks on the queue to be completed by calling the join()<\/code> method of the queue.<\/li>\n<\/ol>\n\n\n\n
    The producer adds a number to the queue every second, and the consumer process a number from the queue every two seconds. It also displays the numbers on the queue every second.<\/p>\n\n\n\n
    Summary #<\/a><\/h2>\n\n\n\n
      \n
    • Use the Queue<\/code> class of the queue<\/code> module to safely exchange data between multiple threads.<\/li>\n<\/ul>\n
      \n\t
      \n\t\t
      Was this tutorial helpful ?<\/div>\n\t\t
      \n\t\t\t\n\t\t\t\t\n\t\t\t\t\t<\/path>\n\t\t\t\t<\/svg>\n\t\t\t\t Yes <\/span>\n\t\t\t<\/button>\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t<\/path>\n\t\t\t\t<\/svg>\n\t\t\t\t No <\/span>\n\t\t\t<\/button>\n\t\t<\/div>\n\t<\/header>\n\n\t
      \n\t\t
      \n\t\t\t
      <\/div>\n\t\t\t