{"id":3510,"date":"2022-05-16T02:22:59","date_gmt":"2022-05-16T02:22:59","guid":{"rendered":"https:\/\/www.pythontutorial.net\/?page_id=3510"},"modified":"2023-06-02T12:15:28","modified_gmt":"2023-06-02T12:15:28","slug":"python-multiprocessing","status":"publish","type":"page","link":"https:\/\/www.pythontutorial.net\/python-concurrency\/python-multiprocessing\/","title":{"rendered":"Python Multiprocessing"},"content":{"rendered":"\n

Summary<\/strong>: in this tutorial, you’ll learn how to run code in parallel using the Python multiprocessing module.<\/p>\n\n\n\n

Introduction to the Python multiprocessing #<\/a><\/h2>\n\n\n\n

Generally, programs deal with two types of tasks:<\/p>\n\n\n\n

    \n
  1. I\/O-bound tasks: if a task does a lot of input\/output operations, it’s called an I\/O-bound task. Typical examples of I\/O-bound tasks are reading from files, writing to files, connecting to databases, and making a network request. For I\/O-bound tasks, you can use multithreading<\/a> to speed them up.<\/li>\n\n\n\n
  2. CPU-bound tasks: when a task does a lot of operations using a CPU, it’s called a CPU-bound task. For example, number calculation, image resizing, and video streaming are CPU-bound tasks. To speed up the program with lots of CPU-bound tasks, you use multiprocessing.<\/li>\n<\/ol>\n\n\n\n

    Multiprocessing allows two or more processors to simultaneously process two or more different parts of a program. <\/p>\n\n\n\n

    In Python, you use the multiprocessing<\/code> module to implement multiprocessing.<\/p>\n\n\n\n

    Python multiprocessing example #<\/a><\/h2>\n\n\n\n

    See the following program:<\/p>\n\n\n

    import<\/span> time\r\n\r\ndef<\/span> task<\/span>()<\/span>:<\/span>\r\n    result = 0<\/span>\r\n    for<\/span> _ in<\/span> range(10<\/span>**8<\/span>):\r\n        result += 1<\/span>\r\n    return<\/span> result\r\n\r\nif<\/span> __name__ == '__main__'<\/span>:\r\n    start = time.perf_counter()\r\n    task()\r\n    task()\r\n    finish = time.perf_counter()\r\n\r\n    print(f'It took {finish-start:.2<\/span>f}<\/span> second(s) to finish'<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Output:<\/p>\n\n\n
    It took  5.55<\/span> second(s) to finish<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    How it works.<\/p>\n\n\n\n
    First, define the task()<\/code><\/code> function is a CPU-bound task because it performs a heavy computation by executing a loop for 100 million iterations and incrementing a variable result<\/code>:<\/p>\n\n\n
    def<\/span> task<\/span>()<\/span>:<\/span>\r\n    result = 0<\/span>\r\n    for<\/span> _ in<\/span> range(10<\/span>**8<\/span>):\r\n        result += 1<\/span>\r\n    return<\/span> result<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Second, call the task()<\/code> functions twice and record the processing time:<\/p>\n\n\n
    if<\/span> __name__ == '__main__'<\/span>:\n    start = time.perf_counter()\n    task()\n    task()\n    finish = time.perf_counter()\n\n    print(f'It took {finish-start: .2<\/span>f}<\/span> second(s) to finish'<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    On our computer, it took 5.55 seconds to complete.<\/p>\n\n\n\n
    Using multiprocessing module #<\/a><\/h3>\n\n\n\n
    The following program uses the multiprocessing module but takes less time:<\/p>\n\n\n
    import<\/span> time\r\nimport<\/span> multiprocessing\r\n\r\ndef<\/span> task<\/span>()<\/span> -> int:<\/span>\r\n    result = 0<\/span>\r\n    for<\/span> _ in<\/span> range(10<\/span>**8<\/span>):\r\n        result += 1<\/span>\r\n    return<\/span> result\r\n\r\nif<\/span> __name__ == '__main__'<\/span>:\r\n    start = time.perf_counter()\r\n\r\n    p1 = multiprocessing.Process(target=task)\r\n    p2 = multiprocessing.Process(target=task)\r\n\r\n    p1.start()\r\n    p2.start()\r\n\r\n    p1.join()\r\n    p2.join()\r\n\r\n    finish = time.perf_counter()\r\n    print(f'It took {finish-start:.2<\/span>f}<\/span> second(s) to finish'<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Output:<\/p>\n\n\n
    It took  3.43<\/span> second(s) to finish<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    How it works.<\/p>\n\n\n\n
    First, import the multiprocessing module:<\/p>\n\n\n
    import<\/span> multiprocessing<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Second, create two processes and pass the task function to each:<\/p>\n\n\n
    p1 = multiprocessing.Process(target=task)\np2 = multiprocessing.Process(target=task)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Note that the Process()<\/code> constructor returns a new Process<\/code> object.<\/p>\n\n\n\n
    Third, call the start()<\/code> method of the Process<\/code> objects to start the process:<\/p>\n\n\n
    p1.start()\np2.start()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Finally, wait for the processes to complete by calling the join()<\/code> method:<\/p>\n\n\n
    p1.join()\np2.join()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Python multiprocessing practical example #<\/a><\/h2>\n\n\n\n
    We’ll use the multiprocessing module to resize the high-resolution images.<\/p>\n\n\n\n
    First, install the Pillow<\/code> library for image processing:<\/p>\n\n\n
    pip install Pillow<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Second, develop a program that creates thumbnails of the pictures in the images<\/code> folder and save them to the thumbs<\/code> folder:<\/p>\n\n\n
    import<\/span> time\r\nimport<\/span> os\r\nfrom<\/span> PIL import<\/span> Image, ImageFilter\r\n\r\nfilenames = [\r\n    'images\/1.jpg'<\/span>,\r\n    'images\/2.jpg'<\/span>,\r\n    'images\/3.jpg'<\/span>,\r\n    'images\/4.jpg'<\/span>,\r\n    'images\/5.jpg'<\/span>,\r\n]\r\n\r\ndef<\/span> create_thumbnail<\/span>(filename, size=(50<\/span>,50<\/span>)<\/span>, thumb_dir ='thumbs'<\/span>)<\/span>:<\/span>\r\n    # open the image<\/span>\r\n    img = Image.open(filename)\r\n    \r\n    # apply the gaussian blur filter<\/span>\r\n    img = img.filter(ImageFilter.GaussianBlur())\r\n\r\n    # create a thumbnail<\/span>\r\n    img.thumbnail(size)\r\n    \r\n    # save the image<\/span>\r\n    img.save(f'{thumb_dir}<\/span>\/{os.path.basename(filename)}<\/span>'<\/span>)\r\n\r\n    # display a message<\/span>\r\n    print(f'{filename}<\/span> was processed...'<\/span>)\r\n\r\n\r\nif<\/span> __name__ == '__main__'<\/span>:\r\n    start = time.perf_counter()\r\n\r\n    for<\/span> filename in<\/span> filenames:\r\n        create_thumbnail(filename)\r\n        \r\n    finish = time.perf_counter()\r\n\r\n    print(f'It took {finish-start:.2<\/span>f}<\/span> second(s) to finish'<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    On our computer, it took about 4.06 seconds to complete:<\/p>\n\n\n
    images\/1.j<\/span>pg was processed...\r\nimages\/2.j<\/span>pg was processed...\r\nimages\/3.j<\/span>pg was processed...\r\nimages\/4.j<\/span>pg was processed...\r\nimages\/5.j<\/span>pg was processed...\r\nIt took 4.06<\/span> second(s) to finish<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Third, modify the program to use multiprocessing. Each process will create a thumbnail for a picture:<\/p>\n\n\n
    import<\/span> time\r\nimport<\/span> os\r\nfrom<\/span> PIL import<\/span> Image, ImageFilter\r\n\r\nimport<\/span> multiprocessing\r\n\r\nfilenames = [\r\n    'images\/1.jpg'<\/span>,\r\n    'images\/2.jpg'<\/span>,\r\n    'images\/3.jpg'<\/span>,\r\n    'images\/4.jpg'<\/span>,\r\n    'images\/5.jpg'<\/span>,\r\n]\r\n\r\ndef<\/span> create_thumbnail<\/span>(filename, size=(50<\/span>,50<\/span>)<\/span>, thumb_dir ='thumbs'<\/span>)<\/span>:<\/span>\r\n    # open the image<\/span>\r\n    img = Image.open(filename)\r\n    \r\n    # apply the gaussian blur filter<\/span>\r\n    img = img.filter(ImageFilter.GaussianBlur())\r\n\r\n    # create a thumbnail<\/span>\r\n    img.thumbnail(size)\r\n    \r\n    # save the image<\/span>\r\n    img.save(f'{thumb_dir}<\/span>\/{os.path.basename(filename)}<\/span>'<\/span>)\r\n\r\n    # display a message<\/span>\r\n    print(f'{filename}<\/span> was processed...'<\/span>)\r\n\r\ndef<\/span> main<\/span>()<\/span>:<\/span>\r\n    start = time.perf_counter()\r\n\r\n    # create processes<\/span>\r\n    processes = [multiprocessing.Process(target=create_thumbnail, args=[filename]) \r\n                for<\/span> filename in<\/span> filenames]\r\n\r\n    # start the processes<\/span>\r\n    for<\/span> process in<\/span> processes:\r\n        process.start()\r\n\r\n    # wait for completion<\/span>\r\n    for<\/span> process in<\/span> processes:\r\n        process.join()\r\n\r\n    finish = time.perf_counter()\r\n\r\n    print(f'It took {finish-start:.2<\/span>f}<\/span> second(s) to finish'<\/span>)\r\n\r\n\r\nif<\/span> __name__ == '__main__'<\/span>:\r\n    main()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    Output:<\/p>\n\n\n
    images\/5.j<\/span>pg was processed...\nimages\/4.j<\/span>pg was processed...\nimages\/1.j<\/span>pg was processed...\nimages\/3.j<\/span>pg was processed...\nimages\/2.j<\/span>pg was processed...\nIt took 2.92<\/span> second(s) to finish<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
    In this case, the output shows that the program processed the pictures faster.<\/p>\n\n\n\n
    Summary #<\/a><\/h2>\n\n\n\n
      \n
    • Use Python multiprocessing to run code in parallel to deal with CPU-bound tasks.<\/li>\n<\/ul>\n
      \n\t
      \n\t\t
      Was this tutorial helpful ?<\/div>\n\t\t
      \n\t\t\t\n\t\t\t\t\n\t\t\t\t\t<\/path>\n\t\t\t\t<\/svg>\n\t\t\t\t Yes <\/span>\n\t\t\t<\/button>\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t<\/path>\n\t\t\t\t<\/svg>\n\t\t\t\t No <\/span>\n\t\t\t<\/button>\n\t\t<\/div>\n\t<\/header>\n\n\t
      \n\t\t
      \n\t\t\t
      <\/div>\n\t\t\t