{"id":3521,"date":"2022-05-16T03:54:25","date_gmt":"2022-05-16T03:54:25","guid":{"rendered":"https:\/\/www.pythontutorial.net\/?page_id=3521"},"modified":"2023-06-02T12:20:09","modified_gmt":"2023-06-02T12:20:09","slug":"python-processpoolexecutor","status":"publish","type":"page","link":"https:\/\/www.pythontutorial.net\/python-concurrency\/python-processpoolexecutor\/","title":{"rendered":"Python ProcessPoolExecutor"},"content":{"rendered":"\n<p><strong>Summary<\/strong>: in this tutorial, you&#8217;ll learn how to use the Python <code>ProcessPoolExecutor<\/code> to create and manage a process pool effectively.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id='introduction-to-the-python-processpoolexecutor-class'>Introduction to the Python ProcessPoolExecutor class <a href=\"#introduction-to-the-python-processpoolexecutor-class\" class=\"anchor\" id=\"introduction-to-the-python-processpoolexecutor-class\" title=\"Anchor for Introduction to the Python ProcessPoolExecutor class\">#<\/a><\/h2>\n\n\n\n<p><a href=\"https:\/\/www.pythontutorial.net\/advanced-python\/python-multiprocessing\/\">In the previous tutorial<\/a>, you learned how to run code in parallel by creating processes manually using the <code>Process<\/code> class from the <code>multiprocessing<\/code> module. However, manually creating processes is not efficient.<\/p>\n\n\n\n<p>To manage the processes more efficiently, you can use a process pool. Like a <a href=\"https:\/\/www.pythontutorial.net\/advanced-python\/python-threadpoolexecutor\/\">thread pool<\/a>, a process pool is a pattern for managing processes automatically.<\/p>\n\n\n\n<p>The <code>ProcessPoolExecutor<\/code> class from the <code>concurrent.futures<\/code> module allows you to create and manage a process pool. <\/p>\n\n\n\n<p>For example, the <code>ProcessPoolExecutor<\/code> class uses the number of CPU cores for creating an optimized number of processes to create.<\/p>\n\n\n\n<p>The <code>ProcessPoolExecutor<\/code> extends the <code>Executor<\/code> class that has three methods:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>submit()<\/code> &#8211; dispatch a function to be executed by the process and return a Future object.<\/li>\n\n\n\n<li><code>map()<\/code> &#8211; call a function to an iterable of elements.<\/li>\n\n\n\n<li><code>shutdown()<\/code> &#8211; shut down the executor.<\/li>\n<\/ul>\n\n\n\n<p>To release the resources held by the executor, you need to call the <code>shutdown()<\/code> method explicitly. To shut down the executor automatically, you can use a <a href=\"https:\/\/www.pythontutorial.net\/advanced-python\/python-context-managers\/\">context manager<\/a>.<\/p>\n\n\n\n<p>The <code>Future<\/code> object represents an eventful result of an asynchronous operation. It has two main methods for getting the result:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>result()<\/code> &#8211; return the result from the asynchronous operation.<\/li>\n\n\n\n<li><code>exception()<\/code> &#8211; return an exception that occurred while running the asynchronous operation.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id='python-processpoolexecutor-example'>Python ProcessPoolExecutor example <a href=\"#python-processpoolexecutor-example\" class=\"anchor\" id=\"python-processpoolexecutor-example\" title=\"Anchor for Python ProcessPoolExecutor example\">#<\/a><\/h2>\n\n\n\n<p>The following program uses a process pool to create thumbnails for pictures in the <code>images<\/code> folder and save them to the <code>thumbs<\/code> folder.<\/p>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-1\" data-shcb-language-name=\"Python\" data-shcb-language-slug=\"python\"><span><code class=\"hljs language-python\"><span class=\"hljs-keyword\">import<\/span> time\r\n<span class=\"hljs-keyword\">import<\/span> os\r\n<span class=\"hljs-keyword\">from<\/span> PIL <span class=\"hljs-keyword\">import<\/span> Image, ImageFilter\r\n\r\n<span class=\"hljs-keyword\">from<\/span> concurrent.futures <span class=\"hljs-keyword\">import<\/span> ProcessPoolExecutor\r\n\r\nfilenames = &#91;\r\n    <span class=\"hljs-string\">'images\/1.jpg'<\/span>,\r\n    <span class=\"hljs-string\">'images\/2.jpg'<\/span>,\r\n    <span class=\"hljs-string\">'images\/3.jpg'<\/span>,\r\n    <span class=\"hljs-string\">'images\/4.jpg'<\/span>,\r\n    <span class=\"hljs-string\">'images\/5.jpg'<\/span>,\r\n]\r\n\r\n<span class=\"hljs-function\"><span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title\">create_thumbnail<\/span><span class=\"hljs-params\">(filename, size=<span class=\"hljs-params\">(<span class=\"hljs-number\">50<\/span>,<span class=\"hljs-number\">50<\/span>)<\/span>, thumb_dir =<span class=\"hljs-string\">'thumbs'<\/span>)<\/span>:<\/span>\r\n    <span class=\"hljs-comment\"># open the image<\/span>\r\n    img = Image.open(filename)\r\n    \r\n    <span class=\"hljs-comment\"># apply the gaussian blur filter<\/span>\r\n    img = img.filter(ImageFilter.GaussianBlur())\r\n\r\n    <span class=\"hljs-comment\"># create a thumbnail<\/span>\r\n    img.thumbnail(size)\r\n    \r\n    <span class=\"hljs-comment\"># save the image<\/span>\r\n    img.save(<span class=\"hljs-string\">f'<span class=\"hljs-subst\">{thumb_dir}<\/span>\/<span class=\"hljs-subst\">{os.path.basename(filename)}<\/span>'<\/span>)\r\n\r\n    <span class=\"hljs-comment\"># display a message<\/span>\r\n    print(<span class=\"hljs-string\">f'<span class=\"hljs-subst\">{filename}<\/span> was processed...'<\/span>)\r\n\r\n<span class=\"hljs-function\"><span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title\">main<\/span><span class=\"hljs-params\">()<\/span>:<\/span>\r\n    start = time.perf_counter()\r\n\r\n    <span class=\"hljs-keyword\">with<\/span> ProcessPoolExecutor() <span class=\"hljs-keyword\">as<\/span> executor:\r\n        executor.map(create_thumbnail, filenames)\r\n   \r\n    finish = time.perf_counter()\r\n\r\n    print(<span class=\"hljs-string\">f'It took <span class=\"hljs-subst\">{finish-start: <span class=\"hljs-number\">.2<\/span>f}<\/span> second(s) to finish'<\/span>)\r\n\r\n<span class=\"hljs-keyword\">if<\/span> __name__ == <span class=\"hljs-string\">'__main__'<\/span>:\r\n    main()<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-1\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">Python<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">python<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<p>Output:<\/p>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-2\" data-shcb-language-name=\"Python\" data-shcb-language-slug=\"python\"><span><code class=\"hljs language-python\">images\/<span class=\"hljs-number\">5.j<\/span>pg was processed...\nimages\/<span class=\"hljs-number\">4.j<\/span>pg was processed...\nimages\/<span class=\"hljs-number\">3.j<\/span>pg was processed...\nimages\/<span class=\"hljs-number\">2.j<\/span>pg was processed...\nimages\/<span class=\"hljs-number\">1.j<\/span>pg was processed...\nIt took  <span class=\"hljs-number\">0.79<\/span> second(s) to finish<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-2\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">Python<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">python<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<p>Note that to run the program, you need to install the <code>Pillow<\/code> which is a popular library for image processing by running the pip command <code>pip install Pillow<\/code>.<\/p>\n\n\n\n<p>How it works.<\/p>\n\n\n\n<p>First, declare a list of files for creating thumbnails:<\/p>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-3\" data-shcb-language-name=\"Python\" data-shcb-language-slug=\"python\"><span><code class=\"hljs language-python\">filenames = &#91;\n    <span class=\"hljs-string\">'images\/1.jpg'<\/span>,\n    <span class=\"hljs-string\">'images\/2.jpg'<\/span>,\n    <span class=\"hljs-string\">'images\/3.jpg'<\/span>,\n    <span class=\"hljs-string\">'images\/4.jpg'<\/span>,\n    <span class=\"hljs-string\">'images\/5.jpg'<\/span>,\n]<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-3\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">Python<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">python<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<p>Second, define a function that creates a thumbnail from an image file and saves the output to the thumbnail folder:<\/p>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-4\" data-shcb-language-name=\"Python\" data-shcb-language-slug=\"python\"><span><code class=\"hljs language-python\"><span class=\"hljs-function\"><span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title\">create_thumbnail<\/span><span class=\"hljs-params\">(filename, size=<span class=\"hljs-params\">(<span class=\"hljs-number\">50<\/span>,<span class=\"hljs-number\">50<\/span>)<\/span>, thumb_dir =<span class=\"hljs-string\">'thumbs'<\/span>)<\/span>:<\/span>\r\n    <span class=\"hljs-comment\"># open the image<\/span>\r\n    img = Image.open(filename)\r\n    \r\n    <span class=\"hljs-comment\"># apply the gaussian blur filter<\/span>\r\n    img = img.filter(ImageFilter.GaussianBlur())\r\n\r\n    <span class=\"hljs-comment\"># create a thumbnail<\/span>\r\n    img.thumbnail(size)\r\n    \r\n    <span class=\"hljs-comment\"># save the image<\/span>\r\n    img.save(<span class=\"hljs-string\">f'<span class=\"hljs-subst\">{thumb_dir}<\/span>\/<span class=\"hljs-subst\">{os.path.basename(filename)}<\/span>'<\/span>)\r\n\r\n    <span class=\"hljs-comment\"># display a message<\/span>\r\n    print(<span class=\"hljs-string\">f'<span class=\"hljs-subst\">{filename}<\/span> was processed...'<\/span>)<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-4\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">Python<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">python<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<p>Third, create a process pool and call the <code>create_thumbnail()<\/code> function for each picture specified in the filenames:<\/p>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-5\" data-shcb-language-name=\"Python\" data-shcb-language-slug=\"python\"><span><code class=\"hljs language-python\"><span class=\"hljs-keyword\">with<\/span> ProcessPoolExecutor() <span class=\"hljs-keyword\">as<\/span> executor:\n    executor.map(create_thumbnail, filenames)<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-5\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">Python<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">python<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<h2 class=\"wp-block-heading\" id='summary'>Summary <a href=\"#summary\" class=\"anchor\" id=\"summary\" title=\"Anchor for Summary\">#<\/a><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use the Python ProcessPoolExecutor class to create and manage a process pool automatically.<\/li>\n<\/ul>\n<div class=\"helpful-block-content\" data-title=\"\">\n\t<header>\n\t\t<div class=\"wth-question\">Was this tutorial helpful ?<\/div>\n\t\t<div class=\"wth-thumbs\">\n\t\t\t<button\n\t\t\t\tdata-post=\"3521\"\n\t\t\t\tdata-post-url=\"https:\/\/www.pythontutorial.net\/python-concurrency\/python-processpoolexecutor\/\"\n\t\t\t\tdata-post-title=\"Python ProcessPoolExecutor\"\n\t\t\t\tdata-response=\"1\"\n\t\t\t\tclass=\"wth-btn-rounded wth-yes-btn\"\n\t\t\t>\n\t\t\t\t<svg\n\t\t\t\t\txmlns=\"http:\/\/www.w3.org\/2000\/svg\"\n\t\t\t\t\tviewBox=\"0 0 24 24\"\n\t\t\t\t\tfill=\"none\"\n\t\t\t\t\tstroke=\"currentColor\"\n\t\t\t\t\tstroke-width=\"2\"\n\t\t\t\t\tstroke-linecap=\"round\"\n\t\t\t\t\tstroke-linejoin=\"round\"\n\t\t\t\t\tclass=\"feather feather-thumbs-up block w-full h-full\"\n\t\t\t\t>\n\t\t\t\t\t<path\n\t\t\t\t\t\td=\"M14 9V5a3 3 0 0 0-3-3l-4 9v11h11.28a2 2 0 0 0 2-1.7l1.38-9a2 2 0 0 0-2-2.3zM7 22H4a2 2 0 0 1-2-2v-7a2 2 0 0 1 2-2h3\"\n\t\t\t\t\t><\/path>\n\t\t\t\t<\/svg>\n\t\t\t\t<span class=\"sr-only\"> Yes <\/span>\n\t\t\t<\/button>\n\n\t\t\t<button\n\t\t\t\tdata-response=\"0\"\n\t\t\t\tdata-post=\"3521\"\n\t\t\t\tdata-post-url=\"https:\/\/www.pythontutorial.net\/python-concurrency\/python-processpoolexecutor\/\"\n\t\t\t\tdata-post-title=\"Python ProcessPoolExecutor\"\n\t\t\t\tclass=\"wth-btn-rounded wth-no-btn\"\n\t\t\t>\n\t\t\t\t<svg\n\t\t\t\t\txmlns=\"http:\/\/www.w3.org\/2000\/svg\"\n\t\t\t\t\tviewBox=\"0 0 24 24\"\n\t\t\t\t\tfill=\"none\"\n\t\t\t\t\tstroke=\"currentColor\"\n\t\t\t\t\tstroke-width=\"2\"\n\t\t\t\t\tstroke-linecap=\"round\"\n\t\t\t\t\tstroke-linejoin=\"round\"\n\t\t\t\t>\n\t\t\t\t\t<path\n\t\t\t\t\t\td=\"M10 15v4a3 3 0 0 0 3 3l4-9V2H5.72a2 2 0 0 0-2 1.7l-1.38 9a2 2 0 0 0 2 2.3zm7-13h2.67A2.31 2.31 0 0 1 22 4v7a2.31 2.31 0 0 1-2.33 2H17\"\n\t\t\t\t\t><\/path>\n\t\t\t\t<\/svg>\n\t\t\t\t<span class=\"sr-only\"> No <\/span>\n\t\t\t<\/button>\n\t\t<\/div>\n\t<\/header>\n\n\t<div class=\"wth-form hidden\">\n\t\t<div class=\"wth-form-wrapper\">\n\t\t\t<div class=\"wth-title\"><\/div>\n\t\t\t<textarea class=\"wth-message\"><\/textarea>\n\t\t\t<input type=\"button\" name=\"wth-submit\" class=\"wth-btn wth-btn-submit\" id=\"wth-submit\" \/>\n\t\t\t<input type=\"button\" class=\"wth-btn wth-btn-cancel\" value=\"Cancel\" \/>\n\t\t<\/div>\n\t<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you&#8217;ll learn how to use the Python ProcessPoolExecutor executor to create and manage a process pool effectively.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":4104,"menu_order":13,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-3521","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/pages\/3521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/comments?post=3521"}],"version-history":[{"count":0,"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/pages\/3521\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/pages\/4104"}],"wp:attachment":[{"href":"https:\/\/www.pythontutorial.net\/wp-json\/wp\/v2\/media?parent=3521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}