{"id":16716,"date":"2017-03-28T12:15:32","date_gmt":"2017-03-28T09:15:32","guid":{"rendered":"https:\/\/www.webcodegeeks.com\/?p=16716"},"modified":"2017-03-27T12:03:55","modified_gmt":"2017-03-27T09:03:55","slug":"luigi-externalprogramtask-example-converting-json-csv","status":"publish","type":"post","link":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/","title":{"rendered":"Luigi: An ExternalProgramTask example – Converting JSON to CSV"},"content":{"rendered":"

I\u2019ve been playing around with the Python library Luigi<\/a> which is used to build pipelines of batch jobs and I struggled to find an example of an ExternalProgramTask<\/a> so this is my attempt at filling that void.
\n 
\n 
\n 
\n 
\n 
\n 
\n 
\n 
\n 
\n\"\"<\/a><\/p>\n

I\u2019m building a little data pipeline to get data from the meetup.com API<\/a> and put it into CSV files that can be loaded into Neo4j<\/a> using the LOAD CSV<\/a> command.<\/p>\n

The first task I created calls the \/groups<\/cite> endpoint and saves the result into a JSON file:<\/p>\n

import luigi\r\nimport requests\r\nimport json\r\nfrom collections import Counter\r\n\u00a0\r\nclass GroupsToJSON(luigi.Task):\r\n    key = luigi.Parameter()\r\n    lat = luigi.Parameter()\r\n    lon = luigi.Parameter()\r\n\u00a0\r\n    def run(self):\r\n        seed_topic = \"nosql\"\r\n        uri = \"https:\/\/api.meetup.com\/2\/groups?\u22a4ic={0}\u2aab={1}&lon={2}&key={3}\".format(seed_topic, self.lat, self.lon, self.key)\r\n\u00a0\r\n        r = requests.get(uri)\r\n        all_topics = [topic[\"urlkey\"]  for result in r.json()[\"results\"] for topic in result[\"topics\"]]\r\n        c = Counter(all_topics)\r\n\u00a0\r\n        topics = [entry[0] for entry in c.most_common(10)]\r\n\u00a0\r\n        groups = {}\r\n        for topic in topics:\r\n            uri = \"https:\/\/api.meetup.com\/2\/groups?\u22a4ic={0}\u2aab={1}&lon={2}&key={3}\".format(topic, self.lat, self.lon, self.key)\r\n            r = requests.get(uri)\r\n            for group in r.json()[\"results\"]:\r\n                groups[group[\"id\"]] = group\r\n\u00a0\r\n        with self.output().open('w') as groups_file:\r\n            json.dump(list(groups.values()), groups_file, indent=4, sort_keys=True)\r\n\u00a0\r\n    def output(self):\r\n        return luigi.LocalTarget(\"\/tmp\/groups.json\")<\/pre>\n
We define a few parameters at the top of the class which will be passed in when this task is executed. The most interesting lines of the run<\/cite> function are the last couple where we write the JSON to a file. self.output()<\/cite> refers to the target defined in the output<\/cite> function which in this case is \/tmp\/groups.json<\/cite>.<\/p>\n
Now we need to create a task to convert that JSON file into CSV format. The jq<\/a> command line tool does this job well so we\u2019ll use that. The following task does the job:<\/p>\n
from luigi.contrib.external_program import ExternalProgramTask\r\n\u00a0\r\nclass GroupsToCSV(luigi.contrib.external_program.ExternalProgramTask):\r\n    file_path = \"\/tmp\/groups.csv\"\r\n    key = luigi.Parameter()\r\n    lat = luigi.Parameter()\r\n    lon = luigi.Parameter()\r\n\u00a0\r\n    def program_args(self):\r\n        return [\".\/groups.sh\", self.input()[0].path, self.output().path]\r\n\u00a0\r\n    def output(self):\r\n        return luigi.LocalTarget(self.file_path)\r\n\u00a0\r\n    def requires(self):\r\n        yield GroupsToJSON(self.key, self.lat, self.lon)<\/pre>\n
groups.sh<\/cite><\/p>\n
#!\/bin\/bash\r\n\u00a0\r\nin=${1}\r\nout=${2}\r\n\u00a0\r\necho \"id,name,urlname,link,rating,created,description,organiserName,organiserMemberId\" > ${out}\r\njq -r '.[] | [.id, .name, .urlname, .link, .rating, .created, .description, .organizer.name, .organizer.member_id] | @csv' ${in} >> ${out}<\/pre>\n
I wanted to call jq<\/cite> directly from the Python code but I couldn\u2019t figure out how to do it so putting that code in a shell script is my workaround.<\/p>\n
The last piece of the puzzle is a wrapper task that launches the others:<\/p>\n
import os\r\n\u00a0\r\nclass Meetup(luigi.WrapperTask):\r\n    def run(self):\r\n        print(\"Running Meetup\")\r\n\u00a0\r\n    def requires(self):\r\n        key = os.environ['MEETUP_API_KEY']\r\n        lat = os.getenv('LAT', \"51.5072\")\r\n        lon = os.getenv('LON', \"0.1275\")\r\n\u00a0\r\n        yield GroupsToCSV(key, lat, lon)<\/pre>\n
Now we\u2019re ready to run the tasks:<\/p>\n
$ PYTHONPATH=\".\" luigi --module blog --local-scheduler Meetup\r\nDEBUG: Checking if Meetup() is complete\r\nDEBUG: Checking if GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275) is complete\r\nINFO: Informed scheduler that task   Meetup__99914b932b   has status   PENDING\r\nDEBUG: Checking if GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275) is complete\r\nINFO: Informed scheduler that task   GroupsToCSV_xxx_51_5072_0_1275_e07372cebf   has status   PENDING\r\nINFO: Informed scheduler that task   GroupsToJSON_xxx_51_5072_0_1275_e07372cebf   has status   PENDING\r\nINFO: Done scheduling tasks\r\nINFO: Running Worker with 1 processes\r\nDEBUG: Asking scheduler for work...\r\nDEBUG: Pending tasks: 3\r\nINFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) running   GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275)\r\nINFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) done      GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275)\r\nDEBUG: 1 running tasks, waiting for next task to finish\r\nINFO: Informed scheduler that task   GroupsToJSON_xxx_51_5072_0_1275_e07372cebf   has status   DONE\r\nDEBUG: Asking scheduler for work...\r\nDEBUG: Pending tasks: 2\r\nINFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) running   GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275)\r\nINFO: Running command: .\/groups.sh \/tmp\/groups.json \/tmp\/groups.csv\r\nINFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) done      GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275)\r\nDEBUG: 1 running tasks, waiting for next task to finish\r\nINFO: Informed scheduler that task   GroupsToCSV_xxx_51_5072_0_1275_e07372cebf   has status   DONE\r\nDEBUG: Asking scheduler for work...\r\nDEBUG: Pending tasks: 1\r\nINFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) running   Meetup()\r\nRunning Meetup\r\nINFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) done      Meetup()\r\nDEBUG: 1 running tasks, waiting for next task to finish\r\nINFO: Informed scheduler that task   Meetup__99914b932b   has status   DONE\r\nDEBUG: Asking scheduler for work...\r\nDEBUG: Done\r\nDEBUG: There are no more tasks to run at this time\r\nINFO: Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) was stopped. Shutting down Keep-Alive thread\r\nINFO: \r\n===== Luigi Execution Summary =====\r\n\u00a0\r\nScheduled 3 tasks of which:\r\n* 3 ran successfully:\r\n    - 1 GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275)\r\n    - 1 GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275)\r\n    - 1 Meetup()\r\n\u00a0\r\nThis progress looks :) because there were no failed tasks or missing external dependencies\r\n\u00a0\r\n===== Luigi Execution Summary =====<\/pre>\n
Looks good! Let\u2019s quickly look at our CSV file:<\/p>\n
$ head -n10 \/tmp\/groups.csv \r\nid,name,urlname,link,rating,created,description,organiserName,organiserMemberId\r\n1114381,\"London NoSQL, MySQL, Open Source Community\",\"london-nosql-mysql\",\"https:\/\/www.meetup.com\/london-nosql-mysql\/\",4.28,1208505614000,\"<p>Meet others in London interested in NoSQL, MySQL, and Open Source Databases.<\/p>\",\"Sinead Lawless\",185675230\r\n1561841,\"Enterprise Search London Meetup\",\"es-london\",\"https:\/\/www.meetup.com\/es-london\/\",4.66,1259157419000,\"<p>Enterprise Search London is a meetup for anyone interested in building search and discovery experiences \u2014 from intranet search and site search, to advanced discovery applications and beyond.<\/p>\r\n<p>Disclaimer: This meetup is NOT about SEO or search engine marketing.<\/p>\r\n<p><strong>What people are saying:<\/strong><\/p>\r\n<ul>\r\n<li><span>\"\"Join this meetup if you have a passion for enterprise search and user experience that you would like to share with other able-minded practitioners.\"\" \u2014 Vegard Sandvold<\/span><\/li>\r\n<li><span>\"\"Full marks for vision and execution. Looking forward to the next Meetup.\"\" \u2014 Martin White<\/span><\/li>\r\n<li><span>\u201cConsistently excellent\u201d \u2014 Helen Lippell<\/span><\/li>\r\n<\/ul><\/pre>\n
Sweet! And what if we run it again?<\/p>\n
$ PYTHONPATH=\".\" luigi --module blog --local-scheduler Meetup\r\nDEBUG: Checking if Meetup() is complete\r\nINFO: Informed scheduler that task   Meetup__99914b932b   has status   DONE\r\nINFO: Done scheduling tasks\r\nINFO: Running Worker with 1 processes\r\nDEBUG: Asking scheduler for work...\r\nDEBUG: Done\r\nDEBUG: There are no more tasks to run at this time\r\nINFO: Worker Worker(salt=172768377, workers=1, host=Marks-MBP-4, username=markneedham, pid=4531) was stopped. Shutting down Keep-Alive thread\r\nINFO: \r\n===== Luigi Execution Summary =====\r\n\u00a0\r\nScheduled 1 tasks of which:\r\n* 1 present dependencies were encountered:\r\n    - 1 Meetup()\r\n\u00a0\r\nDid not run any tasks\r\nThis progress looks :) because there were no failed tasks or missing external dependencies\r\n\u00a0\r\n===== Luigi Execution Summary =====<\/pre>\n
As expected nothing happens since our dependencies are already satisfied and we have our first Luigi pipeline up and running.<\/p>\n
\n\n\n\n
Reference: <\/span><\/td>\nLuigi: An ExternalProgramTask example – Converting JSON to CSV<\/a> from our WCG partner<\/a> Mark Needham at the Mark Needham Blog<\/a> blog.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an ExternalProgramTask so this is my attempt at filling that void.                   I\u2019m building a little data pipeline to get data …<\/p>\n","protected":false},"author":48,"featured_media":921,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[53],"tags":[291,40],"class_list":["post-16716","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","tag-csv","tag-json"],"yoast_head":"\nLuigi: An ExternalProgramTask example - Converting JSON to CSV - Web Code Geeks - 2026<\/title>\n<meta name=\"description\" content=\"I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Luigi: An ExternalProgramTask example - Converting JSON to CSV - Web Code Geeks - 2026\" \/>\n<meta property=\"og:description\" content=\"I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\" \/>\n<meta property=\"og:site_name\" content=\"Web Code Geeks\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/webcodegeeks\" \/>\n<meta property=\"article:published_time\" content=\"2017-03-28T09:15:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"150\" \/>\n\t<meta property=\"og:image:height\" content=\"150\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Mark Needham\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@webcodegeeks\" \/>\n<meta name=\"twitter:site\" content=\"@webcodegeeks\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mark Needham\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\"},\"author\":{\"name\":\"Mark Needham\",\"@id\":\"https:\/\/www.webcodegeeks.com\/#\/schema\/person\/848a54e2ee724e46069ce36c2e52e98e\"},\"headline\":\"Luigi: An ExternalProgramTask example – Converting JSON to CSV\",\"datePublished\":\"2017-03-28T09:15:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\"},\"wordCount\":316,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg\",\"keywords\":[\"csv\",\"JSON\"],\"articleSection\":[\"Python\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\",\"url\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\",\"name\":\"Luigi: An ExternalProgramTask example - Converting JSON to CSV - Web Code Geeks - 2026\",\"isPartOf\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg\",\"datePublished\":\"2017-03-28T09:15:32+00:00\",\"description\":\"I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an\",\"breadcrumb\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage\",\"url\":\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg\",\"contentUrl\":\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg\",\"width\":150,\"height\":150},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.webcodegeeks.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Python\",\"item\":\"https:\/\/www.webcodegeeks.com\/category\/python\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Luigi: An ExternalProgramTask example – Converting JSON to CSV\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.webcodegeeks.com\/#website\",\"url\":\"https:\/\/www.webcodegeeks.com\/\",\"name\":\"Web Code Geeks\",\"description\":\"Web Developers Resource Center\",\"publisher\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.webcodegeeks.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.webcodegeeks.com\/#organization\",\"name\":\"Exelixis Media P.C.\",\"url\":\"https:\/\/www.webcodegeeks.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.webcodegeeks.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png\",\"contentUrl\":\"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png\",\"width\":864,\"height\":246,\"caption\":\"Exelixis Media P.C.\"},\"image\":{\"@id\":\"https:\/\/www.webcodegeeks.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/webcodegeeks\",\"https:\/\/x.com\/webcodegeeks\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.webcodegeeks.com\/#\/schema\/person\/848a54e2ee724e46069ce36c2e52e98e\",\"name\":\"Mark Needham\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.webcodegeeks.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5489baed26ce2d932bf951ecfb47afe80bec45d3648c23521d87c83b8f1c3ea9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5489baed26ce2d932bf951ecfb47afe80bec45d3648c23521d87c83b8f1c3ea9?s=96&d=mm&r=g\",\"caption\":\"Mark Needham\"},\"sameAs\":[\"http:\/\/www.markhneedham.com\/blog\/\"],\"url\":\"https:\/\/www.webcodegeeks.com\/author\/mark-needham\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Luigi: An ExternalProgramTask example - Converting JSON to CSV - Web Code Geeks - 2026","description":"I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/","og_locale":"en_US","og_type":"article","og_title":"Luigi: An ExternalProgramTask example - Converting JSON to CSV - Web Code Geeks - 2026","og_description":"I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an","og_url":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/","og_site_name":"Web Code Geeks","article_publisher":"https:\/\/www.facebook.com\/webcodegeeks","article_published_time":"2017-03-28T09:15:32+00:00","og_image":[{"width":150,"height":150,"url":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg","type":"image\/jpeg"}],"author":"Mark Needham","twitter_card":"summary_large_image","twitter_creator":"@webcodegeeks","twitter_site":"@webcodegeeks","twitter_misc":{"Written by":"Mark Needham","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#article","isPartOf":{"@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/"},"author":{"name":"Mark Needham","@id":"https:\/\/www.webcodegeeks.com\/#\/schema\/person\/848a54e2ee724e46069ce36c2e52e98e"},"headline":"Luigi: An ExternalProgramTask example – Converting JSON to CSV","datePublished":"2017-03-28T09:15:32+00:00","mainEntityOfPage":{"@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/"},"wordCount":316,"commentCount":0,"publisher":{"@id":"https:\/\/www.webcodegeeks.com\/#organization"},"image":{"@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage"},"thumbnailUrl":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg","keywords":["csv","JSON"],"articleSection":["Python"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/","url":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/","name":"Luigi: An ExternalProgramTask example - Converting JSON to CSV - Web Code Geeks - 2026","isPartOf":{"@id":"https:\/\/www.webcodegeeks.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage"},"image":{"@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage"},"thumbnailUrl":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg","datePublished":"2017-03-28T09:15:32+00:00","description":"I\u2019ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an","breadcrumb":{"@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#primaryimage","url":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg","contentUrl":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2014\/10\/json-logo.jpg","width":150,"height":150},{"@type":"BreadcrumbList","@id":"https:\/\/www.webcodegeeks.com\/python\/luigi-externalprogramtask-example-converting-json-csv\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.webcodegeeks.com\/"},{"@type":"ListItem","position":2,"name":"Python","item":"https:\/\/www.webcodegeeks.com\/category\/python\/"},{"@type":"ListItem","position":3,"name":"Luigi: An ExternalProgramTask example – Converting JSON to CSV"}]},{"@type":"WebSite","@id":"https:\/\/www.webcodegeeks.com\/#website","url":"https:\/\/www.webcodegeeks.com\/","name":"Web Code Geeks","description":"Web Developers Resource Center","publisher":{"@id":"https:\/\/www.webcodegeeks.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.webcodegeeks.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.webcodegeeks.com\/#organization","name":"Exelixis Media P.C.","url":"https:\/\/www.webcodegeeks.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webcodegeeks.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","contentUrl":"https:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","width":864,"height":246,"caption":"Exelixis Media P.C."},"image":{"@id":"https:\/\/www.webcodegeeks.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/webcodegeeks","https:\/\/x.com\/webcodegeeks"]},{"@type":"Person","@id":"https:\/\/www.webcodegeeks.com\/#\/schema\/person\/848a54e2ee724e46069ce36c2e52e98e","name":"Mark Needham","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webcodegeeks.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5489baed26ce2d932bf951ecfb47afe80bec45d3648c23521d87c83b8f1c3ea9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5489baed26ce2d932bf951ecfb47afe80bec45d3648c23521d87c83b8f1c3ea9?s=96&d=mm&r=g","caption":"Mark Needham"},"sameAs":["http:\/\/www.markhneedham.com\/blog\/"],"url":"https:\/\/www.webcodegeeks.com\/author\/mark-needham\/"}]}},"_links":{"self":[{"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/posts\/16716","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/comments?post=16716"}],"version-history":[{"count":0,"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/posts\/16716\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/media\/921"}],"wp:attachment":[{"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/media?parent=16716"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/categories?post=16716"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webcodegeeks.com\/wp-json\/wp\/v2\/tags?post=16716"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}