{"@attributes":{"version":"2.0"},"channel":{"title":"FireDucks \u2013 FireDucks","link":"https:\/\/fireducks-dev.github.io\/","description":"Recent content on FireDucks","generator":"Hugo -- gohugo.io","language":"en","lastBuildDate":"Fri, 31 Jan 2025 00:00:00 +0000","item":[{"title":"Posts: Unveiling the Optimization Benefit of FireDucks Lazy Execution: Part #3","link":"https:\/\/fireducks-dev.github.io\/posts\/data_flow_optimization\/","pubDate":"Fri, 31 Jan 2025 00:00:00 +0000","guid":"https:\/\/fireducks-dev.github.io\/posts\/data_flow_optimization\/","description":"\n

In the previous article<\/a>, we have talked about how FireDucks lazy-execution can take care of the\ncaching for the intermediate results in order to avoid recomputation of an expensive operation.\nIn today’s article, we will focus on the efficient data flow optimization<\/strong> by its JIT compiler.\nWe will first try to understand some best practices when performing large-scale data analysis in pandas\nand then discuss how those can be automatically taken care by FireDucks lazy execution model.<\/p>\n

Challenge #1<\/h2>\n

Let’s consider the following two queries solving the same problem: Find top 2 “A” based on the “B” column<\/em>.<\/p>\n

\ud83d\udc49 Can you guess which one is better from performance point of view?<\/strong><\/p>\n

(1) version 1<\/p>\n

res =<\/span> df.<\/span>sort_values(by=<\/span>"B"<\/span>)["A"<\/span>].<\/span>head(2<\/span>)\n<\/span><\/span><\/code><\/pre><\/div>
(2) version 2<\/p>\n
tmp =<\/span> df[["A"<\/span>, "B"<\/span>]]\n<\/span><\/span>res =<\/span> tmp.<\/span>sort_values(by=<\/span>"B"<\/span>)["A"<\/span>].<\/span>head(2<\/span>)\n<\/span><\/span><\/code><\/pre><\/div>
Well, when we conducted this quiz in one of the recent Data Science events,\n45% of the participants answered the first one is more efficient, while the remaining 55% answered the second one is more efficient.<\/p>\n
Congratulations, if your answer is (2) as well. \ud83d\udc4f<\/p>\n
In real world situation the target data might have many columns and when we invoked sort operation on df<\/code> instance,\nit performed sorting the entire data involving a significant cost in terms of memory and computational power.<\/p>\n
As depicted in the following diagram, if the data have columns from ‘a’ to ‘j’, when performing the first query,\nit also sorts the column ‘c’ to ‘j’ that is not of our interest. Hence, it is a wise call to create a view of\nthe part of data that is of our interest (as shown in the following figure) before performing an computationally intensive operation\nlike sort, groupby, join etc. At this we can save significant amount of runtime memory and computational time.\nSuch optimization is typically known as projection pushdown<\/code>.\n\"projection<\/p>\n
Challenge #2<\/h2>\n
Let’s now consider another example:<\/p>\n
m =<\/span> employee.<\/span>merge(country, on=<\/span>"C_Code"<\/span>)\n<\/span><\/span>f =<\/span> m[m["Gender"<\/span>] ==<\/span> "Male"<\/span>]\n<\/span><\/span>r =<\/span> f.<\/span>groupby("C_Name"<\/span>)["E_Name"<\/span>].<\/span>count()\n<\/span><\/span><\/code><\/pre><\/div>
The following diagram illustrates the operations that takes place while executing the query above:\n\"country-wise<\/p>\n
\ud83d\udc49 Can you guess the performance bottleneck involved in the above query?<\/strong><\/p>\n
Probably you guessed it correct!!<\/p>\n
The query wants to analyze only the male<\/code> employees.\nThen why to include all the employees at the very first step while joining the two dataframes employee<\/code> and country<\/code>?\nWe could simply filter only the male employees from the employee<\/code> data and\nperform the rest of the operations like merge, groupby etc. on the filtered result as shown below.\nAt this we could save significant execution time and memory during the expensive merge operation.<\/p>\n
f =<\/span> employee[employee["Gender"<\/span>] ==<\/span> "Male"<\/span>]\n<\/span><\/span>m =<\/span> f.<\/span>merge(country, on=<\/span>"C_Code"<\/span>)\n<\/span><\/span>r =<\/span> m.<\/span>groupby("C_Name"<\/span>)["E_Name"<\/span>].<\/span>count()\n<\/span><\/span><\/code><\/pre><\/div>
Such optimization is typically known as predicate pushdown<\/code>.\n\"predicate<\/p>\n
Let’s follow these best practices<\/h2>\n
When dealing with large-scale data, sometime we might not be interested on all part of the data.\nHence, its always the best practice to reduce the scope of your data before applying an\nexpensive operation on it to reduce a significant amount of runtime memory and computational time.<\/p>\n