Have you run in situations where concurrent execution could speed up\nyour Python code? Are you using a GUI toolkit?<\/p>\n
This talk gives you the background to use concurrency in your code\nwithout shooting yourself in the foot - which is quite easy if you don't\nunderstand how concurrent execution differs from linear execution!<\/p>\n
The presentation starts with explaining some concepts like concurrency,\nparallelism, resources, atomic operations, race conditions and\ndeadlocks.<\/p>\n
Then we discuss the commonly-used approaches to concurrency:\nmultithreading with the threading<\/tt> module, multiprocessing with the\nmultiprocessing<\/tt> module, and event loops (which include the\nasyncio<\/tt> framework). Each of these approaches has its typical use\ncases, which are explained.<\/p>\n
You can implement concurrency on a number of abstraction levels. The\nlowest level consists of primitives like locks, events, semaphores and\nso on. A higher abstraction level is using queues, typically with worker\nthreads or processes. Even higher abstraction levels are active objects\n(hiding primitives or queues behind an API; this includes "actors" if\nyou heard of them), the thread and process pools in\nconcurrent.futures<\/tt> and the asyncio<\/tt> framework. Finally, you can\n"outsource" concurrency by leaving it to a message broker, which is a\ndistinct process that receives and distributes messages.<\/p>\n
The talk closes with some tips and best practices, mainly:<\/p>\n","pubDate":"Fri, 26 Oct 2018 00:00:00 +0000","guid":"tag:pyvideo.org,2018-10-26:\/pycon-de-2018\/concurrency-in-python-concepts-frameworks-and-best-practices.html","category":["PyCon DE 2018","Parallel Programming","Programming","Python"]},{"title":"Strongly typed datasets in a weakly typed world","link":"https:\/\/pyvideo.org\/pycon-de-2018\/strongly-typed-datasets-in-a-weakly-typed-world.html","description":"
We at Blue Yonder use Pandas quite a lot during our daily data science\nand engineering work. This choice, together with Python as an underlying\nprogramming language gives us flexibility, a feature-rich interface, and\naccess to a large community and ecosystem. When it comes to preserving\nthe data and exchanging it with different software stacks, we rely on\nParquet Datasets \/ Hive Tables. During the write process, there is a\nshift from a rather weakly typed world to a strongly typed one. For\nexample, Pandas may convert integers to floats for many operations\nwithout asking, but parquet files and the schema information stored\nalongside them dictate very precise types. The type situation may get\neven more "colorful", when datasets are written by multiple code\nversions or different software solutions over time. This then results in\nimportant questions regarding type compatibility.<\/p>\n
This talk will first represent an overview on types at different layers\n(like NumPy, Pandas, Arrow and Parquet) and the transition between this\nlayers. The second part of the talk will present examples of type\ncompatibility we have seen and why+how we think they should be handled.\nAt the end there will be a Q+A, which can be seen as the start of a\npotentially longer RFC process to align different software stacks (like\nHive and Dask) to handle types in a similar way.<\/p>\n","pubDate":"Fri, 26 Oct 2018 00:00:00 +0000","guid":"tag:pyvideo.org,2018-10-26:\/pycon-de-2018\/strongly-typed-datasets-in-a-weakly-typed-world.html","category":["PyCon DE 2018","Algorithms","Big Data","Data Science","Parallel Programming"]},{"title":"Big Data Systems Performance: The Little Shop of Horrors","link":"https:\/\/pyvideo.org\/pycon-de-2018\/big-data-systems-performance-the-little-shop-of-horrors.html","description":"
The confusion around terms such as like NoSQL, Big Data, Data Science,\nSpark, SQL, and Data Lakes often creates more fog than clarity. However,\nclarity about the underlying technologies is crucial to designing the\nbest technical solution in any field relying on huge amounts of data\nincluding data science, machine learning, but also more traditional\nanalytical systems such as data integration, data warehousing,\nreporting, and OLAP.<\/p>\n
In my presentation, I will show that often at least three dimensions are\ncluttered and confused in discussions when it comes to data management:\nFirst, buzzwords (labels & terms like "big data", "AI", "data lake");\nsecond, data design patterns (principles & best practices like:\nselection push-down, materialization, indexing); and Third, software\nplatforms (concrete implementations & frameworks like: Python, DBMS,\nSpark, and NoSQL-systems).<\/p>\n
Only by keeping these three dimensions apart, it is possible to create\ntechnically-sound architectures in the field of big data analytics.<\/p>\n
I will show concrete examples, which through a simple redesign and wise\nchoice of the right tools and technologies, run thereby up to 1000 times\nfaster. This in turn triggers tremendous savings in terms of development\ntime, hardware costs, and maintenance effort.<\/p>\n","pubDate":"Thu, 25 Oct 2018 00:00:00 +0000","guid":"tag:pyvideo.org,2018-10-25:\/pycon-de-2018\/big-data-systems-performance-the-little-shop-of-horrors.html","category":["PyCon DE 2018","Algorithms","Big Data","Data Science","Infrastructure","Parallel Programming","Programming","Python","Science"]},{"title":"Fulfilling Apache Arrow's Promises: Pandas on JVM memory without a copy","link":"https:\/\/pyvideo.org\/pycon-de-2018\/fulfilling-apache-arrows-promises-pandas-on-jvm-memory-without-a-copy.html","description":"
Apache Arrow established a standard for columnar in-memory analytics to\nredefine the performance and interoperability of most Big Data\ntechnologies in early 2016. Since then implementations in Java, C++,\nPython, Glib, Ruby, Go, JavaScript and Rust have been added. Although\nApache Arrow (pyarrow<\/tt>) is already known to many Python\/Pandas users\nfor reading Apache Parquet files, its main benefit is the cross-language\ninteroperability. With feather and PySpark, you can already benefit from\nthis in Python and R\/Java via the filesystem or network. While they\nimprove data sharing and remove serialization overhead, data still needs\nto be copied as it is passed between processes.<\/p>\n
In the 0.23 release of Pandas, the concept of ExtensionArrays was\nintroduced. They allow the extension of Pandas DataFrames and Series\nwith custom, user- defined typed. The most prominent example is\ncyberpandas<\/tt> which adds an IP dtype that is backed by the appropriate\nrepresentation using NumPy arrays. These ExtensionArrays are not limited\nto arrays backed by NumPy but can take an arbitrary storage as long as\nthey fulfill a certain interfaces. Using Apache Arrow we can implement\nExtensionArrays that are of the same dtype as the built-in types of\nPandas but memory management is not tied to Pandas' internal\nBlockManager. On the other hand Apache Arrow has a much more wider set\nof efficient types that we can also expose as an ExtensionArray. These\ntypes include a native string type as well as a arbitrarily nested types\nsuch as list of \u2026<\/tt> or struct of (\u2026, \u2026, \u2026)<\/tt>.<\/p>\n
To show the real-world benefits of this, we take the example of a data\npipeline that pulls data from a relational store, transforms it and then\npasses it into a machine learning model. A typical setup nowadays most\nlikely involves a data lake that is queried with a JVM based query\nengine. The machine learning model is then normally implemented in\nPython using popular frameworks like CatBoost or Tensorflow.<\/p>\n
While sometimes these query engines provide Python clients, their\nperformance is normally not optimized for large results sets. In the\ncase of a machine learning model, we will do some feature\ntransformations and possibly aggregations with the query engine but feed\nas many rows as possible into the model. This will lead then to result\nsets that have above a million rows. In contrast to the Python clients,\nthese engines often come with efficient JDBC drivers that can cope with\nresult sets of this size but then the conversion from Java objects to\nPython objects in the JVM bridge will slow things down again. In our\nexample, we will show how to use Arrow to retrieve a large result in the\nJVM and then pass it on to Python without running into these\nbottlenecks.<\/p>\n","pubDate":"Thu, 25 Oct 2018 00:00:00 +0000","guid":"tag:pyvideo.org,2018-10-25:\/pycon-de-2018\/fulfilling-apache-arrows-promises-pandas-on-jvm-memory-without-a-copy.html","category":["PyCon DE 2018","Algorithms","Big Data","Data Science","Parallel Programming"]},{"title":"Cython to speed up your Python code","link":"https:\/\/pyvideo.org\/pycon-de-2018\/cython-to-speed-up-your-python-code.html","description":"