{"@attributes":{"version":"2.0"},"channel":{"title":"DEV Community: Nikita Sobolev","description":"The latest articles on DEV Community by Nikita Sobolev (@sobolevn).","link":"https:\/\/dev.to\/sobolevn","image":{"url":"https:\/\/media2.dev.to\/dynamic\/image\/width=90,height=90,fit=cover,gravity=auto,format=auto\/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F25139%2F86c1bc8c-87a9-401d-8830-fa73cc01e569.jpg","title":"DEV Community: Nikita Sobolev","link":"https:\/\/dev.to\/sobolevn"},"language":"en","item":[{"title":"Typeclasses in Python","pubDate":"Thu, 08 Jul 2021 14:17:23 +0000","link":"https:\/\/dev.to\/wemake-services\/typeclasses-in-python-3ma6","guid":"https:\/\/dev.to\/wemake-services\/typeclasses-in-python-3ma6","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2021\/06\/typeclasses-in-python\">https:\/\/sobolevn.me\/2021\/06\/typeclasses-in-python<\/a><\/p>\n\n<p>Today I am going to introduce a new concept for Python developers: typeclasses.<br>\nIt is a concept behind our new <code>dry-python<\/code> library called <a href=\"https:\/\/github.com\/dry-python\/classes\/\"><code>classes<\/code><\/a>.<\/p>\n\n<p>I will tell you in advance, that it will look very familiar to what you already know and possibly even use. Moreover, we reuse a lot of existing code from Python's standard library. So, you can call this approach \"native\" and \"pythonic\". And it is still going to be interesting: I am showing examples in 4 different languages!<\/p>\n\n<p>But, before discussing typeclasses themselves, let's discuss what problem they do solve.<\/p>\n<h2>\n  \n  \n  Some functions must behave differently\n<\/h2>\n\n<p>Ok, this one is a familiar problem to all of the devs out there.<br>\nHow can we write a function that will behave differently for different types?<\/p>\n\n<p>Let's create an example. We want to <code>greet<\/code> different types differently (yes, \"hello world\" examples, here we go).<br>\nWe want to <code>greet<\/code>:<\/p>\n\n<ul>\n<li>\n<code>str<\/code> instances as <code>Hello, {string_content}!<\/code>\n<\/li>\n<li>\n<code>MyUser<\/code> instances as <code>Hello again, {username}<\/code>\n<\/li>\n<\/ul>\n\n<p>Note, that <code>greet<\/code> as a simple example does not really make much \"business\" sense, but more complicated things like <code>to_json<\/code>, <code>from_json<\/code>, <code>to_sql<\/code>, <code>from_sql<\/code>, and <code>to_binary<\/code> do make a lot of sense and can be found in almost any project.<br>\nBut, for the sake of implementation simplicity, I'm going to stick to our <code>greet<\/code> example.<\/p>\n\n<p>The first approach that comes to our minds is to use <code>isinstance()<\/code> checks inside the function itself.<br>\nAnd it can work in some cases! The only requirement is that we <strong>must<\/strong> know all the types we will work with in advance.<\/p>\n\n<p>Here's how it would look like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"n\">MyUser<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"s\">'Hello, \"{0}\"!'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">elif<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">,<\/span> <span class=\"n\">MyUser<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"s\">'Hello again, {0}'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">name<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">raise<\/span> <span class=\"nb\">NotImplementedError<\/span><span class=\"p\">(<\/span>\n        <span class=\"s\">'Cannot greet \"{0}\" type'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"nb\">type<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)),<\/span>\n    <span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>The main limitation is that we cannot extend this function for other type easily (we can use wrapper function, but I consiser this a redefinition).<\/p>\n\n<p>But, in some cases - <code>isinstance<\/code> won't be enough, because we need extendability. We need to support other types, which are unknown in advance.<br>\nOur users might need to <code>greet<\/code> their custom types.<\/p>\n\n<p>And that's the part where things begin to get interesting.<\/p>\n\n<p>All programming languages address this problem differently.<br>\nLet's start with Python's traditional OOP approach.<\/p>\n<h2>\n  \n  \n  OOP extendability and over-abstraction problems\n<\/h2>\n\n<p>So, how does Python solve this problem?<\/p>\n\n<p>We all know that Python has magic methods for some builtin functions like <code>len()<\/code> and <code>__len__<\/code>, it solves exactly the same problem.<\/p>\n\n<p>Let's say we want to greet a user:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"s\">'Hello again, {0}'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">name<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>You can use this method directly or you can create a helper with <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0544\/\"><code>typing.Protocol<\/code><\/a>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing_extensions<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Protocol<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">CanGreet<\/span><span class=\"p\">(<\/span><span class=\"n\">Protocol<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"\n        It will match any object that has the ``greet`` method.\n\n        Mypy will also check that ``greet`` must return ``str``.\n        \"\"\"<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">CanGreet<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">greet<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And then we can use it:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"o\">=<\/span><span class=\"s\">'example'<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\"># Hello again, example\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>So, it works? <em>Not really<\/em>.<\/p>\n\n<p>There are several problems.<\/p>\n\n<p><strong>First<\/strong>, some classes do not want to know some details about themselves to maintain abstraction integrity.<br>\nFor example:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">Person<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">become_friends<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">friend<\/span><span class=\"p\">:<\/span> <span class=\"s\">'Person'<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n         <span class=\"p\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">is_friend_of<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">person<\/span><span class=\"p\">:<\/span> <span class=\"s\">'Person'<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"p\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">get_pets<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Sequence<\/span><span class=\"p\">[<\/span><span class=\"s\">'Pet'<\/span><span class=\"p\">]:<\/span>\n        <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Does this <code>Person<\/code> (pun intended) deserve to know that some <code>to_json<\/code> conversion exists that can turn this poor <code>Person<\/code> into textual data? What about binary pickling?<br>\nOf course not, these details should not be added to a business-level abstraction, this is called a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Leaky_abstraction\">leaky abstraction<\/a> when you do otherwise.<\/p>\n\n<p>Moreover, I think that mixing structure and behavior into a single abstraction is bad. Why? Because you cannot tell in advance what behavior you would need from a given structure.<\/p>\n\n<p>For abstractions on this level, it is way easier to have behavior near the structure, not inside it. Mixing these two only makes sense when we work on a higher level like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Service-oriented_architecture\">services<\/a> or <a href=\"https:\/\/en.wikipedia.org\/wiki\/Open_Telecom_Platform\">processes<\/a>.<\/p>\n\n<p><strong>Second<\/strong>, it only works for custom types. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Expression_problem\">Existing types are hard to extend<\/a>.<br>\nFor example, how would you add the <code>greet<\/code> method to the <code>str<\/code> type?<\/p>\n\n<p>You can create <code>str<\/code> subtype with <code>greet<\/code> method in it:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">MyStr<\/span><span class=\"p\">(<\/span><span class=\"nb\">str<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"s\">'Hello, {0}!'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>But, this would require a change in our usage:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">MyStr<\/span><span class=\"p\">(<\/span><span class=\"s\">'world'<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\"># Hello, world!\n<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"s\">'world'<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># fails with TypeError\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  Monkey-patching\n<\/h3>\n\n<p>Some might suggest that we can just insert the needed methods directly into an object \/ type.<br>\nSome dynamically typed languages went on this path: <code>JavaScript<\/code> (in 2000s and early 2010s, mostly popularized by <code>jQuery<\/code> plugins) and <code>Ruby<\/code> (<a href=\"https:\/\/guides.rubyonrails.org\/active_support_core_extensions.html\">still happening right now<\/a>). Here's how it looks:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight javascript\"><code><span class=\"nb\">String<\/span><span class=\"p\">.<\/span><span class=\"nx\">prototype<\/span><span class=\"p\">.<\/span><span class=\"nx\">greet<\/span> <span class=\"o\">=<\/span> <span class=\"kd\">function<\/span> <span class=\"p\">(<\/span><span class=\"nx\">string<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s2\">`Hello, <\/span><span class=\"p\">${<\/span><span class=\"nx\">string<\/span><span class=\"p\">}<\/span><span class=\"s2\">!`<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>It is quite obvious, that it is not going to work for anything complex. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Monkey_patch#Pitfalls\">Why<\/a>?<\/p>\n\n<ul>\n<li>Different parts of your program might use monkey-patching of methods with the same name, but with different functionality. And nothing will work<\/li>\n<li>It is hard to read because the original source does not contain the patched method and the patching location might be hidden deeply in other files<\/li>\n<li>It is hard to type, for example, <code>mypy<\/code> does not support it at all<\/li>\n<li>Python community is not used to this style, it would be rather hard to persuade them to write their code like this (and that's a good thing!)<\/li>\n<\/ul>\n\n<p>I hope that it is clear: we won't fall into this trap. Let's consider another alternative.<\/p>\n\n<h3>\n  \n  \n  Extra abstractions\n<\/h3>\n\n<p>People familiar with things like <code>django-rest-framework<\/code> might recommend to add <a href=\"https:\/\/www.django-rest-framework.org\/api-guide\/serializers\/\">special abstractions<\/a> to <code>greet<\/code> different types:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">abc<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Generic<\/span><span class=\"p\">,<\/span> <span class=\"n\">TypeVar<\/span>\n\n<span class=\"n\">_Wrapped<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_Wrapped'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">BaseGreet<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_Wrapped<\/span><span class=\"p\">]):<\/span>\n    <span class=\"s\">\"\"\"Abstract class of all other \"\"\"<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">wrapped<\/span><span class=\"p\">:<\/span> <span class=\"n\">_Wrapped<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_wrapped<\/span> <span class=\"o\">=<\/span> <span class=\"n\">wrapped<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">abc<\/span><span class=\"p\">.<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">raise<\/span> <span class=\"nb\">NotImplementedError<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">StrGreet<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseGreet<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">]):<\/span>\n    <span class=\"s\">\"\"\"Wrapped instance of built-in type ``str``.\"\"\"<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"s\">'Hello, {0}!'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_wrapped<\/span><span class=\"p\">)<\/span>\n\n<span class=\"c1\"># Our custom type:\n<\/span>\n<span class=\"o\">@<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">MyUserGreet<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseGreet<\/span><span class=\"p\">[<\/span><span class=\"n\">MyUser<\/span><span class=\"p\">]):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"s\">'Hello again, {0}'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_wrapped<\/span><span class=\"p\">.<\/span><span class=\"n\">name<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And we can use it like so:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">MyStrGreet<\/span><span class=\"p\">(<\/span><span class=\"s\">'world'<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\"># Hello, world!\n<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">MyUserGreet<\/span><span class=\"p\">(<\/span><span class=\"n\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"o\">=<\/span><span class=\"s\">'example'<\/span><span class=\"p\">))))<\/span>\n<span class=\"c1\"># Hello again, example\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>But, now we have a different problem: we have a gap between real types and their wrappers. There's no easy way to wrap a type into its wrapper. How can we match them? We have to do it either by hand or use some kind of registry like <code>Dict[type, Type[BaseGreet]]<\/code>.<\/p>\n\n<p>And it is still not enough, there will be runtime errors! In practice, it ends up like <code>&lt;X&gt; is not json-serializable<\/code> as many of us might have seen it with <code>drf<\/code>'s serializers when trying to serialize a custom unregistered type.<\/p>\n\n<h2>\n  \n  \n  Typeclasses and similar concepts\n<\/h2>\n\n<p>Let's look at how functional languages (and <code>Rust<\/code>, people still <a href=\"https:\/\/www.fpcomplete.com\/blog\/2018\/10\/is-rust-functional\/\">argue<\/a> whether it is functional or not) handle this problem.<\/p>\n\n<p>Some common knowledge:<\/p>\n\n<ul>\n<li>All these languages don't have <code>class<\/code> concept as we know it in Python and, of course, there's no subclassing<\/li>\n<li>All the languages below don't have <code>object<\/code>s as we do in Python, they don't mix behavior and structure (however, <code>Elixir<\/code> has Alan Kay's <a href=\"https:\/\/www.quora.com\/What-does-Alan-Kay-think-about-Joe-Armstrong-claiming-that-Erlang-might-be-the-only-object-oriented-language-and-also-his-thesis-supervisor-s-claim-that-Erlang-is-extremely-object-oriented\">real objects<\/a>)<\/li>\n<li>Instead, these languages use <a href=\"https:\/\/en.wikipedia.org\/wiki\/Ad_hoc_polymorphism\">ad-hoc polymorphism<\/a> to make functions behave differently for different types via overloading<\/li>\n<li>And, of course, you don't have to know any of the languages below to understand what is going on<\/li>\n<\/ul>\n\n<h3>\n  \n  \n  Elixir\n<\/h3>\n\n<p>Let's start with one of my favorites.<br>\n<code>Elixir<\/code> has <a href=\"https:\/\/elixir-lang.org\/getting-started\/protocols.html\"><code>Protocol<\/code>s<\/a> to achieve what we want:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight elixir\"><code><span class=\"nv\">@doc<\/span> <span class=\"s2\">\"Our custom protocol\"<\/span>\n<span class=\"k\">defprotocol<\/span> <span class=\"no\">Greet<\/span> <span class=\"k\">do<\/span>\n  <span class=\"c1\"># This is an abstract function,<\/span>\n  <span class=\"c1\"># that will behave differently for each type.<\/span>\n  <span class=\"k\">def<\/span> <span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">end<\/span>\n\n<span class=\"nv\">@doc<\/span> <span class=\"s2\">\"Enhancing built-in type\"<\/span>\n<span class=\"k\">defimpl<\/span> <span class=\"no\">Greet<\/span><span class=\"p\">,<\/span> <span class=\"ss\">for:<\/span> <span class=\"no\">BitString<\/span> <span class=\"k\">do<\/span>\n  <span class=\"k\">def<\/span> <span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">string<\/span><span class=\"p\">),<\/span> <span class=\"k\">do<\/span><span class=\"p\">:<\/span> <span class=\"s2\">\"Hello, <\/span><span class=\"si\">#{<\/span><span class=\"n\">string<\/span><span class=\"si\">}<\/span><span class=\"s2\">!\"<\/span>\n<span class=\"k\">end<\/span>\n\n<span class=\"nv\">@doc<\/span> <span class=\"s2\">\"Custom data type\"<\/span>\n<span class=\"k\">defmodule<\/span> <span class=\"no\">MyUser<\/span> <span class=\"k\">do<\/span>\n  <span class=\"k\">defstruct<\/span> <span class=\"p\">[<\/span><span class=\"ss\">:name<\/span><span class=\"p\">]<\/span>\n<span class=\"k\">end<\/span>\n\n<span class=\"nv\">@doc<\/span> <span class=\"s2\">\"Enhancing our own type\"<\/span>\n<span class=\"k\">defimpl<\/span> <span class=\"no\">Greet<\/span><span class=\"p\">,<\/span> <span class=\"ss\">for:<\/span> <span class=\"no\">MyUser<\/span> <span class=\"k\">do<\/span>\n  <span class=\"k\">def<\/span> <span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">),<\/span> <span class=\"k\">do<\/span><span class=\"p\">:<\/span> <span class=\"s2\">\"Hello again, <\/span><span class=\"si\">#{<\/span><span class=\"n\">user<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"si\">}<\/span><span class=\"s2\">\"<\/span>\n<span class=\"k\">end<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>I am pretty sure that my readers were able to read and understand <code>Elixir<\/code> even if they are not familiar with this language. That's what I call beauty!<\/p>\n\n<p>Usage of the code above:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight elixir\"><code><span class=\"c1\"># Using our `Greet.greet` function with both our data types:<\/span>\n<span class=\"no\">IO<\/span><span class=\"o\">.<\/span><span class=\"n\">puts<\/span><span class=\"p\">(<\/span><span class=\"no\">Greet<\/span><span class=\"o\">.<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"s2\">\"world\"<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Hello, world!<\/span>\n<span class=\"no\">IO<\/span><span class=\"o\">.<\/span><span class=\"n\">puts<\/span><span class=\"p\">(<\/span><span class=\"no\">Greet<\/span><span class=\"o\">.<\/span><span class=\"n\">greet<\/span><span class=\"p\">(%<\/span><span class=\"no\">MyUser<\/span><span class=\"p\">{<\/span><span class=\"ss\">name:<\/span> <span class=\"s2\">\"example\"<\/span><span class=\"p\">}))<\/span>\n<span class=\"c1\"># Hello again, example<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>The thing with <code>Elixir<\/code>'s <code>Protocol<\/code>s is that it is <a href=\"https:\/\/github.com\/elixir-lang\/elixir\/issues\/7541\">not currently possible<\/a> to express that some type does support our <code>Greet.greet<\/code> for <code>Elixir<\/code>'s <a href=\"https:\/\/github.com\/jeremyjh\/dialyxir\">type checker<\/a>.<br>\nBut, this is not a big deal for <code>Elixir<\/code>, which is 100% dynamically typed.<\/p>\n\n<p>Protocols are very widely used, they power lots of the language's features.<br>\nHere are some real-life examples:<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/hexdocs.pm\/elixir\/1.11.0\/Enumerable.html\"><code>Enumerable<\/code><\/a> allows to work with collections: counting elements, finding members, reducing, and slicing<\/li>\n<li>\n<a href=\"https:\/\/hexdocs.pm\/elixir\/1.11.0\/String.Chars.html\"><code>String.Chars<\/code><\/a> is something like <code>__str__<\/code> in Python, it converts structures to human-readable format<\/li>\n<\/ul>\n<h3>\n  \n  \n  Rust\n<\/h3>\n\n<p><code>Rust<\/code> has <a href=\"https:\/\/doc.rust-lang.org\/book\/ch10-02-traits.html\"><code>Trait<\/code>s<\/a>. The concept is pretty similar to <code>Protocol<\/code>s in <code>Elixir<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"c\">\/\/ Our custom trait<\/span>\n<span class=\"k\">trait<\/span> <span class=\"n\">Greet<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">String<\/span><span class=\"p\">;<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"c\">\/\/ Enhancing built-in type<\/span>\n<span class=\"k\">impl<\/span> <span class=\"n\">Greet<\/span> <span class=\"k\">for<\/span> <span class=\"nb\">String<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">String<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">return<\/span> <span class=\"nd\">format!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"Hello, {}!\"<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"c\">\/\/ Defining our own type<\/span>\n<span class=\"k\">struct<\/span> <span class=\"n\">MyUser<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"nb\">String<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"c\">\/\/ Enhancing it<\/span>\n<span class=\"k\">impl<\/span> <span class=\"n\">Greet<\/span> <span class=\"k\">for<\/span> <span class=\"n\">MyUser<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">String<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">return<\/span> <span class=\"nd\">format!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"Hello again, {}\"<\/span><span class=\"p\">,<\/span> <span class=\"k\">self<\/span><span class=\"py\">.name<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And of course, due to <code>Rust<\/code>'s static typing, we can express that some function's argument supports the trait we have just defined:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"c\">\/\/ We can express that `greet` function only accepts types<\/span>\n<span class=\"c\">\/\/ that implement `Greet` trait:<\/span>\n<span class=\"k\">fn<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">dyn<\/span> <span class=\"n\">Greet<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">String<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"nf\">.greet<\/span><span class=\"p\">();<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">pub<\/span> <span class=\"k\">fn<\/span> <span class=\"nf\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"c\">\/\/ Using our `greet` function with both our data types:<\/span>\n    <span class=\"nd\">println!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"{}\"<\/span><span class=\"p\">,<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"s\">\"world\"<\/span><span class=\"nf\">.to_string<\/span><span class=\"p\">()));<\/span>\n    <span class=\"c\">\/\/ Hello, world!<\/span>\n    <span class=\"nd\">println!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"{}\"<\/span><span class=\"p\">,<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">MyUser<\/span> <span class=\"p\">{<\/span> <span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"s\">\"example\"<\/span><span class=\"nf\">.to_string<\/span><span class=\"p\">()<\/span> <span class=\"p\">}));<\/span>\n    <span class=\"c\">\/\/ Hello again, example<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>See? The idea is so similar, that it uses almost the same syntax as <code>Elixir<\/code>.<\/p>\n\n<p>Notable real-life examples of how <code>Rust<\/code> uses its <code>Trait<\/code>s:<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/doc.rust-lang.org\/std\/marker\/trait.Copy.html\"><code>Copy<\/code><\/a> and <a href=\"https:\/\/doc.rust-lang.org\/std\/clone\/trait.Clone.html\"><code>Clone<\/code><\/a> - duplicating objects<\/li>\n<li>\n<a href=\"https:\/\/doc.rust-lang.org\/std\/fmt\/trait.Debug.html\"><code>Debug<\/code><\/a> to show better <code>repr<\/code> of an object, again like <code>__str__<\/code> in Python<\/li>\n<\/ul>\n\n<p>Basically, <code>Trait<\/code>s are the core of this language, it is widely used in cases when you need to define any shared behavior.<\/p>\n\n<h3>\n  \n  \n  Haskell\n<\/h3>\n\n<p><code>Haskell<\/code> has <a href=\"http:\/\/learnyouahaskell.com\/making-our-own-types-and-typeclasses\">typeclasses<\/a> to do almost the same thing.<\/p>\n\n<p>So, what's a typeclass?<br>\nTypeclass is a group of types, all of which satisfy some common contract.<br>\nIt is also a form of ad-hoc polymorphism that is mostly used for overloading.<\/p>\n\n<p>I am a bit sorry for the <code>Haskell<\/code> syntax below, it might be not very pleasant and clear to read, especially for people who are not familiar with this brilliant language, but we have what we have:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight haskell\"><code><span class=\"cp\">{-# LANGUAGE FlexibleInstances #-}<\/span>\n\n<span class=\"c1\">-- Our custom typeclass<\/span>\n<span class=\"kr\">class<\/span> <span class=\"kt\">Greet<\/span> <span class=\"kr\">instance<\/span> <span class=\"kr\">where<\/span>\n  <span class=\"n\">greet<\/span> <span class=\"o\">::<\/span> <span class=\"kr\">instance<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"kt\">String<\/span>\n\n<span class=\"c1\">-- Enhancing built-in type with it<\/span>\n<span class=\"kr\">instance<\/span> <span class=\"kt\">Greet<\/span> <span class=\"kt\">String<\/span> <span class=\"kr\">where<\/span>\n  <span class=\"n\">greet<\/span> <span class=\"n\">str<\/span> <span class=\"o\">=<\/span> <span class=\"s\">\"Hello, \"<\/span> <span class=\"o\">++<\/span> <span class=\"n\">str<\/span> <span class=\"o\">++<\/span> <span class=\"s\">\"!\"<\/span>\n\n<span class=\"c1\">-- Defining our own type<\/span>\n<span class=\"kr\">data<\/span> <span class=\"kt\">MyUser<\/span> <span class=\"o\">=<\/span> <span class=\"kt\">MyUser<\/span> <span class=\"p\">{<\/span> <span class=\"n\">name<\/span> <span class=\"o\">::<\/span> <span class=\"kt\">String<\/span> <span class=\"p\">}<\/span>\n\n<span class=\"c1\">-- Enhancing it<\/span>\n<span class=\"kr\">instance<\/span> <span class=\"kt\">Greet<\/span> <span class=\"kt\">MyUser<\/span> <span class=\"kr\">where<\/span>\n  <span class=\"n\">greet<\/span> <span class=\"n\">user<\/span> <span class=\"o\">=<\/span> <span class=\"s\">\"Hello again, \"<\/span> <span class=\"o\">++<\/span> <span class=\"p\">(<\/span><span class=\"n\">name<\/span> <span class=\"n\">user<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Basically, we do the same thing as we have already done for <code>Rust<\/code> and <code>Elixir<\/code>:<\/p>\n\n<ol>\n<li>We define a <code>Greet<\/code> typeclass that has a single function to implement: <code>greet<\/code>\n<\/li>\n<li>Then we define instance implementation for <code>String<\/code> type, which is a built-in (alias for <code>[Char]<\/code>)<\/li>\n<li>Then we define custom <code>MyUser<\/code> type with <code>name<\/code> field of <code>String<\/code> type<\/li>\n<li>Implementing the <code>Greet<\/code> typeclass for <code>MyUser<\/code> is the last thing we do<\/li>\n<\/ol>\n\n<p>Then we can use our new <code>greet<\/code> function:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight haskell\"><code><span class=\"c1\">-- Here you can see that we can use `Greet` typeclass to annotate our types.<\/span>\n<span class=\"c1\">-- I have made this alias entirely for this annotation demo,<\/span>\n<span class=\"c1\">-- in real life we would just use `greet` directly:<\/span>\n<span class=\"n\">greetAlias<\/span> <span class=\"o\">::<\/span> <span class=\"kt\">Greet<\/span> <span class=\"kr\">instance<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"kr\">instance<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"kt\">String<\/span>\n<span class=\"n\">greetAlias<\/span> <span class=\"o\">=<\/span> <span class=\"n\">greet<\/span>\n\n<span class=\"n\">main<\/span> <span class=\"o\">=<\/span> <span class=\"kr\">do<\/span>\n  <span class=\"n\">print<\/span> <span class=\"o\">$<\/span> <span class=\"n\">greetAlias<\/span> <span class=\"s\">\"world\"<\/span>\n  <span class=\"c1\">-- Hello, world!<\/span>\n  <span class=\"n\">print<\/span> <span class=\"o\">$<\/span> <span class=\"n\">greetAlias<\/span> <span class=\"kt\">MyUser<\/span> <span class=\"p\">{<\/span> <span class=\"n\">name<\/span><span class=\"o\">=<\/span><span class=\"s\">\"example\"<\/span> <span class=\"p\">}<\/span>\n  <span class=\"c1\">-- Hello again, example<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Some real-life examples of typeclasses:<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/hackage.haskell.org\/package\/base-4.15.0.0\/docs\/Text-Show.html#t:Show\"><code>Show<\/code><\/a> to convert things into user-readable representations<\/li>\n<li>\n<a href=\"https:\/\/wiki.haskell.org\/Functor\"><code>Functor<\/code><\/a>, <a href=\"https:\/\/hackage.haskell.org\/package\/base-4.10.1.0\/docs\/Control-Applicative.html#t:Applicative\"><code>Applicate<\/code><\/a>, and <a href=\"https:\/\/wiki.haskell.org\/Monad\"><code>Monad<\/code><\/a> are all typeclasses<\/li>\n<\/ul>\n\n<p>I would say that among our three examples, <code>Haskell<\/code> relies on its typeclasses the heaviest.<\/p>\n\n<p>It is important to note that typeclasses from <code>Haskell<\/code> and traits from <code>Rust<\/code> <a href=\"https:\/\/stackoverflow.com\/questions\/28123453\/what-is-the-difference-between-traits-in-rust-and-typeclasses-in-haskell\">are a bit different<\/a>, but we won't go into these details to keep this article rather short.<\/p>\n\n<p>But, what about Python?<\/p>\n\n<h2>\n  \n  \n  dry-python\/classes\n<\/h2>\n\n<p>There's an awesome function in the Python standard library called <a href=\"https:\/\/docs.python.org\/3\/library\/functools.html#functools.singledispatch\"><code>singledispatch<\/code><\/a>.<\/p>\n\n<p>It does exactly what we need. Do you still remember that we are finding a way to change the function's behavior based on the input type?<\/p>\n\n<p>Let's have a look!<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">functools<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">singledispatch<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">singledispatch<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"s\">\"\"\"Default case.\"\"\"<\/span>\n    <span class=\"k\">raise<\/span> <span class=\"nb\">NotImplementedError<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">register<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_str<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">'Hello, {0}!'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span>\n\n<span class=\"c1\"># Custom type\n<\/span>\n<span class=\"o\">@<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">register<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_myuser<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">MyUser<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">'Hello again, {0}'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">name<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Looks cool, moreover, it is in standard lib, you even don't have to install anything!<\/p>\n\n<p>And we can use it like a normal function:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"s\">'world'<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Hello, world!\n<\/span><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">MyUser<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"o\">=<\/span><span class=\"s\">'example'<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\"># Hello again, example\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>So, what's the point in writing a completely different library like we did with <code>dry-python\/classes<\/code>?<\/p>\n\n<p>We even reuse some parts of <code>singledispatch<\/code> implementation,<br>\nbut there are several key differences.<\/p>\n<h3>\n  \n  \n  Better typing\n<\/h3>\n\n<p>With <code>singledispatch<\/code> you cannot be sure that everything will work, because it is not supported by <code>mypy<\/code>.<\/p>\n\n<p>For example, you can pass unsupported types:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># mypy is ok with that :(\n# runtime will raise `NotImplementedError`\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>In <code>dry-python\/classes<\/code> we have fixed that.<br>\nYou can only pass types that are supported:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">classes<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">typeclass<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">typeclass<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">instance<\/span><span class=\"p\">(<\/span><span class=\"nb\">str<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_str<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">'Iterable!'<\/span>\n\n<span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n<span class=\"c1\"># Argument 1 to \"greet\" has incompatible type \"int\"; expected \"str\"\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Or you can break the <code>@singledispatch<\/code> signature contract:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">register<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_dict<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">,<\/span> <span class=\"n\">key<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"p\">[<\/span><span class=\"n\">key<\/span><span class=\"p\">]<\/span>  <span class=\"c1\"># still no mypy error\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>But, not with <code>dry-python\/classes<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">instance<\/span><span class=\"p\">(<\/span><span class=\"nb\">dict<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_dict<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">,<\/span> <span class=\"n\">key<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n<span class=\"c1\"># Instance callback is incompatible\n# \"def (instance: builtins.dict[Any, Any], key: builtins.str) -&gt; builtins.int\";\n# expected\n# \"def (instance: builtins.dict[Any, Any]) -&gt; builtins.str\"\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p><code>@singledispatch<\/code> also does not allow defining generic functions:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">singledispatch<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">copy<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">X<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">X<\/span><span class=\"p\">:<\/span>\n    <span class=\"s\">\"\"\"Default case.\"\"\"<\/span>\n    <span class=\"k\">raise<\/span> <span class=\"nb\">NotImplementedError<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">copy<\/span><span class=\"p\">.<\/span><span class=\"n\">register<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_copy_int<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span>\n<span class=\"c1\"># Argument 1 to \"register\" of \"_SingleDispatchCallable\"\n# has incompatible type \"Callable[[int], int]\";\n# expected \"Callable[..., X]\"\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">copy<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Revealed type is \"X`-1\"\n# Should be: `int`\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Which is, again, possible with <code>dry-python\/classes<\/code>, we fully support <a href=\"https:\/\/classes.readthedocs.io\/en\/latest\/pages\/generics.html\">generic functions<\/a>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">TypeVar<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">classes<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">typeclass<\/span>\n\n<span class=\"n\">X<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'X'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">typeclass<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">copy<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">X<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">X<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">copy<\/span><span class=\"p\">.<\/span><span class=\"n\">instance<\/span><span class=\"p\">(<\/span><span class=\"nb\">int<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_copy_int<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>  <span class=\"c1\"># ok\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">copy<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">))<\/span>  <span class=\"c1\"># int\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And you cannot <a href=\"https:\/\/classes.readthedocs.io\/en\/latest\/pages\/concept.html#type-restrictions\">restrict<\/a> <code>@singledispatch<\/code> to work with only subtypes of specific types, even if you want to.<\/p>\n\n<h3>\n  \n  \n  Protocols are unsupported\n<\/h3>\n\n<p>Protocols are an important part of Python. Sadly, they are not supported by <code>@singledispatch<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">register<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_iterable<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Iterable<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">'Iterable!'<\/span>\n<span class=\"c1\"># TypeError: Invalid annotation for 'instance'.\n# typing.Iterable is not a class\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p><a href=\"https:\/\/classes.readthedocs.io\/en\/latest\/pages\/concept.html#protocols\">Protocols<\/a> support is also solved with <code>dry-python\/classes<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Iterable<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">classes<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">typeclass<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">typeclass<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">instance<\/span><span class=\"p\">(<\/span><span class=\"n\">Iterable<\/span><span class=\"p\">,<\/span> <span class=\"n\">is_protocol<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_str<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Iterable<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">'Iterable!'<\/span>\n\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">]))<\/span>\n<span class=\"c1\"># Iterable!\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  No way to annotate types\n<\/h3>\n\n<p>Let's say you want to write a function and annotate one of its arguments that it must support the <code>greet<\/code> function. Something like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">greet_and_print<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"s\">'???'<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>It is impossible with <code>@singledispatch<\/code>.<br>\nBut, you can do it with <code>dry-python\/classes<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">classes<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">AssociatedType<\/span><span class=\"p\">,<\/span> <span class=\"n\">Supports<\/span><span class=\"p\">,<\/span> <span class=\"n\">typeclass<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Greet<\/span><span class=\"p\">(<\/span><span class=\"n\">AssociatedType<\/span><span class=\"p\">):<\/span>\n    <span class=\"s\">\"\"\"Special type to represent that some instance can `greet`.\"\"\"<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">typeclass<\/span><span class=\"p\">(<\/span><span class=\"n\">Greet<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"s\">\"\"\"No implementation needed.\"\"\"<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">greet<\/span><span class=\"p\">.<\/span><span class=\"n\">instance<\/span><span class=\"p\">(<\/span><span class=\"nb\">str<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_greet_str<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">'Hello, {0}!'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">greet_and_print<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Supports<\/span><span class=\"p\">[<\/span><span class=\"n\">Greet<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">greet<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">))<\/span>\n\n<span class=\"n\">greet_and_print<\/span><span class=\"p\">(<\/span><span class=\"s\">'world'<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># ok\n<\/span><span class=\"n\">greet_and_print<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># type error with mypy, exception in runtime\n# Argument 1 to \"greet_and_print\" has incompatible type \"int\";\n# expected \"Supports[Greet]\"\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>We have come a long way, from basic stacked <code>isinstance()<\/code> conditions - through OOP - to typeclasses.<\/p>\n\n<p>I have shown, that this native and pythonic idea deserves wider recognition and usage. And our extra features in <code>dry-python\/classes<\/code> can save you from lots of mistakes and help to write more expressive and safe business logic.<\/p>\n\n<p>As a result of using typeclasses, you will untangle your structures from behavior, which will allow you to get rid of useless and complex abstractions and write dead-simple typesafe code. You will have your behavior near the structures, not inside them. This will also solve the extendability problem of OOP.<\/p>\n\n<p>Combine it with other <code>dry-python<\/code> libraries for extra effect!<\/p>\n\n<h2>\n  \n  \n  Future work\n<\/h2>\n\n<p>What do we plan for the future?<\/p>\n\n<p>There are several key aspects to improve:<\/p>\n\n<ol>\n<li>Our <code>Supports<\/code> should take any amount of type arguments: <code>Supports[A, B, C]<\/code>. This type will represent a type that supports all three typeclasses <code>A<\/code>, <code>B<\/code>, and <code>C<\/code> <a href=\"https:\/\/github.com\/dry-python\/classes\/issues\/206\">at the same time<\/a>\n<\/li>\n<li>We don't <a href=\"https:\/\/github.com\/dry-python\/classes\/issues\/24\">support concrete generics<\/a> just yet. So, for example, it is impossible to define different cases for <code>List[int]<\/code> and <code>List[str]<\/code>. This might require adding runtime typecheker to <code>dry-python\/classes<\/code>\n<\/li>\n<li>I am planning <a href=\"https:\/\/sobolevn.me\/2021\/02\/make-tests-a-part-of-your-app\">to make tests a part of this app<\/a> as well! We will ship a <a href=\"https:\/\/github.com\/dry-python\/classes\/issues\/234\">hypothesis plugin<\/a> to test users' typeclasses in a single line of code<\/li>\n<\/ol>\n\n<p>Stay tuned!<\/p>\n\n<p>If you like this article you can:<\/p>\n\n<ol>\n<li>Donate to future <code>dry-python<\/code> development on <a href=\"https:\/\/github.com\/sponsors\/dry-python\">GitHub<\/a>\n<\/li>\n<li><a href=\"https:\/\/github.com\/dry-python\/classes\/stargazers\">Star our <code>classes<\/code> repo<\/a><\/li>\n<li>\n<a href=\"https:\/\/sobolevn.me\/subscribe\/\">Subscribe<\/a> to my blog for more content!<\/li>\n<\/ol>\n\n","category":["python","rust","elixir","haskell"]},{"title":"Make tests a part of your app","pubDate":"Wed, 10 Mar 2021 09:30:05 +0000","link":"https:\/\/dev.to\/sobolevn\/make-tests-a-part-of-your-app-8nm","guid":"https:\/\/dev.to\/sobolevn\/make-tests-a-part-of-your-app-8nm","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2021\/02\/make-tests-a-part-of-your-app\">https:\/\/sobolevn.me\/2021\/02\/make-tests-a-part-of-your-app<\/a><\/p>\n\n<p>Today I am going to discuss quite a new idea for Python users, an idea of making tests a valuable part of your application.<\/p>\n\n<p>Let's jump into it.<\/p>\n\n<h2>\n  \n  \n  Current status\n<\/h2>\n\n<p>Right now the status-quo for source code\/tests dualism is that you ship source code to your library users and most often do not include your tests in any manner.<\/p>\n\n<p>Sometimes people also attach the <code>tests\/<\/code> folder to a release, so they are just laying around just in case. Most of the time they are useless to the end-user.<\/p>\n\n<p>And what is the most important part of it all is that our users are often find themselves in a situation when they have to reimplement some tests for library-specific things.<\/p>\n\n<p>Let me give you an example: you have a Django view for authorized users only.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">django.contrib.auth.decorators<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">login_required<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">,<\/span> <span class=\"n\">HttpResponse<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">login_required<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">:<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">HttpRespose<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>So, in our tests we would have to write at least two tests:<\/p>\n\n<ol>\n<li>For the successful auth case and our business logic<\/li>\n<li>For the failed auth case<\/li>\n<\/ol>\n\n<p>Wouldn't it be amazing if we could just skip the second one and rely on some existing test-logic that we can re-use from the library itself?<\/p>\n\n<p>Imagine an API like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># tests\/test_views\/test_my_view.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"nn\">myapp.views<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">my_view<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">test_authed_successfully<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">):<\/span>\n    <span class=\"s\">\"\"\"Test case for our own logic.\"\"\"<\/span>\n\n<span class=\"c1\"># Not authed case:\n<\/span><span class=\"n\">my_view<\/span><span class=\"p\">.<\/span><span class=\"n\">test_not_authed<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And then - boom - we have our second use-case tested with just a single line of code!<\/p>\n\n<p>And it goes further than this. For example, in Django you can stack function <a href=\"https:\/\/docs.djangoproject.com\/en\/3.1\/topics\/http\/decorators\/\">decorators<\/a> to do multiple things. Imagine this situation:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">django.views.decorators.cache<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">never_cache<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.contrib.auth.decorators<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">login_required<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.views.decorators.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">require_http_methods<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">require_http_methods<\/span><span class=\"p\">([<\/span><span class=\"s\">'GET'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'POST'<\/span><span class=\"p\">])<\/span>\n<span class=\"o\">@<\/span><span class=\"n\">login_required<\/span>\n<span class=\"o\">@<\/span><span class=\"n\">never_cache<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">:<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">HttpRespose<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>So, the API might be a little more magical to include a test for all the possible cases:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># tests\/test_views\/test_my_view.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"nn\">myapp.views<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">my_view<\/span>\n\n<span class=\"n\">my_view<\/span><span class=\"p\">.<\/span><span class=\"n\">run_tests<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And it can potentially execute:<\/p>\n\n<ol>\n<li>Tests for http methods that are not allowed<\/li>\n<li>Tests for http methods that are allowed<\/li>\n<li>Test that <code>Cache-Control<\/code> header is there and has a correct value<\/li>\n<li>Test that unauthorized users are not allowed<\/li>\n<li>And possibly others!<\/li>\n<\/ol>\n\n<p>All you have to do is testing your green path with a possibility to customize some particular generated test cases if you have some specifics, like returning a custom http code for unauthorized users.<\/p>\n\n<p>The bad part of this chapter is that the discussed API does not exist. And probably won't ever exist in Django.<\/p>\n\n<p>But, there are other less-known projects (but ones that I help to maintain) that already have these features. Let's see what you can do with them!<\/p>\n\n<h2>\n  \n  \n  deal\n<\/h2>\n\n<p><a href=\"https:\/\/github.com\/life4\/deal\"><code>deal<\/code><\/a> is a library for <a href=\"https:\/\/en.wikipedia.org\/wiki\/Design_by_contract\">Design-by-Contract<\/a>.<\/p>\n\n<p>In other words, it allows decorating your functions and classes with some extra checks that are not representable by types (at least in Python-land).<\/p>\n\n<p>Let's say you have a function to divide two positive integers (which are just <code>int<\/code> in Python):<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">deal<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">deal<\/span><span class=\"p\">.<\/span><span class=\"n\">pre<\/span><span class=\"p\">(<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">a<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">:<\/span> <span class=\"n\">a<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"mi\">0<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">b<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n<span class=\"o\">@<\/span><span class=\"n\">deal<\/span><span class=\"p\">.<\/span><span class=\"n\">raises<\/span><span class=\"p\">(<\/span><span class=\"nb\">ZeroDivisionError<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># this function can raise if `b=0`, it is ok\n<\/span><span class=\"k\">def<\/span> <span class=\"nf\">div<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">a<\/span> <span class=\"o\">\/<\/span> <span class=\"n\">b<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>It has all the contract information in the function's definition:<\/p>\n\n<ul>\n<li>\n<code>@deal.pre(lambda a, b: a &gt;= 0 and b &gt;= 0)<\/code> checks that passed arguments are positive<\/li>\n<li>\n<code>@deal.raises(ZeroDivisionError)<\/code> allows this function to explicitly raise <code>ZeroDivisionError<\/code> without breaking the contract, by default functions cannot raise any exceptions<\/li>\n<\/ul>\n\n<p>Note: type annotations like in <code>(a: int, b: int) -&gt; float<\/code> are not enforced, you should use <code>mypy<\/code> to catch typing errors.<\/p>\n\n<p>Usage (remember, it is still just a function!):<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">div<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># ok\n<\/span><span class=\"n\">div<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># ok, runtime ZeroDivisionError\n<\/span>\n<span class=\"n\">div<\/span><span class=\"p\">(<\/span><span class=\"o\">-<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># not ok\n# deal.PreContractError: expected a &gt;= 0 and b &gt;= 0 (where a=-1, b=1)\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Ok, the simple use-case is clear. Now, let's put a bug in this function on purpose:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">deal<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">deal<\/span><span class=\"p\">.<\/span><span class=\"n\">pre<\/span><span class=\"p\">(<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">a<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">:<\/span> <span class=\"n\">a<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"mi\">0<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">b<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n<span class=\"o\">@<\/span><span class=\"n\">deal<\/span><span class=\"p\">.<\/span><span class=\"n\">raises<\/span><span class=\"p\">(<\/span><span class=\"nb\">ZeroDivisionError<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># this function can raise if `b=0`, it is ok\n<\/span><span class=\"k\">def<\/span> <span class=\"nf\">div<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">a<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">50<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># Custom, in real life this would be a bug in our logic:\n<\/span>        <span class=\"k\">raise<\/span> <span class=\"nb\">Exception<\/span><span class=\"p\">(<\/span><span class=\"s\">'Oh no! Bug happened!'<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">a<\/span> <span class=\"o\">\/<\/span> <span class=\"n\">b<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Luckily, <code>deal<\/code> follows the core idea of this article and ships tests with itself. To run them all we need to do is write just a single test case:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">deal<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">my_lib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">div<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">deal<\/span><span class=\"p\">.<\/span><span class=\"n\">cases<\/span><span class=\"p\">(<\/span><span class=\"n\">div<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># That's all we have to do to test deal-based functions!\n<\/span><span class=\"k\">def<\/span> <span class=\"nf\">test_div<\/span><span class=\"p\">(<\/span><span class=\"n\">case<\/span><span class=\"p\">:<\/span> <span class=\"n\">deal<\/span><span class=\"p\">.<\/span><span class=\"n\">TestCase<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">case<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Here's what the output would be like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight plaintext\"><code>\u00bb pytest test_deal.py\n============================= test session starts ==============================\ncollected 1 item\n\ntest_deal.py F                                                            [100%]\n\n=================================== FAILURES ===================================\n___________________________________ test_div ___________________________________\n\na = 51, b = 0\n\n    @deal.raises(ZeroDivisionError)\n    @deal.pre(lambda a, b: a &gt;= 0 and b &gt;= 0)\n    def div(a: int, b: int) -&gt; float:\n        if a &gt; 50:\n&gt;           raise Exception('Oh no! Bug happened!')\nE           Exception: Oh no! Bug happened!\n\ntest_deal.py:8: Exception\n============================== 1 failed in 0.35s ===============================\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>As you can see our tests did find the bug! But how?<\/p>\n\n<p>There are a lot of questions to ask:<\/p>\n\n<ul>\n<li>Where did the data for the test come from?\nIt comes from another awesome library called <a href=\"https:\/\/github.com\/HypothesisWorks\/hypothesis\"><code>hypothesis<\/code><\/a>. It smartly generates lots of different test data according to some specific rules we define.<\/li>\n<\/ul>\n\n<p>In our case, we have two rules. First rule generate two <code>int<\/code> arguments as defined in <code>def div(a: int, b: int)<\/code>. The second rule is that these integers must be <code>&gt;= 0<\/code> as defined in <code>@deal.pre(lambda a, b: a &gt;= 0 and b &gt;= 0)<\/code>.<\/p>\n\n<p>We can control how many examples would be generated and do other small tweaks.<br>\n  More about it is in the <a href=\"https:\/\/deal.readthedocs.io\/basic\/tests.html\">docs<\/a>.<\/p>\n\n<ul>\n<li><p>Why <code>ZeroDivisionError<\/code> didn't break the test while raw <code>Exception<\/code> did?<br>\nBecause that's how contracts work: you clearly define all possible cases. If something strange happens - the contract is violated. In our example, <code>ZeroDivisionError<\/code> is a part of the contract via <code>deal.raises<\/code> decorator. So, we know that it can (and will) happen. That's why we don't treat it as a test failure, while raw <code>Exception<\/code> is not a part of our contract and we treat it as a failure.<\/p><\/li>\n<li><p>Will it find all bugs in my code?<br>\nThat's the most interesting question. And the answer is <strong>no<\/strong>. Sad, but true.<br>\nThere are endless use-cases, logic, combitations, and bugs in them. And I know for sure that it is impossible to catch all bugs your app has.<\/p><\/li>\n<\/ul>\n\n<p>But, in reality, it will catch <strong>a lot<\/strong> of bugs. In my opinion, it is still worth it.<\/p>\n\n<p>We can even go one step further and represent our contracts as <a href=\"https:\/\/github.com\/Z3Prover\/z3\">Theorems to be proved<\/a>.<br>\nFor example, <code>deal<\/code> has an ongoing research companion project - <a href=\"https:\/\/github.com\/orsinium-labs\/deal-solver\"><code>deal-solver<\/code><\/a> - that can help with that. But, this is a subject for another article of its own, so let's move on for now.<\/p>\n<h2>\n  \n  \n  dry-python\/returns\n<\/h2>\n\n<p><a href=\"https:\/\/github.com\/dry-python\/returns\"><code>dry-python\/returns<\/code><\/a> is a library with primitives that make typed functional programming in Python easier.<\/p>\n\n<p>Inside we have a bunch of interfaces that our users can extend for their own primitives\/objects. In the recent article about <a href=\"https:\/\/sobolevn.me\/2020\/10\/higher-kinded-types-in-python\">Higher Kinded Types<\/a> I have shown how this can be done in a type-safe way.<\/p>\n\n<p>Now, I am going to show that it is not enough on its own. And most likely you will need some extra laws on how your objects should behave.<\/p>\n\n<p>We call this feature \"Monad laws as values\".<\/p>\n<h3>\n  \n  \n  Identity laws\n<\/h3>\n\n<p>Let's start from the easiest Higher Kinded Interface we have: <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/master\/returns\/interfaces\/equable.py\"><code>Equable<\/code><\/a>. It is an interface that allows type-safe equality checks. Because you can use <code>==<\/code> for everything in Python. But, our <code>.equals()<\/code> method will allow us to only check the object of the same type which has real values inside.<\/p>\n\n<p>For example:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">returns.io<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">IO<\/span>\n\n<span class=\"n\">IO<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>  <span class=\"c1\"># type-checks, but pointless, always false\n<\/span>\n<span class=\"n\">IO<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">).<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># does not type-check at all\n# error: Argument 1 has incompatible type \"int\";\n# expected \"KindN[IO[Any], Any, Any, Any]\"\n<\/span>\n<span class=\"n\">other<\/span><span class=\"p\">:<\/span> <span class=\"n\">IO<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]<\/span>\n<span class=\"n\">IO<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">).<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">other<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># ok, might be true or false\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Here's how it looks like at the moment:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">_EqualType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_EqualType'<\/span><span class=\"p\">,<\/span> <span class=\"n\">bound<\/span><span class=\"o\">=<\/span><span class=\"s\">'Equable'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Equable<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"o\">@<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">equals<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span> <span class=\"n\">other<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"Type-safe equality check for values of the same type.\"\"\"<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's say we want to create a bad implementation for this interface (because of science):<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">returns.interfaces.equable<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Equable<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Example<\/span><span class=\"p\">(<\/span><span class=\"n\">Equable<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">inner_value<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_inner_value<\/span> <span class=\"o\">=<\/span> <span class=\"n\">inner_value<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">equals<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">other<\/span><span class=\"p\">:<\/span> <span class=\"s\">'Example'<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"bp\">False<\/span>  <span class=\"c1\"># it breaks how `.equals` is supposed to be used!\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>It is clearly wrong because it always returns <code>False<\/code> without actually checking the <code>inner_value<\/code> of an object. But, it still satisfies the interface definition: it will type-check. That's how we can tell that just the interface is not enough. We need to test the implementation as well.<\/p>\n\n<p>But, equality has known laws from math to catch cases like this:<\/p>\n\n<ul>\n<li>Reflexive law: a value must be equal to itself<\/li>\n<li>Symmetry law: <code>a.equals(b) == b.equals(a)<\/code>\n<\/li>\n<li>Transitivity law: if <code>a<\/code> equals <code>b<\/code> and <code>b<\/code> equals <code>c<\/code>, then <code>a<\/code> equals <code>c<\/code>\n<\/li>\n<\/ul>\n\n<p>We can create a test that will ensure that our implementation holds these laws. Or we might forget about it. Or make a mistake in our test logic.<\/p>\n\n<p>That's why it is important for library authors to think about their users and ship tests with their apps.<\/p>\n\n<p>For example, we encode laws into the interface definition itself:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">abc<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">abstractmethod<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">ClassVar<\/span><span class=\"p\">,<\/span> <span class=\"n\">Sequence<\/span><span class=\"p\">,<\/span> <span class=\"n\">TypeVar<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing_extensions<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">final<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.primitives.laws<\/span> <span class=\"kn\">import<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">Law<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">Law1<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">Law2<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">Law3<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">Lawful<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">LawSpecDef<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">law_definition<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span>\n\n<span class=\"n\">_EqualType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_EqualType'<\/span><span class=\"p\">,<\/span> <span class=\"n\">bound<\/span><span class=\"o\">=<\/span><span class=\"s\">'Equable'<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"o\">@<\/span><span class=\"n\">final<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">_LawSpec<\/span><span class=\"p\">(<\/span><span class=\"n\">LawSpecDef<\/span><span class=\"p\">):<\/span>  <span class=\"c1\"># LOOKATME: our laws def!\n<\/span>    <span class=\"o\">@<\/span><span class=\"n\">law_definition<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">reflexive_law<\/span><span class=\"p\">(<\/span>\n        <span class=\"n\">first<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"Value should be equal to itself.\"\"\"<\/span>\n        <span class=\"k\">assert<\/span> <span class=\"n\">first<\/span><span class=\"p\">.<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">first<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">law_definition<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">symmetry_law<\/span><span class=\"p\">(<\/span>\n        <span class=\"n\">first<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">second<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"If ``A == B`` then ``B == A``.\"\"\"<\/span>\n        <span class=\"k\">assert<\/span> <span class=\"n\">first<\/span><span class=\"p\">.<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">second<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"n\">second<\/span><span class=\"p\">.<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">first<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">law_definition<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">transitivity_law<\/span><span class=\"p\">(<\/span>\n        <span class=\"n\">first<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">second<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">third<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"If ``A == B`` and ``B == C`` then ``A == C``.\"\"\"<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">first<\/span><span class=\"p\">.<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">second<\/span><span class=\"p\">)<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">second<\/span><span class=\"p\">.<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">third<\/span><span class=\"p\">):<\/span>\n            <span class=\"k\">assert<\/span> <span class=\"n\">first<\/span><span class=\"p\">.<\/span><span class=\"n\">equals<\/span><span class=\"p\">(<\/span><span class=\"n\">third<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Equable<\/span><span class=\"p\">(<\/span><span class=\"n\">Lawful<\/span><span class=\"p\">[<\/span><span class=\"s\">'Equable'<\/span><span class=\"p\">]):<\/span>\n    <span class=\"n\">_laws<\/span><span class=\"p\">:<\/span> <span class=\"n\">ClassVar<\/span><span class=\"p\">[<\/span><span class=\"n\">Sequence<\/span><span class=\"p\">[<\/span><span class=\"n\">Law<\/span><span class=\"p\">]]<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span>\n        <span class=\"n\">Law1<\/span><span class=\"p\">(<\/span><span class=\"n\">_LawSpec<\/span><span class=\"p\">.<\/span><span class=\"n\">reflexive_law<\/span><span class=\"p\">),<\/span>\n        <span class=\"n\">Law2<\/span><span class=\"p\">(<\/span><span class=\"n\">_LawSpec<\/span><span class=\"p\">.<\/span><span class=\"n\">symmetry_law<\/span><span class=\"p\">),<\/span>\n        <span class=\"n\">Law3<\/span><span class=\"p\">(<\/span><span class=\"n\">_LawSpec<\/span><span class=\"p\">.<\/span><span class=\"n\">transitivity_law<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">equals<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">,<\/span> <span class=\"n\">other<\/span><span class=\"p\">:<\/span> <span class=\"n\">_EqualType<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"Type-safe equality check for values of the same type.\"\"\"<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>That's what I call \"making tests a part of your app\"!<\/p>\n\n<p>Now, when we have laws in place, the only thing left to do is to enforce them. But, we need some data to do that. Luckily, we have <code>hypothesis<\/code> that can generate lots of random data for us.<\/p>\n\n<p>So, here's what we are going to do:<\/p>\n\n<ol>\n<li>We will pass a class definition that has <code>_laws<\/code> property defined<\/li>\n<li>\n<code>hypothesis<\/code> will get all its laws<\/li>\n<li>For each law we will generate a unique test case<\/li>\n<li>For each test case we will generate lots of input data to be sure that the law holds for any possible input<\/li>\n<\/ol>\n\n<p><a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/master\/returns\/contrib\/hypothesis\/laws.py\">Source code<\/a> for ones who are interested in the implementation details.<\/p>\n\n<p>And we should provide a simple API for an end-user to do all these in one function call! That's what we came up with:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># test_example.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"nn\">returns.contrib.hypothesis.laws<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">check_all_laws<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">your_app<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Example<\/span>\n\n<span class=\"n\">check_all_laws<\/span><span class=\"p\">(<\/span><span class=\"n\">Example<\/span><span class=\"p\">,<\/span> <span class=\"n\">use_init<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And here's the result:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight plaintext\"><code>\u00bb pytest test_example.py\n============================ test session starts ===============================\ncollected 3 items\n\ntest_example.py .F.                                                   [100%]\n\n=================================== FAILURES ===================================\n____________________ test_Example_equable_reflexive_law _____________________\nfirst = &lt;ex.Example object at 0x104d61b90&gt;\n\n    @law_definition\n    def reflexive_law(\n        first: _EqualType,\n    ) -&gt; None:\n        \"\"\"Value should be equal to itself.\"\"\"\n&gt;       assert first.equals(first)\nE       AssertionError\n\nreturns\/interfaces\/equable.py:32: AssertionError\n========================= 1 failed, 2 passed in 0.22s ==========================\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>As we can see <code>test_Example_equable_reflexive_law<\/code> fails, because <code>equals<\/code> always returns <code>False<\/code> in our <code>Example<\/code> class. And <code>reflexive_law<\/code> which states <code>(a == a) is True<\/code> does not hold.<\/p>\n\n<p>We can refactor <code>Example<\/code> to use the correct logic with actually checking <code>inner_value<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">Example<\/span><span class=\"p\">(<\/span><span class=\"n\">Equable<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">inner_value<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_inner_value<\/span> <span class=\"o\">=<\/span> <span class=\"n\">inner_value<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">equals<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">other<\/span><span class=\"p\">:<\/span> <span class=\"s\">'Example'<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_inner_value<\/span> <span class=\"o\">==<\/span> <span class=\"n\">other<\/span><span class=\"p\">.<\/span><span class=\"n\">_inner_value<\/span>  <span class=\"c1\"># no we are talking!\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And run our tests once again:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight plaintext\"><code>\u00bb pytest test_example.py\n============================= test session starts ==============================\ncollected 3 items\n\ntest_example.py ...                                                   [100%]\n\n============================== 3 passed in 1.57s ===============================\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>But, we didn't actually write a single test for <code>Example<\/code>. Instead, we wrote laws once and for all future implementations! That's how caring about users looks like.<\/p>\n\n<p>And again, awesome <code>hypothesis<\/code> helps us by generating random data to feed it into our tests (that's why the package is called <code>returns.contrib.hypothesis.laws<\/code>).<\/p>\n\n<h3>\n  \n  \n  Other functional laws\n<\/h3>\n\n<p>Of course, <code>Equable<\/code> is not the only interface we have in <code>dry-python\/returns<\/code>, we have <a href=\"https:\/\/github.com\/dry-python\/returns\/tree\/master\/returns\/interfaces\">lots of them<\/a>, covering most of the traditional functional instances, read our <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/interfaces.html\">docs<\/a> if you are interested.<\/p>\n\n<p>These interfaces will help people if they are wondering what <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/master\/returns\/interfaces\/container.py\"><code>Monad<\/code><\/a> actually is and what laws it has.<\/p>\n\n<p>Most of them have laws attached to the definition. This helps our users to be sure that their implementations are correct with as few steps as possible.<\/p>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>Shipping tests with your app might be a very cool feature in some use-cases.<\/p>\n\n<p>And use-cases are really-really different! As I have shown, they can vary from Web frameworks to architecture tools and math-ish libraries.<\/p>\n\n<p>I would love to see more of this in the future. I hope that I have shown possible benefits for current and future library authors.<\/p>\n\n","category":["python","django","webdev","beginners"]},{"title":"Higher Kinded Types in Python","pubDate":"Tue, 03 Nov 2020 12:39:34 +0000","link":"https:\/\/dev.to\/wemake-services\/higher-kinded-types-in-python-51n8","guid":"https:\/\/dev.to\/wemake-services\/higher-kinded-types-in-python-51n8","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2020\/10\/higher-kinded-types-in-python\">https:\/\/sobolevn.me\/2020\/10\/higher-kinded-types-in-python<\/a><\/p>\n\n<p><code>dry-python\/returns@0.15<\/code> is <a href=\"https:\/\/github.com\/dry-python\/returns\/releases\/tag\/0.15.0\">released<\/a>! And it means that now anyone can use our Higher Kinded Types emulation in their projects.<\/p>\n\n<p>In this post I will explain:<\/p>\n\n<ul>\n<li>What Higher Kinded Types (HKTs) are and why they are useful<\/li>\n<li>How they are implemented and what limitations there are<\/li>\n<li>How can you use them in your own projects<\/li>\n<\/ul>\n\n<p>Without further ado, let's talk about typing!<\/p>\n\n<h2>\n  \n  \n  Simple types\n<\/h2>\n\n<p>Typing is layered. Like a good cake. There are at least three layers that we are going to cover.<\/p>\n\n<p>Simple (or flat) typing, like <code>x: int = 1<\/code>. This allows us to express simple types and their transformations. Like <code>int -&gt; str<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">stringify<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>A lot of languages like <code>go<\/code> and <code>php<\/code> do not go beyond this line. And they still work pretty well! These types are also sometimes called <code>*<\/code>-kinded. It can be understood as \"a place for just a single type argument\".<\/p>\n\n<h2>\n  \n  \n  Generic\n<\/h2>\n\n<p>Generic level is required to express \"nested\" types. For example, you have a list of integers. In Python we annotate it as <code>List[int]<\/code> or <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0585\/\"><code>list[int]<\/code><\/a> in Python <code>3.9<\/code>. This allows us to have types with other types as arguments. <code>List<\/code> can receive <code>int<\/code> or <code>str<\/code> or even another <code>List<\/code> as the type argument. This way we can nest type and types start to have their own structure.<\/p>\n\n<p>Generics are much more interesting than simple types. And they can have multiple type arguments:<\/p>\n\n<ul>\n<li>List has one: for values, so it has a kind of <code>* -&gt; *<\/code>. It can be understood as a type transformation <code>List -&gt; T = List[T]<\/code>\n<\/li>\n<li>Dict has two: for keys and values, so it has a kind of <code>* -&gt; * -&gt; *<\/code>. It can be understood as a type transformation <code>Dict -&gt; K -&gt; V = Dict[K, V]<\/code>\n<\/li>\n<li>And so on!<\/li>\n<\/ul>\n\n<p>This would be very helpful for us in the future, I promise.<\/p>\n\n<p>We can also write transformations for generic types:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">stringify_list_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"n\">List<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">List<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"n\">item<\/span><span class=\"p\">)<\/span> <span class=\"k\">for<\/span> <span class=\"n\">item<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">arg<\/span><span class=\"p\">]<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>But, that is where things begin to be quite complicated.<\/p>\n\n<p>How can this function work with other iterables like <code>set<\/code>, <code>frozenset<\/code>, <code>tuple<\/code>?<br>\nWe can express this in Python as easy as:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">type<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)(<\/span><span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"n\">item<\/span><span class=\"p\">)<\/span> <span class=\"k\">for<\/span> <span class=\"n\">item<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>But the typing part would be quite challenging. Let's try several things.<\/p>\n\n<h3>\n  \n  \n  Common interface\n<\/h3>\n\n<p>The first obvious thing to try is <code>Iterable<\/code> protocol. It is builtin into Python and does what we need.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Iterable<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"n\">Iterable<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Iterable<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">type<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)(<\/span><span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"n\">item<\/span><span class=\"p\">)<\/span> <span class=\"k\">for<\/span> <span class=\"n\">item<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's try it out:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">]))<\/span>\n<span class=\"c1\"># Revealed type is 'typing.Iterable[builtins.str]'\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">({<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">}))<\/span>\n<span class=\"c1\">#  Revealed type is 'typing.Iterable[builtins.str]'\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">((<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\">#  Revealed type is 'typing.Iterable[builtins.str]'\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>You can see that a part of our typing information is lost. We pass <code>List<\/code> or <code>Set<\/code> or <code>Tuple<\/code> and always get the <code>Iterable<\/code> back.<\/p>\n\n<p>Sometimes - this is ok. But, in some cases, this is not enough. Let's try some other technique!<\/p>\n\n<h3>\n  \n  \n  Methods\n<\/h3>\n\n<p>One can say: we are all using Object-Oriented Programming! Why cannot we just create a new method for each type we need? And specify the exact return type there!<\/p>\n\n<p>Well, it is a possible solution. But, there are some reasonable problems:<\/p>\n\n<ul>\n<li>You cannot add methods to existing types and extend them with this approach. Only create new ones, probably via subtyping, and add new methods there. In our example you would have to create your own versions of <code>List<\/code>, <code>Set<\/code>, and <code>Tuple<\/code>. Which is not desirable in most situations<\/li>\n<li>It really starts to be messy if you have a lot of methods to add. A type with more than <code>X<\/code> (choose the number yourself) methods starts to be really complex to read, understand, and use. Instead, using separate functions is much easier, because we don't have to put everything into a single namespace<\/li>\n<\/ul>\n\n<p>Let's try something else.<\/p>\n\n<h3>\n  \n  \n  overloads\n<\/h3>\n\n<p>Another solution that might solve our problem is using the <code>@overload<\/code> decorator with proper types for each required case.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">List<\/span><span class=\"p\">,<\/span> <span class=\"n\">Set<\/span><span class=\"p\">,<\/span> <span class=\"n\">overload<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">overload<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"n\">List<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">List<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">]:<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">overload<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"n\">Set<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Set<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">]:<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">type<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)(<\/span><span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"n\">item<\/span><span class=\"p\">)<\/span> <span class=\"k\">for<\/span> <span class=\"n\">item<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's test it:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">]))<\/span>\n<span class=\"c1\"># Revealed type is 'builtins.list[builtins.str]'\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">({<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">}))<\/span>\n<span class=\"c1\"># Revealed type is 'builtins.set[builtins.str]'\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Awesome! Looks like we've achieved our goal, haven't we? But, there's a new problem. We have to manually list all possible cases in a function's signature. This works for cases when all possible arguments and outcomes are known in advance. But, not in this case. In Python <code>Iterable<\/code> is a protocol. We can use this function with any type with <code>__iter__<\/code> method defined: with both builtin and custom types. So, the number of possible arguments and outcomes is endless.<\/p>\n\n<p>To illusrate the problem, let's see what happens for <code>Tuple<\/code> which is not listed in the function's overloads:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">((<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\"># error: No overload variant of \"stringify_iterable_items\" matches argument type \"Tuple[int, int, int]\"\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>We, in <code>dry-python<\/code> <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/0.14.0\/returns\/_generated\/converters\/flatten.pyi\">used this technique<\/a> with <code>@overload<\/code> decorator for our previous versions. This allowed us to write correct definitions of functions working with generic types. But, they were limited to the pre-defined set of our own types. And we wanted to allow our users to create their custom types based on our interfaces. With the full existing code reuse.<\/p>\n\n<h2>\n  \n  \n  Higher Kinded Types\n<\/h2>\n\n<p>That's where the idea of Higher Kinded Types becomes useful. We need HKTs when we want to change the inner structure of generics with full type information preserving and openness to the extension. In theory, you can write something like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Iterable<\/span>\n\n<span class=\"n\">T<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'T'<\/span><span class=\"p\">,<\/span> <span class=\"n\">bound<\/span><span class=\"o\">=<\/span><span class=\"n\">Iterable<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"n\">T<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">T<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">type<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)(<\/span><span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"n\">item<\/span><span class=\"p\">)<\/span> <span class=\"k\">for<\/span> <span class=\"n\">item<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And this would solve our problem! What happens here is that we abstract away the <code>Iterable<\/code> type itself. And then ask <code>mypy<\/code> to figure this out for us.<\/p>\n\n<p>This way we can potentially have <code>stringify_iterable_items<\/code> working for any <code>Iterable<\/code> type, but with the exact same type returned back without any information lost. And it would work for all types.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">]))<\/span>\n<span class=\"c1\"># Revealed type is 'builtins.list[builtins.str]'\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">({<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">}))<\/span>\n<span class=\"c1\"># Revealed type is 'builtins.set[builtins.str]'\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">stringify_iterable_items<\/span><span class=\"p\">(<\/span><span class=\"n\">MyCustomIterable<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">)))<\/span>\n<span class=\"c1\"># Revealed type is 'my_module.MyCustomIterable[builtins.str]'\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Unfortunately, <a href=\"https:\/\/github.com\/python\/typing\/issues\/548\">it is not supported<\/a> at the moment.<\/p>\n\n<h3>\n  \n  \n  Emulation\n<\/h3>\n\n<p>Turns out we are not alone in this situation. There are multiple languages where Higher Kinded Types are not natively supported yet. But, they can be emulated:<\/p>\n\n<ul>\n<li><a href=\"https:\/\/github.com\/gcanti\/fp-ts\/blob\/master\/docs\/guides\/HKT.md\">TypeScript<\/a><\/li>\n<li><a href=\"https:\/\/bow-swift.io\/docs\/fp-concepts\/higher-kinded-types\/\">Swift<\/a><\/li>\n<li><a href=\"https:\/\/arrow-kt.io\/docs\/0.10\/patterns\/glossary\/#higher-kinds\">Kotlin<\/a><\/li>\n<\/ul>\n\n<p>And now with <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/hkt.html\">Python<\/a> too!<\/p>\n\n<p>There's also <a href=\"https:\/\/www.cl.cam.ac.uk\/~jdy22\/papers\/lightweight-higher-kinded-polymorphism.pdf\">an original whitepaper<\/a> for ones who are interested.<\/p>\n\n<p>The core idea of HKT emulation is that we can write types the other way around: not like <code>T[int]<\/code>, but rather like <code>Kind[T, int]<\/code> (which is absolutely the same thing).<\/p>\n\n<p>This way we can transform the inner structure of generics, but maintain the simple context without reinventing <code>TypeVar<\/code> with type arguments. And our function's type signature will look like: <code>Kind[T, int] -&gt; Kind[T, str]<\/code>.<\/p>\n\n<p>Let's see the implementation.<\/p>\n\n<h2>\n  \n  \n  Implementation\n<\/h2>\n\n<p><strong>TLDR<\/strong>: here's <a href=\"https:\/\/gist.github.com\/sobolevn\/7f8ffd885aec70e55dd47928a1fb3e61\">the final working code<\/a> with all the logic, all the hacks, and everything. In this article, we going to write and explain it step by step.<\/p>\n\n<p>We would need a better example to test our implementation. Let's build two types: a <code>Box<\/code> and a <code>Bag<\/code>. <code>Box<\/code> is defined by its size, while a <code>Bag<\/code> is an item of fashion: it has a brand name and a model name (I have a wife, I know this stuff!).<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">dataclasses<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">dataclass<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">,<\/span> <span class=\"n\">Generic<\/span><span class=\"p\">,<\/span> <span class=\"n\">TypeVar<\/span>\n\n<span class=\"n\">_ValueType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_ValueType'<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">_NewValueType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_NewValueType'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"n\">value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_ValueType<\/span>\n    <span class=\"n\">length<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">width<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">height<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"n\">value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_ValueType<\/span>\n    <span class=\"n\">brand<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n    <span class=\"n\">model<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And we can create <code>Box<\/code>es and <code>Bag<\/code>s of different types, because we can put different things inside them:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">box<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"o\">=<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span> <span class=\"n\">length<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"n\">width<\/span><span class=\"o\">=<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">height<\/span><span class=\"o\">=<\/span><span class=\"mi\">3<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># Box[int]\n<\/span><span class=\"n\">bag<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"o\">=<\/span><span class=\"mi\">5<\/span><span class=\"p\">,<\/span> <span class=\"n\">brand<\/span><span class=\"o\">=<\/span><span class=\"s\">'Fancy'<\/span><span class=\"p\">,<\/span> <span class=\"n\">model<\/span><span class=\"o\">=<\/span><span class=\"s\">'Baggy'<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># Bag[int]\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Now, we need a function with a type transformation. Let's say we want to apply a function to the value inside boxes and bags. Let's use fake <code>BoxOrBag<\/code> type for now to illustrate our intent:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span>\n\n<span class=\"n\">_NewValueType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_NewValueType'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">apply_function<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">BoxOrBag<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span>  <span class=\"c1\"># fake type for now\n<\/span>    <span class=\"n\">callback<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">BoxOrBag<\/span><span class=\"p\">[<\/span><span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>  <span class=\"c1\"># fake type for now\n<\/span>    <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>It is going to work like so:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">assert<\/span> <span class=\"n\">apply_function<\/span><span class=\"p\">(<\/span><span class=\"n\">box<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"n\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"o\">=<\/span><span class=\"s\">'10'<\/span><span class=\"p\">,<\/span> <span class=\"n\">length<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"n\">width<\/span><span class=\"o\">=<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">height<\/span><span class=\"o\">=<\/span><span class=\"mi\">3<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">assert<\/span> <span class=\"n\">apply_function<\/span><span class=\"p\">(<\/span><span class=\"n\">bag<\/span><span class=\"p\">,<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"n\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">,<\/span> <span class=\"n\">brand<\/span><span class=\"o\">=<\/span><span class=\"s\">'Fancy'<\/span><span class=\"p\">,<\/span> <span class=\"n\">model<\/span><span class=\"o\">=<\/span><span class=\"s\">'Baggy'<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>We only need to change current fake <code>BoxOrBag<\/code> type to a real HKT. We would need to define a new <code>Kind<\/code> type to make the emulation:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">_InstanceType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_InstanceType'<\/span><span class=\"p\">,<\/span> <span class=\"n\">covariant<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">_FirstTypeArgType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_FirstTypeArgType'<\/span><span class=\"p\">,<\/span> <span class=\"n\">covariant<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Kind<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_FirstTypeArgType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"s\">\"\"\"Used for HKT emulation.\"\"\"<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>One pro-tip about <code>Kind<\/code>: it won't not exist during runtime. Only during type-checking.<\/p>\n\n<p>Now, let's change <code>apply_function<\/code> to use <code>Kind<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">_InstanceKind<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_InstanceKind'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">apply_function<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span>\n    <span class=\"n\">callback<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n    <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And last, but not least: we need an implementation for <code>apply_function<\/code>!<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">apply_function<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span>\n    <span class=\"n\">callback<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">new_value<\/span> <span class=\"o\">=<\/span> <span class=\"n\">callback<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">value<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># creating new value\n<\/span>    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">with_value<\/span><span class=\"p\">(<\/span><span class=\"n\">new_value<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># creating a new instance from it\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>If we try to run this code, we would see something like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight plaintext\"><code>error: \"_InstanceKind\" has no attribute \"value\"\nerror: \"_InstanceKind\" has no attribute \"with_value\"\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>This happens because <code>_InstanceKind<\/code> does not know anything about an interface we are working with right now. We need to bind it to something. And we also need an interface to work with:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">abc<\/span>\n\n<span class=\"n\">_InstanceKind<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_InstanceKind'<\/span><span class=\"p\">,<\/span> <span class=\"n\">bound<\/span><span class=\"o\">=<\/span><span class=\"s\">'HasValue'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">HasValue<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"o\">@<\/span><span class=\"n\">abc<\/span><span class=\"p\">.<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"o\">@<\/span><span class=\"nb\">property<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">value<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"Returns a value property.\"\"\"<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">abc<\/span><span class=\"p\">.<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">with_value<\/span><span class=\"p\">(<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">:<\/span> <span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">new_value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind1<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n        <span class=\"s\">\"\"\"Creates a new instance with a changed value.\"\"\"<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Take an extra look at the abstract <code>with_value<\/code> definition: it is also Higher Kinded Typed. Because the interface knows that we are dealing with the internal structure change of a generic type. We would also need to subclass <code>HasValue<\/code> and implement the <code>with_value<\/code> method for both <code>Bag<\/code> and <code>Box<\/code> types. The good thing about subclassing <code>HasValue<\/code> is that HKTs as well as regular types will protect us from defining incompatible methods and properties.<\/p>\n\n<p>Let's review what we have done so far:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">abc<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">dataclasses<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">,<\/span> <span class=\"n\">Generic<\/span><span class=\"p\">,<\/span> <span class=\"n\">TypeVar<\/span>\n\n<span class=\"n\">_InstanceType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_InstanceType'<\/span><span class=\"p\">,<\/span> <span class=\"n\">covariant<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">_FirstTypeArgType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_FirstTypeArgType'<\/span><span class=\"p\">,<\/span> <span class=\"n\">covariant<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Kind<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_FirstTypeArgType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"s\">\"\"\"Used for HKT emulation.\"\"\"<\/span>\n\n<span class=\"n\">_ValueType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_ValueType'<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">_NewValueType<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_NewValueType'<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">_InstanceKind<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_InstanceKind'<\/span><span class=\"p\">,<\/span> <span class=\"n\">bound<\/span><span class=\"o\">=<\/span><span class=\"s\">'HasValue'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">HasValue<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"o\">@<\/span><span class=\"n\">abc<\/span><span class=\"p\">.<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"o\">@<\/span><span class=\"nb\">property<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">value<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">:<\/span>\n        <span class=\"s\">\"\"\"Returns a value property.\"\"\"<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">abc<\/span><span class=\"p\">.<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">with_value<\/span><span class=\"p\">(<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">:<\/span> <span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">new_value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n        <span class=\"s\">\"\"\"Creates a new instance with a changed value.\"\"\"<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">dataclasses<\/span><span class=\"p\">.<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">HasValue<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"n\">value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_ValueType<\/span>\n    <span class=\"n\">length<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">width<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">height<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">with_value<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">new_value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"s\">'Box[_NewValueType]'<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">new_value<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">length<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">width<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">height<\/span><span class=\"p\">)<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">dataclasses<\/span><span class=\"p\">.<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">HasValue<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"n\">value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_ValueType<\/span>\n    <span class=\"n\">brand<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n    <span class=\"n\">model<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">with_value<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">new_value<\/span><span class=\"p\">:<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"s\">'Bag[_NewValueType]'<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">new_value<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">brand<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">model<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">apply_function<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span>\n    <span class=\"n\">callback<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">new_value<\/span> <span class=\"o\">=<\/span> <span class=\"n\">callback<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">value<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">with_value<\/span><span class=\"p\">(<\/span><span class=\"n\">new_value<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>That's pretty much it! I can proudly call this \"unhacked\" version. In an ideal world, our HKT emulation would <em>just work<\/em>. But, in our real-life world - we still need some hacks to make this possible.<\/p>\n\n<h2>\n  \n  \n  Hacks\n<\/h2>\n\n<p>First of all, let's try to type-check our code to see what will happen:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code>\u00bb mypy ex.py\nex.py:57: error: <span class=\"s2\">\"Kind[_InstanceKind, _ValueType]\"<\/span> has no attribute <span class=\"s2\">\"value\"<\/span>\nex.py:58: error: Returning Any from <span class=\"k\">function <\/span>declared to <span class=\"k\">return<\/span> <span class=\"s2\">\"Kind[_InstanceKind, _NewValueType]\"<\/span>\nex.py:58: error: <span class=\"s2\">\"Kind[_InstanceKind, _ValueType]\"<\/span> has no attribute <span class=\"s2\">\"with_value\"<\/span>\nFound 3 errors <span class=\"k\">in <\/span>1 file <span class=\"o\">(<\/span>checked 1 <span class=\"nb\">source <\/span>file<span class=\"o\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's think about it for a moment. Why <code>\"Kind[_InstanceKind, _ValueType]\"<\/code> does not have attribute <code>\"value\"<\/code>?<\/p>\n\n<h3>\n  \n  \n  getattr hook\n<\/h3>\n\n<p>This happens because we wrote exactly that in <code>apply_function<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">]<\/span>  <span class=\"c1\"># Kind has 0 methods and props\n<\/span>\n<span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">value<\/span>  <span class=\"c1\"># this errors, because, again, Kind has 0 methods and values\n<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">with_value<\/span>  <span class=\"c1\"># the same\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>We need to build something that understands our intention: when we use dot access on <code>Kind<\/code> - in reality, we want to dot access its <code>_InstanceKind<\/code> type.<\/p>\n\n<p>Since it can take quite a lot of time, I am going to skip re-implementing this part and give links to the original source code here:<\/p>\n\n<ol>\n<li>Add <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/0.15.0\/returns\/primitives\/hkt.py#L92-L99\"><code>__getattr__<\/code> method<\/a> to <code>Kind<\/code>\n<\/li>\n<li>Write <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/0.15.0\/returns\/contrib\/mypy\/_features\/kind.py#L22-L67\">a custom <code>mypy<\/code> plugin<\/a> to catch and modify attribute access on <code>Kind<\/code>\n<\/li>\n<\/ol>\n\n<p>In the result we would have a working type inference:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">apply_function<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span>\n    <span class=\"n\">callback<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">value<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Revealed type is '_ValueType`-2'\n<\/span>    <span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">with_value<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Revealed type is 'def [_NewValueType] (new_value: _NewValueType`6) -&gt; Kind[_InstanceKind`-1, _NewValueType`6]'\n<\/span>    <span class=\"n\">new_value<\/span> <span class=\"o\">=<\/span> <span class=\"n\">callback<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">value<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">with_value<\/span><span class=\"p\">(<\/span><span class=\"n\">new_value<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  SupportsKind\n<\/h3>\n\n<p>There are two more things to solve.<\/p>\n\n<p>First, since <code>Kind<\/code> is a type and <code>mypy<\/code> uses nominal inheritance to check subtyping (I have tried to make <code>Kind<\/code> a <code>Protocol<\/code> several times, but it does not work this way for now) it won't be happy that we declare <code>Kind<\/code> as a return type, but return <code>Bag<\/code> or <code>Box<\/code> instead. Since these two types are unrelated in <code>mypy<\/code>'s opinion - we would need to fix that with a direct inheritance from <code>Kind<\/code>.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">dataclasses<\/span><span class=\"p\">.<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"s\">'Bag'<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">HasValue<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">dataclasses<\/span><span class=\"p\">.<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"s\">'Box'<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">HasValue<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Now <code>Box<\/code> and <code>Bag<\/code> are both nominal subtypes of <code>Kind<\/code> and <code>mypy<\/code> will be happy with that.<\/p>\n\n<p>Also, notice this notation <code>Kind['Box', _ValueType]<\/code>: what it means is that we inherit from <code>Kind<\/code> and pass not-yet-defined <code>Box<\/code> type inside as type argument. Sometimes it is also called the <code>URI<\/code> parameter.<\/p>\n\n<p>Second, since <code>Kind<\/code> defines <code>__getattr__<\/code> we would end up with catching all dot access behaviour during type-checking. For example, <code>box.missing_attr<\/code> won't raise a type error anymore. But, would still fail during execution. We need to fix that! We can create a intermediate layer called <code>SupportsKind<\/code> with removed <code>__getattr__<\/code> method:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">SupportsKind<\/span><span class=\"p\">(<\/span><span class=\"n\">KindN<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_FirstTypeArgType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"n\">__getattr__<\/span><span class=\"p\">:<\/span> <span class=\"bp\">None<\/span>  <span class=\"c1\"># type: ignore\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Yes, this is a bit ugly, but we have to only define it once and then just use it:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">dataclasses<\/span><span class=\"p\">.<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">SupportsKind<\/span><span class=\"p\">[<\/span><span class=\"s\">'Bag'<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">HasValue<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">dataclasses<\/span><span class=\"p\">.<\/span><span class=\"n\">dataclass<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">SupportsKind<\/span><span class=\"p\">[<\/span><span class=\"s\">'Box'<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">HasValue<\/span><span class=\"p\">[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">]):<\/span>\n    <span class=\"p\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Now, we have <code>box.missing_attr<\/code> fixed!<\/p>\n\n<h3>\n  \n  \n  kinded\n<\/h3>\n\n<p>And the last hack we need is to translate <code>Kind[_InstanceKind, _NewValueType]<\/code> to <code>_InstanceKind[_NewValueType]<\/code> in the function's return type. Let's see how it works right now without this transformation:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">box<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Box<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"o\">=<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span> <span class=\"n\">length<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"n\">width<\/span><span class=\"o\">=<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">height<\/span><span class=\"o\">=<\/span><span class=\"mi\">3<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">bag<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Bag<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"o\">=<\/span><span class=\"mi\">5<\/span><span class=\"p\">,<\/span> <span class=\"n\">brand<\/span><span class=\"o\">=<\/span><span class=\"s\">'Fancy'<\/span><span class=\"p\">,<\/span> <span class=\"n\">model<\/span><span class=\"o\">=<\/span><span class=\"s\">'Baggy'<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">apply_function<\/span><span class=\"p\">(<\/span><span class=\"n\">box<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Revealed type is 'ex.Kind[ex.Box[Any], builtins.str*]'\n<\/span><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">apply_function<\/span><span class=\"p\">(<\/span><span class=\"n\">bag<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Revealed type is 'ex.Kind[ex.Bag[Any], builtins.str*]'\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Almost what we need!<\/p>\n\n<p>We can use <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/0.15.0\/returns\/primitives\/hkt.py#L208\">a <code>@kinded<\/code> decorator<\/a> for that together with <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/0.15.0\/returns\/contrib\/mypy\/_features\/kind.py#L116-L130\">one more custom <code>mypy<\/code> plugin<\/a>. The idea of this hack is that we catch all calls of functions decorated with <code>@kinded<\/code> and transform their return type from <code>Kind[I, T]<\/code> into <code>I[T]<\/code>.<\/p>\n\n<p>The final iteration of <code>apply_function<\/code> will look like this:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">kinded<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">apply_function<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span>\n    <span class=\"n\">callback<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_ValueType<\/span><span class=\"p\">],<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">_NewValueType<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">new_value<\/span> <span class=\"o\">=<\/span> <span class=\"n\">callback<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">value<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">with_value<\/span><span class=\"p\">(<\/span><span class=\"n\">new_value<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And would work as we want:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">apply_function<\/span><span class=\"p\">(<\/span><span class=\"n\">box<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Revealed type is 'ex.Box[builtins.str*]'\n<\/span><span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">apply_function<\/span><span class=\"p\">(<\/span><span class=\"n\">bag<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># Revealed type is 'ex.Bag[builtins.str*]'\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's discuss the future of this hacks. Because, we don't want them to stay!<\/p>\n\n<h2>\n  \n  \n  Limitations\n<\/h2>\n\n<p>The main limitation, for now, is that <code>Kind<\/code> uses nominal inheritance and is a custom type. So, for now, it is impossible to work with 1st or 3rd party generics like <code>List<\/code>, <code>Set<\/code>, etc. We can only work with types who directly inherit from <code>Kind<\/code> for now.<\/p>\n\n<p>The second limitation comes from the fact we can have generics with a different number of type arguments, as I have shown you before:<\/p>\n\n<ul>\n<li>\n<code>List<\/code> has a kind of <code>* -&gt; *<\/code>\n<\/li>\n<li>\n<code>Dict<\/code> has a kind of <code>* -&gt; * -&gt; *<\/code>\n<\/li>\n<\/ul>\n\n<p>So, we would need several <a href=\"https:\/\/github.com\/dry-python\/returns\/blob\/master\/returns\/primitives\/hkt.py#L26\"><code>Kind<\/code> alises<\/a> to work with different amount of type arguments. It would also have a reasonable limit of three type arguments:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">KindN<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">Generic<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType1<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType2<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType3<\/span><span class=\"p\">],<\/span>\n<span class=\"p\">):<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"c1\">#: Type alias for kinds with one type argument.\n<\/span><span class=\"n\">Kind1<\/span> <span class=\"o\">=<\/span> <span class=\"n\">KindN<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType1<\/span><span class=\"p\">,<\/span> <span class=\"n\">Any<\/span><span class=\"p\">,<\/span> <span class=\"n\">Any<\/span><span class=\"p\">]<\/span>\n\n<span class=\"c1\">#: Type alias for kinds with two type arguments.\n<\/span><span class=\"n\">Kind2<\/span> <span class=\"o\">=<\/span> <span class=\"n\">KindN<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType1<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType2<\/span><span class=\"p\">,<\/span> <span class=\"n\">Any<\/span><span class=\"p\">]<\/span>\n\n<span class=\"c1\">#: Type alias for kinds with three type arguments.\n<\/span><span class=\"n\">Kind3<\/span> <span class=\"o\">=<\/span> <span class=\"n\">KindN<\/span><span class=\"p\">[<\/span><span class=\"n\">_InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType1<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType2<\/span><span class=\"p\">,<\/span> <span class=\"n\">_TypeArgType3<\/span><span class=\"p\">]<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>A lot of readers might be shocked by the number of hacks we have to apply to make this work. I guess it is time for me to explain what hacks are temporary and which are going to stay with us.<\/p>\n\n<p>The good news, <strong>all<\/strong> hacks can be solved in the future.<\/p>\n\n<p>Most of the hacks come from the fact that <code>mypy<\/code> does not know what <code>Kind<\/code> is. Possibly, in the future, it can be built into <code>mypy<\/code>. And <code>@kinded<\/code>, <code>__getattr__<\/code>, and <code>SupportsKind<\/code> hacks would go away. Together with the fact that <code>Kind<\/code> needs to be a direct supertype of types we want to represent as HKTs.<\/p>\n\n<p>Variadic generics are also in-the-work right now. You can have a look at the <a href=\"https:\/\/mail.python.org\/archives\/list\/typing-sig@python.org\/thread\/SQVTQYWIOI4TIO7NNBTFFWFMSMS2TA4J\/\">PEP draft<\/a>. So, in the future <code>KindN<\/code>, <code>Kind1<\/code>, and <code>Kind2<\/code> would hopefully go away.<\/p>\n\n<h2>\n  \n  \n  Real life use-case\n<\/h2>\n\n<p>I am going to finish this article with an example <a href=\"https:\/\/sobolevn.me\/2020\/06\/how-async-should-have-been\">I have promised you in my previous article<\/a> (it took almost half a year of hard work to fulfill!).<\/p>\n\n<p>Last time we talked about making a fully typed function that can work with sync and async HTTP clients with the same API.<\/p>\n\n<p>And here it is:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">,<\/span> <span class=\"n\">TypeVar<\/span>\n\n<span class=\"kn\">import<\/span> <span class=\"nn\">anyio<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">httpx<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.future<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">future_safe<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.interfaces.specific.ioresult<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">IOResultLike2<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.io<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">impure_safe<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.primitives.hkt<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Kind2<\/span><span class=\"p\">,<\/span> <span class=\"n\">kinded<\/span>\n\n<span class=\"n\">_IOKind<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'_IOKind'<\/span><span class=\"p\">,<\/span> <span class=\"n\">bound<\/span><span class=\"o\">=<\/span><span class=\"n\">IOResultLike2<\/span><span class=\"p\">)<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">kinded<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">Kind2<\/span><span class=\"p\">[<\/span><span class=\"n\">_IOKind<\/span><span class=\"p\">,<\/span> <span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">,<\/span> <span class=\"nb\">Exception<\/span><span class=\"p\">]],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Kind2<\/span><span class=\"p\">[<\/span><span class=\"n\">_IOKind<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"nb\">Exception<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>When used with sync <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/io.html\"><code>IO<\/code><\/a> type, it would work as a regular function.<br>\nBut, it can also work with async <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/future.html\"><code>Future<\/code><\/a> type.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># Sync:\n<\/span><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">impure_safe<\/span><span class=\"p\">(<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">),<\/span>\n    <span class=\"s\">'https:\/\/sobolevn.me'<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">))<\/span>\n<span class=\"c1\"># =&gt; &lt;IOResult: &lt;Success: 27972&gt;&gt;\n<\/span>\n<span class=\"c1\"># Async:\n<\/span><span class=\"n\">page_size<\/span> <span class=\"o\">=<\/span> <span class=\"n\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">future_safe<\/span><span class=\"p\">(<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">AsyncClient<\/span><span class=\"p\">().<\/span><span class=\"n\">get<\/span><span class=\"p\">),<\/span>\n    <span class=\"s\">'https:\/\/sobolevn.me'<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">page_size<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">anyio<\/span><span class=\"p\">.<\/span><span class=\"n\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">page_size<\/span><span class=\"p\">.<\/span><span class=\"n\">awaitable<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># =&gt; &lt;FutureResult: &lt;coroutine object async_map at 0x10b17c320&gt;&gt;\n# =&gt; &lt;IOResult: &lt;Success: 27972&gt;&gt;\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>The possibilities with Higher Kinded Types are endless! It shines with:<\/p>\n\n<ul>\n<li>Functional programming, like we do in <code>dry-python<\/code>\n<\/li>\n<li>Computational libraries, when people type things like <code>Tensor<\/code> or <code>Matrix<\/code> or <code>Array<\/code>\n<\/li>\n<\/ul>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>Wow, it was a cool ride! I hope you have enjoyed it as much as I did! Working on Higher Kinded Types was super fun and I am excited for you to try it.<\/p>\n\n<ul>\n<li>Docs: <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/index.html\">https:\/\/returns.readthedocs.io\/en\/latest\/index.html<\/a>\n<\/li>\n<li>Repository: <a href=\"https:\/\/github.com\/dry-python\/returns\">https:\/\/github.com\/dry-python\/returns<\/a> Don't forget to star it!<\/li>\n<\/ul>\n\n<p>P.S. Does your company want corporate training on using types in Python with <code>mypy<\/code>? We, in <a href=\"https:\/\/drylabs.io\/\">DryLabs<\/a> can help! Drop us a line.<\/p>\n\n","category":["python","mypy","webdev","beginners"]},{"title":"How async should have been","pubDate":"Tue, 16 Jun 2020 07:57:46 +0000","link":"https:\/\/dev.to\/wemake-services\/how-async-should-have-been-5df8","guid":"https:\/\/dev.to\/wemake-services\/how-async-should-have-been-5df8","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2020\/06\/how-async-should-have-been\">https:\/\/sobolevn.me\/2020\/06\/how-async-should-have-been<\/a><\/p>\n\n<p>In the last few years <code>async<\/code> keyword and semantics made its way into many popular programming languages: <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/JavaScript\/Reference\/Statements\/async_function\">JavaScript<\/a>, <a href=\"https:\/\/doc.rust-lang.org\/std\/keyword.async.html\">Rust<\/a>, <a href=\"https:\/\/docs.microsoft.com\/en-us\/dotnet\/csharp\/language-reference\/operators\/await\">C#<\/a>, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Async\/await\">many others<\/a> languages that I don't know or don't use.<\/p>\n\n<p>Of course, Python also has <code>async<\/code> and <code>await<\/code> keywords since <code>python3.5<\/code>.<\/p>\n\n<p>In this article, I would like to provide my opinion about this feature, think of alternatives, and provide a new solution.<\/p>\n\n<h2>\n  \n  \n  Colours of functions\n<\/h2>\n\n<p>When introducing <code>async<\/code> functions into the languages, we actually end up with a split world. Now, some functions start to be red (or <code>async<\/code>) and old ones continue to be blue (sync).<\/p>\n\n<p>The thing about this division is that blue functions cannot call red ones.<br>\nRed ones potentially can call blue ones. In Python, for example, it is partially true. Async functions can only call sync non-blocking functions. Is it possible to tell whether this function is blocking or not by its definition? Of course not! Python is a scripting language, don't forget about that!<\/p>\n\n<p>This division creates two subsets of a single language: sync and async ones.<br>\n5 years passed since the release of <code>python3.5<\/code>, but <code>async<\/code> support is not even near to what we have in the sync python world.<\/p>\n\n<p>Read <a href=\"http:\/\/journal.stuffwithstuff.com\/2015\/02\/01\/what-color-is-your-function\/\">this brilliant piece<\/a> if you want to learn more about colors of functions.<\/p>\n<h2>\n  \n  \n  Code duplication\n<\/h2>\n\n<p>Different colors of functions lead to a more practical problem: code duplication.<\/p>\n\n<p>Imagine, that you are writing a CLI tool to fetch sizes of web pages. And you want to support both sync and async ways of doing it. This might be very useful for library authors when you don't know how your code is going to be used. It is not limited to just PyPI libraries, but also includes your in-company libraries with shared logic for different services written, for example, in Django and aiohttp. Or any other sync and async code. But, I must admit that single applications are mostly written in sync or async way only.<\/p>\n\n<p>Let's start with the sync pseudo-code:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Looking pretty good! Now, let's add its async counterpart:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">async<\/span> <span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"k\">await<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>It is basically the same code, but filled with <code>async<\/code> and <code>await<\/code> keywords!<br>\nAnd I am not making this up, just compare code sample in <code>httpx<\/code> tutorial:<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/www.python-httpx.org\/quickstart\/\">Sync<\/a> code<\/li>\n<li>vs <a href=\"https:\/\/www.python-httpx.org\/async\/\">Async<\/a> code<\/li>\n<\/ul>\n\n<p>They show exactly the same picture.<\/p>\n<h2>\n  \n  \n  Abstraction and Composition\n<\/h2>\n\n<p>Ok, we find ourselves in a situation where we need to rewrite all sync code and add <code>async<\/code> and <code>await<\/code> keywords here and there, so our program would become asynchronous.<\/p>\n\n<p>These two principles can help us in solving this problem.<\/p>\n\n<p>First of all, let's rewrite our imperative pseudo-code into a functional pseudo-code. This will allow us to see the pattern more clearly:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Abstraction<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>What is this <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/container.html#working-with-a-container\"><code>.map<\/code><\/a> method? What does it do?<\/p>\n\n<p>This is a functional way of composing complex abstractions and pure functions. This method allows creating a new abstraction from the existing one with the new state. Let's say that when we call <code>client_get(url)<\/code> it initially returns <code>Abstraction[Response]<\/code> and calling <code>.map(lambda response: len(response.content))<\/code> transforms it to the needed <code>Abstraction[int]<\/code> instance.<\/p>\n\n<p>Now the steps are pretty clear! Notice how easily we went from several independent steps into a single pipeline of function calls. We have also changed the return type of this function: now it returns some <code>Abstraction<\/code>.<\/p>\n\n<p>Now, let's rewrite our code to work with async version:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">AsyncAbstraction<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Wow, that's mostly it! The only thing that is different is the <code>AsyncAbstraction<\/code> return type. Other than that, our code stayed exactly the same. We also don't need to use <code>async<\/code> and <code>await<\/code> keywords anymore. We don't use <code>await<\/code> at all (that's the whole point of our journey!), and <code>async<\/code> functions do not make any sense without <code>await<\/code>.<\/p>\n\n<p>The last thing we need is to decide which client we want: async or sync one. Let's fix that!<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">AbstactionType<\/span><span class=\"p\">[<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">AbstactionType<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Our <code>client_get<\/code> is now an argument of a callable type that receives a single URL string as an input and returns some <code>AbstractionType<\/code> over <code>Response<\/code> object. This <code>AbstractionType<\/code> is either <code>Abstraction<\/code> or <code>AsyncAbstraction<\/code> we have already seen on the previous samples.<\/p>\n\n<p>When we pass <code>Abstraction<\/code> our code works like a sync one, when <code>AsyncAbstraction<\/code> is passed, the same code automatically starts to work asynchronously.<\/p>\n\n<h2>\n  \n  \n  IOResult and FutureResult\n<\/h2>\n\n<p>Luckily, we already have the right abstractions in <a href=\"https:\/\/github.com\/dry-python\/returns\"><code>dry-python\/returns<\/code><\/a>!<\/p>\n\n<p>Let me introduce to you type-safe, <code>mypy<\/code>-friendly, framework-independent, pure-python tool to provide you awesome abstractions you can use in any project!<\/p>\n\n<h3>\n  \n  \n  Sync version\n<\/h3>\n\n<p>Before we go any further, to make this example reproducible, I need to provide dependencies that are going to be used later:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight shell\"><code>pip <span class=\"nb\">install <\/span>returns httpx anyio\n<\/code><\/pre><\/div>\n\n\n\n<p>Let's move on!<\/p>\n\n<p>One can rewrite this pseudo-code as a real working python code. Let's start with the sync version:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span>\n\n<span class=\"kn\">import<\/span> <span class=\"nn\">httpx<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.io<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">,<\/span> <span class=\"n\">impure_safe<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">[<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">impure_safe<\/span><span class=\"p\">(<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">),<\/span>\n    <span class=\"s\">'https:\/\/sobolevn.me'<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">))<\/span>\n<span class=\"c1\"># =&gt; &lt;IOResult: &lt;Success: 27972&gt;&gt;\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>We have changed a couple of things to make our pseudo-code real:<\/p>\n\n<ol>\n<li>We now use <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/io.html\"><code>IOResultE<\/code><\/a> which is a functional way to handle sync <code>IO<\/code> that might fail. Remember, <a href=\"https:\/\/sobolevn.me\/2019\/02\/python-exceptions-considered-an-antipattern\">exceptions are not always welcome<\/a>! <code>Result<\/code>-based types allow modeling exceptions as separate <code>Failure()<\/code> values. While successful values are wrapped in <code>Success<\/code> type. In a traditional approach, no one cares about exceptions. But, we do care \u2764\ufe0f<\/li>\n<li>We use <a href=\"https:\/\/github.com\/encode\/httpx\/\"><code>httpx<\/code><\/a> that can work with sync and async requests<\/li>\n<li>We use <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/io.html#impure-safe\"><code>impure_safe<\/code><\/a> function to convert the return type of <code>httpx.get<\/code> to return the abstraction we need: <code>IOResultE<\/code>\n<\/li>\n<\/ol>\n\n<p>Now, let's try the async version!<\/p>\n\n<h3>\n  \n  \n  Async version\n<\/h3>\n\n\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span>\n\n<span class=\"kn\">import<\/span> <span class=\"nn\">anyio<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">httpx<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.future<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">,<\/span> <span class=\"n\">future_safe<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">[<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n\n<span class=\"n\">page_size<\/span> <span class=\"o\">=<\/span> <span class=\"n\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">future_safe<\/span><span class=\"p\">(<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">AsyncClient<\/span><span class=\"p\">().<\/span><span class=\"n\">get<\/span><span class=\"p\">),<\/span>\n    <span class=\"s\">'https:\/\/sobolevn.me'<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">page_size<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">anyio<\/span><span class=\"p\">.<\/span><span class=\"n\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">page_size<\/span><span class=\"p\">.<\/span><span class=\"n\">awaitable<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># =&gt; &lt;FutureResult: &lt;coroutine object async_map at 0x10b17c320&gt;&gt;\n# =&gt; &lt;IOResult: &lt;Success: 27972&gt;&gt;\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>Notice, that we have exactly the same result,<br>\nbut now our code works asynchronously.<br>\nAnd its core part didn't change at all!<\/p>\n\n<p>However, it has some important notes:<\/p>\n\n<ol>\n<li>We changed sync <code>IOResultE<\/code> into async <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/future.html\"><code>FutureResultE<\/code><\/a> and <code>impure_safe<\/code> to <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/future.html#future-safe\"><code>future_safe<\/code><\/a>, which does the same thing but returns another abstraction: <code>FutureResultE<\/code>\n<\/li>\n<li>We now also use <code>AsyncClient<\/code> from <code>httpx<\/code>\n<\/li>\n<li>We are also required to run the resulting <code>FutureResult<\/code> value. Because red functions cannot run themselves! To demonstrate that this approach works with any async library (<code>asyncio<\/code>, <code>trio<\/code>, <code>curio<\/code>), I am using <a href=\"https:\/\/github.com\/agronholm\/anyio\"><code>anyio<\/code><\/a> utility<\/li>\n<\/ol>\n<h3>\n  \n  \n  Combining the two\n<\/h3>\n\n<p>And now I can show you how you can combine<br>\nthese two versions into a single type-safe API.<\/p>\n\n<p>Sadly, <a href=\"https:\/\/github.com\/dry-python\/returns\/issues\/408\">Higher Kinded Types<\/a> and proper <a href=\"https:\/\/github.com\/dry-python\/classes\">type-classes<\/a> are work-in-progress, so we would use regular <code>@overload<\/code> function cases:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">,<\/span> <span class=\"n\">Union<\/span><span class=\"p\">,<\/span> <span class=\"n\">overload<\/span>\n\n<span class=\"kn\">import<\/span> <span class=\"nn\">anyio<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">httpx<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.future<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">,<\/span> <span class=\"n\">future_safe<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.io<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">,<\/span> <span class=\"n\">impure_safe<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">overload<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">[<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"s\">\"\"\"Sync case.\"\"\"<\/span>\n\n<span class=\"o\">@<\/span><span class=\"n\">overload<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">[<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"s\">\"\"\"Async case.\"\"\"<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">client_get<\/span><span class=\"p\">:<\/span> <span class=\"n\">Union<\/span><span class=\"p\">[<\/span>\n        <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">IOResultE<\/span><span class=\"p\">[<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n        <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">[<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">Response<\/span><span class=\"p\">]],<\/span>\n    <span class=\"p\">],<\/span>\n    <span class=\"n\">url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Union<\/span><span class=\"p\">[<\/span><span class=\"n\">IOResultE<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">],<\/span> <span class=\"n\">FutureResultE<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]]:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">client_get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">).<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span>\n        <span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">content<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>With <code>@overload<\/code> decorators we describe which combinations of inputs are allowed. And what return type will they produce. You can read more about <code>@overload<\/code> decorator <a href=\"https:\/\/sobolevn.me\/2019\/01\/simple-dependent-types-in-python\">here<\/a>.<\/p>\n\n<p>Finally, calling our function with both sync and async client:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"c1\"># Sync:\n<\/span><span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">impure_safe<\/span><span class=\"p\">(<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">),<\/span>\n    <span class=\"s\">'https:\/\/sobolevn.me'<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">))<\/span>\n<span class=\"c1\"># =&gt; &lt;IOResult: &lt;Success: 27972&gt;&gt;\n<\/span>\n<span class=\"c1\"># Async:\n<\/span><span class=\"n\">page_size<\/span> <span class=\"o\">=<\/span> <span class=\"n\">fetch_resource_size<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">future_safe<\/span><span class=\"p\">(<\/span><span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"n\">AsyncClient<\/span><span class=\"p\">().<\/span><span class=\"n\">get<\/span><span class=\"p\">),<\/span>\n    <span class=\"s\">'https:\/\/sobolevn.me'<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">page_size<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">anyio<\/span><span class=\"p\">.<\/span><span class=\"n\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">page_size<\/span><span class=\"p\">.<\/span><span class=\"n\">awaitable<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># =&gt; &lt;FutureResult: &lt;coroutine object async_map at 0x10b17c320&gt;&gt;\n# =&gt; &lt;IOResult: &lt;Success: 27972&gt;&gt;\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>As you can see <code>fetch_resource_size<\/code> with sync client immediately returns <code>IOResult<\/code> and can execute itself.<br>\nIn contrast to it, async version requires some event-loop to execute it. Like a regular coroutine. We use <code>anyio<\/code> for the demo.<\/p>\n\n<p><code>mypy<\/code> is pretty happy about our code too:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight plaintext\"><code>\u00bb mypy async_and_sync.py\nSuccess: no issues found in 1 source file\n<\/code><\/pre><\/div>\n\n\n\n<p>Let's try to screw something up:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight diff\"><code><span class=\"gd\">---lambda response: len(response.content),\n<\/span><span class=\"gi\">+++lambda response: response.content,\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>And check that new error will be catched by <code>mypy<\/code>:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"err\">\u00bb<\/span> <span class=\"n\">mypy<\/span> <span class=\"n\">async_and_sync<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span>\n<span class=\"n\">async_and_sync<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span><span class=\"p\">:<\/span><span class=\"mi\">33<\/span><span class=\"p\">:<\/span> <span class=\"n\">error<\/span><span class=\"p\">:<\/span> <span class=\"n\">Argument<\/span> <span class=\"mi\">1<\/span> <span class=\"n\">to<\/span> <span class=\"s\">\"map\"<\/span> <span class=\"n\">of<\/span> <span class=\"s\">\"IOResult\"<\/span> <span class=\"n\">has<\/span> <span class=\"n\">incompatible<\/span> <span class=\"nb\">type<\/span> <span class=\"s\">\"Callable[[Response], bytes]\"<\/span><span class=\"p\">;<\/span> <span class=\"n\">expected<\/span> <span class=\"s\">\"Callable[[Response], int]\"<\/span>\n<span class=\"n\">async_and_sync<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span><span class=\"p\">:<\/span><span class=\"mi\">33<\/span><span class=\"p\">:<\/span> <span class=\"n\">error<\/span><span class=\"p\">:<\/span> <span class=\"n\">Argument<\/span> <span class=\"mi\">1<\/span> <span class=\"n\">to<\/span> <span class=\"s\">\"map\"<\/span> <span class=\"n\">of<\/span> <span class=\"s\">\"FutureResult\"<\/span> <span class=\"n\">has<\/span> <span class=\"n\">incompatible<\/span> <span class=\"nb\">type<\/span> <span class=\"s\">\"Callable[[Response], bytes]\"<\/span><span class=\"p\">;<\/span> <span class=\"n\">expected<\/span> <span class=\"s\">\"Callable[[Response], int]\"<\/span>\n<span class=\"n\">async_and_sync<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span><span class=\"p\">:<\/span><span class=\"mi\">33<\/span><span class=\"p\">:<\/span> <span class=\"n\">error<\/span><span class=\"p\">:<\/span> <span class=\"n\">Incompatible<\/span> <span class=\"k\">return<\/span> <span class=\"n\">value<\/span> <span class=\"nb\">type<\/span> <span class=\"p\">(<\/span><span class=\"n\">got<\/span> <span class=\"s\">\"bytes\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">expected<\/span> <span class=\"s\">\"int\"<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>As you can see, there's nothing magical in a way how async code can be written with right abstractions. Inside our implementation, there's still no magic. Just good old composition. What we real magic we do is providing the same API for different types - this allows us to abstract away how, for example, HTTP requests work: synchronously or asynchronously.<\/p>\n\n<p>I hope, that this quick demo shows how awesome <code>async<\/code> programs can be!<br>\nFeel free to try new <a href=\"https:\/\/github.com\/dry-python\/returns\/releases\/tag\/0.14.0\"><code>dry-python\/returns@0.14<\/code><\/a> release, it has lots of other goodies!<\/p>\n<h2>\n  \n  \n  Other awesome features\n<\/h2>\n\n<p>Speaking about goodies, I want to highlight several of features I am most proud of:<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/curry.html\">Typed <code>partial<\/code> and <code>@curry<\/code><\/a> functions\n<\/li>\n<\/ul>\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">returns.curry<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">curry<\/span><span class=\"p\">,<\/span> <span class=\"n\">partial<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">example<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n    <span class=\"p\">...<\/span>\n\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">partial<\/span><span class=\"p\">(<\/span><span class=\"n\">example<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># note: Revealed type is 'def (b: builtins.str) -&gt; builtins.float'\n<\/span>\n<span class=\"n\">reveal_type<\/span><span class=\"p\">(<\/span><span class=\"n\">curry<\/span><span class=\"p\">(<\/span><span class=\"n\">example<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># note: Revealed type is 'Overload(def (a: builtins.int) -&gt; def (b: builtins.str) -&gt; builtins.float, def (a: builtins.int, b: builtins.str) -&gt; builtins.float)'\n<\/span><\/code><\/pre><\/div>\n\n\n<p>Which means, that you can use <code>@curry<\/code> like so:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">curry<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">example<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">float<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span> <span class=\"o\">+<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">b<\/span><span class=\"p\">))<\/span>\n\n<span class=\"k\">assert<\/span> <span class=\"n\">example<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"s\">'abc'<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mf\">4.0<\/span>\n<span class=\"k\">assert<\/span> <span class=\"n\">example<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)(<\/span><span class=\"s\">'abc'<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mf\">4.0<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<ul>\n<li><a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/pipeline.html\">Functional pipelines with type-inference<\/a><\/li>\n<\/ul>\n\n<p>You can now use functional pipelines with full type inference that is augmentated by a custom <code>mypy<\/code> plugin:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">returns.pipeline<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">flow<\/span>\n<span class=\"k\">assert<\/span> <span class=\"n\">flow<\/span><span class=\"p\">(<\/span>\n    <span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">],<\/span>\n    <span class=\"k\">lambda<\/span> <span class=\"n\">collection<\/span><span class=\"p\">:<\/span> <span class=\"nb\">max<\/span><span class=\"p\">(<\/span><span class=\"n\">collection<\/span><span class=\"p\">),<\/span>\n    <span class=\"k\">lambda<\/span> <span class=\"n\">max_number<\/span><span class=\"p\">:<\/span> <span class=\"o\">-<\/span><span class=\"n\">max_number<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"o\">-<\/span><span class=\"mi\">3<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>We all know how hard it is to work with <code>lambda<\/code>s in typed code because its arguments always have <code>Any<\/code> type. And this might break regular <code>mypy<\/code> inference.<\/p>\n\n<p>Now, we always know that <code>lambda collection: max(collection)<\/code> has <code>Callable[[List[int]], int]<\/code> type inside this pipeline. And <code>lambda max_number: -max_number<\/code> is just <code>Callable[[int], int]<\/code>. You can pass any number of arguments to <code>flow<\/code>, they all will work perfectly. Thanks to our custom plugin!<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/context.html#requirescontextfutureresult-container\"><code>RequiresContextFutureResult<\/code><\/a> for <a href=\"https:\/\/sobolevn.me\/2020\/02\/typed-functional-dependency-injection\">typed functional dependency injection<\/a>\n<\/li>\n<\/ul>\n\n<p>It is an abstraction over <code>FutureResult<\/code> we have already covered in this article.<br>\nIt might be used to explicitly pass dependencies in a functional manner in your async programs.<\/p>\n\n<h2>\n  \n  \n  To be done\n<\/h2>\n\n<p>However, there are more things to be done before we can hit <code>1.0<\/code>:<\/p>\n\n<ol>\n<li>We need to implement Higher Kinded Types or their emulation, <a href=\"https:\/\/github.com\/dry-python\/returns\/issues\/408\">source<\/a>\n<\/li>\n<li>Adding proper type-classes, so we can implement needed abstractions, <a href=\"https:\/\/github.com\/dry-python\/classes\">source<\/a>\n<\/li>\n<li>We would love to try the <code>mypyc<\/code> compiler. It will potentially allow us to compile our typed-annotated Python program to binary. And as a result, simply dropping in <code>dry-python\/returns<\/code> into your program will make it several times faster, <a href=\"https:\/\/github.com\/dry-python\/returns\/issues\/398\">source<\/a>\n<\/li>\n<li>Finding new ways to write functional Python code, like our on-going investigation of <a href=\"https:\/\/github.com\/dry-python\/returns\/issues\/392\">\"do-notation\"<\/a>\n<\/li>\n<\/ol>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>Composition and abstraction can solve any problem. In this article, I have shown you how they can solve a problem of function colors and allow people to write simple, readable, and still flexible code that works. And type-checks.<\/p>\n\n<p>Check out <a href=\"https:\/\/github.com\/dry-python\/returns\"><code>dry-python\/returns<\/code><\/a>, provide your feedback, learn new ideas, maybe even <a href=\"https:\/\/github.com\/sponsors\/dry-python\/\">help us to sustain this project<\/a>!<\/p>\n\n<p>And as always, <a href=\"https:\/\/github.com\/sobolevn\/\">follow me on GitHub<\/a> to keep up with new awesome Python libraries I make or help with!<\/p>\n\n<h2>\n  \n  \n  Gratis\n<\/h2>\n\n<p>Thanks for reviewing this post to <a href=\"https:\/\/github.com\/AlwxSin\">@AlwxSin<\/a> and <a href=\"https:\/\/github.com\/ditansu\">@ditansu<\/a>.<\/p>\n\n","category":["python","async","webdev","django"]},{"title":"9 best open-source findings, March 2020","pubDate":"Tue, 07 Apr 2020 14:10:24 +0000","link":"https:\/\/dev.to\/sobolevn\/9-best-open-source-findings-march-2020-3ipa","guid":"https:\/\/dev.to\/sobolevn\/9-best-open-source-findings-march-2020-3ipa","description":"<p>Hello, everyone!<\/p>\n\n<p>Let me introduce a list of the best open-source findings for March 2020.<\/p>\n\n<p>If you want to have more awesomeness, including new and useful open-source tools, great articles, and excellent talks - you can join my telegram channel called <a href=\"https:\/\/t.me\/opensource_findings\">@OpensourceFindings<\/a> (<a href=\"https:\/\/tlg.name\/opensource_findings\">mirror link<\/a>).<\/p>\n\n<p>In this list we will discuss: Rust, TypeScript, JavaScript, Go, and Python.<br>\nThis includes web development, devops and QA tooling, documentation, and DX improvements.<\/p>\n<h2>\n  \n  \n  glitch-this\n<\/h2>\n\n<p>A commandline tool + python library to glitchify images and even make GIFs out of them!<\/p>\n\n<p>Written in Python.<\/p>\n\n<p><a href=\"https:\/\/github.com\/TotallyNotChase\/glitch-this\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--XFk-BxV4--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/i_\/ya\/om\/i_yaom-er8tm9bmnyiohv-wuauu.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--XFk-BxV4--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/i_\/ya\/om\/i_yaom-er8tm9bmnyiohv-wuauu.gif\" alt=\"glitch-this\"><\/a><\/p>\n<h2>\n  \n  \n  k9s\n<\/h2>\n\n<p>Kubernetes CLI To Manage Your Clusters In Style!<\/p>\n\n<p>Written in Go.<\/p>\n\n<p><a href=\"https:\/\/github.com\/derailed\/k9s\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--Hk2oWjMl--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/bt\/yu\/qp\/btyuqplfdeaqswfpvufzupc7xfg.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--Hk2oWjMl--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/bt\/yu\/qp\/btyuqplfdeaqswfpvufzupc7xfg.png\" alt=\"k9s\"><\/a><\/p>\n<h2>\n  \n  \n  gqless\n<\/h2>\n\n<p>A GraphQL client without queries \u2728<\/p>\n\n<p>Written in TypeScript.<\/p>\n\n<p><a href=\"https:\/\/github.com\/samdenty\/gqless\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--uQnHoGKs--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/ap\/ly\/vg\/aplyvgvhptlfzqyybumhnw2get8.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--uQnHoGKs--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/ap\/ly\/vg\/aplyvgvhptlfzqyybumhnw2get8.gif\" alt=\"gqless\"><\/a><\/p>\n<h2>\n  \n  \n  django-schema-graph\n<\/h2>\n\n<p>Django-schema-graph makes a colourful diagram out of your Django models. The diagram is interactive, and makes it easy to toggle models and apps on\/off at will.<\/p>\n\n<p>Written in Python.<\/p>\n\n<p><a href=\"https:\/\/github.com\/meshy\/django-schema-graph\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--IVZRIhKj--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/km\/w1\/ya\/kmw1yafz6w-8yycl_ggeyj2psgi.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--IVZRIhKj--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/km\/w1\/ya\/kmw1yafz6w-8yycl_ggeyj2psgi.png\" alt=\"django-schema-graph\"><\/a><\/p>\n<h2>\n  \n  \n  misspell-fixer-action\n<\/h2>\n\n<p>Github Action (<a href=\"https:\/\/sobolevn.me\/talks\/devoops-2019\">what is Github Action?<\/a>) that automatically fixes typos and mistakes in your source code and docs! <a href=\"https:\/\/github.com\/wemake-services\/wemake-python-styleguide\/pull\/1272\">Example PR<\/a>.<\/p>\n\n<p>Written in Shell.<\/p>\n\n<p><a href=\"https:\/\/github.com\/sobolevn\/misspell-fixer-action\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--LfvgC-5U--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/mg\/ym\/o-\/mgymo-ikp0mrzm4nxcjk9ror-8y.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--LfvgC-5U--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/mg\/ym\/o-\/mgymo-ikp0mrzm4nxcjk9ror-8y.png\" alt=\"misspell-fixer-action\"><\/a><\/p>\n<h2>\n  \n  \n  schemathesis\n<\/h2>\n\n<p>A tool that generates and runs test cases for Open API \/ Swagger based apps. It uses property-based tests inside.<\/p>\n\n<p>Written in Python. But, it works with apps written in any languages as long as they do have valid <code>swagger.json<\/code><\/p>\n\n<p><a href=\"https:\/\/github.com\/kiwicom\/schemathesis\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--HO4y7U1J--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/yp\/wb\/f3\/ypwbf3ghnmyvyipzntp2uipjk4y.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--HO4y7U1J--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/yp\/wb\/f3\/ypwbf3ghnmyvyipzntp2uipjk4y.gif\" alt=\"schemathesis\"><\/a><\/p>\n<h2>\n  \n  \n  kmon\n<\/h2>\n\n<p>Linux Kernel Manager and Activity Monitor \ud83d\udc27\ud83d\udcbb<\/p>\n\n<p>Written in Rust.<\/p>\n\n<p><a href=\"https:\/\/github.com\/orhun\/kmon\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--WD2NOoKY--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/6k\/eo\/hc\/6keohcj34uf67fwxx57dvdmi_ws.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--WD2NOoKY--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/6k\/eo\/hc\/6keohcj34uf67fwxx57dvdmi_ws.gif\" alt=\"kmon\"><\/a><\/p>\n<h2>\n  \n  \n  napkin\n<\/h2>\n\n<p>Python as DSL for writing PlantUML sequence diagrams.<\/p>\n\n<p>Written in Python.<\/p>\n\n<p><a href=\"https:\/\/github.com\/pinetr2e\/napkin\">Link<\/a><\/p>\n\n<p>Turns this code:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">distributed_control<\/span><span class=\"p\">(<\/span><span class=\"n\">c<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">user<\/span> <span class=\"o\">=<\/span> <span class=\"n\">c<\/span><span class=\"p\">.<\/span><span class=\"nb\">object<\/span><span class=\"p\">(<\/span><span class=\"s\">'User'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">order<\/span> <span class=\"o\">=<\/span> <span class=\"n\">c<\/span><span class=\"p\">.<\/span><span class=\"nb\">object<\/span><span class=\"p\">(<\/span><span class=\"s\">'Order'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">orderLine<\/span> <span class=\"o\">=<\/span> <span class=\"n\">c<\/span><span class=\"p\">.<\/span><span class=\"nb\">object<\/span><span class=\"p\">(<\/span><span class=\"s\">'OrderLine'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">product<\/span> <span class=\"o\">=<\/span> <span class=\"n\">c<\/span><span class=\"p\">.<\/span><span class=\"nb\">object<\/span><span class=\"p\">(<\/span><span class=\"s\">'Product'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">customer<\/span> <span class=\"o\">=<\/span> <span class=\"n\">c<\/span><span class=\"p\">.<\/span><span class=\"nb\">object<\/span><span class=\"p\">(<\/span><span class=\"s\">'Customer'<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">with<\/span> <span class=\"n\">user<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">with<\/span> <span class=\"n\">order<\/span><span class=\"p\">.<\/span><span class=\"n\">calculatePrice<\/span><span class=\"p\">():<\/span>\n            <span class=\"k\">with<\/span> <span class=\"n\">orderLine<\/span><span class=\"p\">.<\/span><span class=\"n\">calculatePrice<\/span><span class=\"p\">():<\/span>\n                <span class=\"n\">product<\/span><span class=\"p\">.<\/span><span class=\"n\">getPrice<\/span><span class=\"p\">(<\/span><span class=\"s\">'quantity:number'<\/span><span class=\"p\">)<\/span>\n                <span class=\"k\">with<\/span> <span class=\"n\">customer<\/span><span class=\"p\">.<\/span><span class=\"n\">getDiscountedValue<\/span><span class=\"p\">(<\/span><span class=\"n\">order<\/span><span class=\"p\">):<\/span>\n                    <span class=\"n\">order<\/span><span class=\"p\">.<\/span><span class=\"n\">getBaseValue<\/span><span class=\"p\">().<\/span><span class=\"n\">ret<\/span><span class=\"p\">(<\/span><span class=\"s\">'value'<\/span><span class=\"p\">)<\/span>\n                    <span class=\"n\">c<\/span><span class=\"p\">.<\/span><span class=\"n\">ret<\/span><span class=\"p\">(<\/span><span class=\"s\">'discountedValue'<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Into this diagram:<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--nAUyN1g9--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/da\/en\/dd\/daenddh41sevmdzmohafzjzuyqm.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--nAUyN1g9--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/da\/en\/dd\/daenddh41sevmdzmohafzjzuyqm.jpeg\" alt=\"napkin\"><\/a><\/p>\n\n<h2>\n  \n  \n  explainshell\n<\/h2>\n\n<p>Web app to explain to you your Shell commands. Very helpful!<\/p>\n\n<p>Written in Python and JavaScript.<\/p>\n\n<p><a href=\"https:\/\/explainshell.com\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--v0qz4SXq--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/40\/69\/iy\/4069iy0aco5ogsmovfwqibdcqdq.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--v0qz4SXq--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/40\/69\/iy\/4069iy0aco5ogsmovfwqibdcqdq.jpeg\" alt=\"explainshell\"><\/a><\/p>\n\n<h2>\n  \n  \n  Bonus\n<\/h2>\n\n<p><a href=\"https:\/\/opensource.builders\/\">opensource.builders<\/a>: different open-source alternatives for corporate tools.<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--mv8BMEiC--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/cp\/23\/qi\/cp23qibtgwruvwxlr1mnkgflcbi.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--mv8BMEiC--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/cp\/23\/qi\/cp23qibtgwruvwxlr1mnkgflcbi.jpeg\" alt=\"opensource.builders\"><\/a><\/p>\n\n<p>That's it for today! Stay safe. And make sure to <a href=\"https:\/\/t.me\/opensource_findings\">subscribe<\/a> to my telegram channel if you liked this list. I am going to post new ones each month. Also, feel free to post any cool projects you know in the comments. Feedback is always welcome.<\/p>\n\n<p>You can also <a href=\"https:\/\/github.com\/sobolevn\">follow me on Github<\/a> to see how new instruments are made, and what you can help with! It is a great start if you are new to open-source.<\/p>\n\n","category":["python","javascript","webdev","go"]},{"title":"Do not log","pubDate":"Wed, 25 Mar 2020 10:32:17 +0000","link":"https:\/\/dev.to\/wemake-services\/do-not-log-1k4","guid":"https:\/\/dev.to\/wemake-services\/do-not-log-1k4","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2020\/03\/do-not-log\">https:\/\/sobolevn.me\/2020\/03\/do-not-log<\/a><\/p>\n\n<p>Almost every week I accidentally get into this logging argument. Here's the problem: people tend to log different things and call it a best-practice. And I am not sure why. When I start discussing this with other people I always end up repeating the exact same ideas over and over again.<\/p>\n\n<p>So. Today I want to criticize the whole logging culture and provide a bunch of alternatives.<\/p>\n\n<h2>\n  \n  \n  Logging does not make sense\n<\/h2>\n\n<p>Let's start with the most important one. Logging does not make any sense!<\/p>\n\n<p>Let's review a popular code sample that you can find all across the internet:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">do_something_complex<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">except<\/span> <span class=\"n\">MyException<\/span> <span class=\"k\">as<\/span> <span class=\"n\">ex<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">logger<\/span><span class=\"p\">.<\/span><span class=\"n\">log<\/span><span class=\"p\">(<\/span><span class=\"n\">ex<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>So, what is going on here? Some complex computation fails and we log it. It seems like a good thing to do, doesn't it? Well. In this situation, I usually ask several important questions.<\/p>\n\n<p><em>The first question is<\/em>: can we make bad states for <code>do_something_complex()<\/code> unreachable? If so, let's refactor our code to be type-safe. And eliminate all possible exceptions that can happen here with <code>mypy<\/code>. Sometimes this helps. And in this case, we would not have any exception handling and logging at all.<\/p>\n\n<p><em>The second question I ask<\/em>: is it important that <code>do_something_complex()<\/code> did fail? There are a lot of cases when we don't care. Because of the retries, queues, or it might be an optional step that can be skipped. If this failure is not important - then just forget about it. But, if this failure is important - I want to know exactly everything: why and when did it fail. I want to know the whole stack trace, values of local variables, execution context, the total number of failures, and the number of affected users. I also want to be immediately notified of this important failure. And to be able to create a bug ticket from this failure in one click.<\/p>\n\n<p>And yes, you got it correctly: it sounds like a job for Sentry, not logging.<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--a7VvI06e--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/lxkijg1u5ro8t1zz6x5u.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--a7VvI06e--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/lxkijg1u5ro8t1zz6x5u.png\" alt=\"Sentry\"><\/a><\/p>\n\n<p>I either want a quick notification about some critical error with everything inside, or I want nothing: a peaceful morning with tea and youtube videos. There's nothing in between for logging.<\/p>\n\n<p><em>The third question is<\/em>: can we instead apply business monitoring to make sure our app works? We don't really care about exceptions and how to handle them. We care about the business value that we provide with our app. Sometimes your app does not raise any exception to be caught be Sentry. It can be broken in different ways. For example, your form validation can return errors when it should not normally happen. And you have zero exceptions, but a dysfunctional application. That's where business monitoring shines!<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--6EUQ7trw--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/fnqmj0anyfm907cpze5p.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--6EUQ7trw--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/fnqmj0anyfm907cpze5p.png\" alt=\"PostHog\"><\/a><\/p>\n\n<p>We can track different business metrics and make sure there are new orders, new comments, new users, etc. And if not - I want an emergency notification. I don't want to waste extra money on reading logging information after angry clients will call or text me. Please, don't treat your users as a monitoring service!<\/p>\n\n<p><em>The last question is<\/em>: do you normally expect <code>do_something_complex()<\/code> to fail? Like HTTP calls or database access. If so, <a href=\"https:\/\/sobolevn.me\/2019\/02\/python-exceptions-considered-an-antipattern\">don't use exceptions<\/a>, use <a href=\"https:\/\/github.com\/dry-python\/returns\"><code>Result<\/code> monad<\/a> instead. This way you can clearly indicate that something is going to fail. And act with confidence. And do not log anything. Just let it fail.<\/p>\n\n<h2>\n  \n  \n  Logging is a side effect\n<\/h2>\n\n<p>One more important monad-related topic is the pureness of the function that has a logger call inside. Let's compare two similar functions:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">compute<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"p\">(<\/span><span class=\"n\">arg<\/span> <span class=\"o\">\/<\/span> <span class=\"n\">PI<\/span><span class=\"p\">)<\/span> <span class=\"o\">*<\/span> <span class=\"n\">MAGIC_NUMBER<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>And:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">compute<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span><span class=\"n\">arg<\/span> <span class=\"o\">\/<\/span> <span class=\"n\">PI<\/span><span class=\"p\">)<\/span> <span class=\"o\">*<\/span> <span class=\"n\">MAGIC_NUMBER<\/span>\n    <span class=\"n\">logger<\/span><span class=\"p\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"s\">'Computation is:'<\/span><span class=\"p\">,<\/span> <span class=\"n\">result<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">result<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>The main difference between these two functions is that the first one is a perfect pure function and the second one is <code>IO<\/code>-bound impure one.<\/p>\n\n<p>What consequences does it have?<\/p>\n\n<ol>\n<li>We have to change our return type to <code>IOResult[float]<\/code> because logging is impure and can fail (yes, loggers can fail and break your app)<\/li>\n<li>We need to test this side effect. And we need to remember to do so! Unless this side effect is explicit in the return type<\/li>\n<\/ol>\n\n<p>Wise programmers even use special <a href=\"http:\/\/learnhaskellforgood.narod.ru\/learnyouahaskell.com\/for-a-few-monads-more.html\"><code>Writer<\/code> monad<\/a> to make logging pure and explicit. And that requires to significantly change the whole architecture around this function.<\/p>\n\n<p>We also might want to pass the correct <code>logger<\/code> instance, which probably implies that we have to use <a href=\"https:\/\/sobolevn.me\/2020\/02\/typed-functional-dependency-injection\">dependency injection based on <code>RequiresContext<\/code><\/a> monad.<\/p>\n\n<p>All I want to say with all these abstractions is that proper logging architecture is hard. It is not just about writing <code>logger.log()<\/code> here and there. It is a complex process of creating proper abstractions, composing them, and maintaining strict layers of pure and impure code.<\/p>\n\n<p>Is your team ready for this?<\/p>\n\n<h2>\n  \n  \n  Logging is a subsystem\n<\/h2>\n\n<p>We used to log things into a single file. It was fun!<\/p>\n\n<p>Even this simple setup required us to do periodical log file rotation. But still, it was easy. We also used <code>grep<\/code> to find things in this file. But, even then we had problems. Do you happen to have several servers? Good luck finding any required information in all of these files and <code>ssh<\/code> connections.<\/p>\n\n<p>But, now it is not enough! With all these microservices, cloud-native, and other 2020-ish tools we need complex subsystems to work with logging. It includes (but is not limited to):<\/p>\n\n<ul>\n<li>ELK: <a href=\"https:\/\/www.elastic.co\/what-is\/elk-stack\">ElasticSearch + Logstash + Kibana<\/a>\n<\/li>\n<li><a href=\"https:\/\/grafana.com\/oss\/loki\/\">Graphana + Loki<\/a><\/li>\n<\/ul>\n\n<p>And here's <a href=\"https:\/\/github.com\/deviantony\/docker-elk\">an example<\/a> of what it takes to set this thing up with ELK stack:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight yaml\"><code><span class=\"na\">version<\/span><span class=\"pi\">:<\/span> <span class=\"s1\">'<\/span><span class=\"s\">3.2'<\/span>\n\n<span class=\"na\">services<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">elasticsearch<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">build<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">context<\/span><span class=\"pi\">:<\/span> <span class=\"s\">elasticsearch\/<\/span>\n      <span class=\"na\">args<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">ELK_VERSION<\/span><span class=\"pi\">:<\/span> <span class=\"s\">$ELK_VERSION<\/span>\n    <span class=\"na\">volumes<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">type<\/span><span class=\"pi\">:<\/span> <span class=\"s\">bind<\/span>\n        <span class=\"na\">source<\/span><span class=\"pi\">:<\/span> <span class=\"s\">.\/elasticsearch\/config\/elasticsearch.yml<\/span>\n        <span class=\"na\">target<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/usr\/share\/elasticsearch\/config\/elasticsearch.yml<\/span>\n        <span class=\"na\">read_only<\/span><span class=\"pi\">:<\/span> <span class=\"no\">true<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">type<\/span><span class=\"pi\">:<\/span> <span class=\"s\">volume<\/span>\n        <span class=\"na\">source<\/span><span class=\"pi\">:<\/span> <span class=\"s\">elasticsearch<\/span>\n        <span class=\"na\">target<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/usr\/share\/elasticsearch\/data<\/span>\n    <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">9200:9200\"<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">9300:9300\"<\/span>\n    <span class=\"na\">environment<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">ES_JAVA_OPTS<\/span><span class=\"pi\">:<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">-Xmx256m<\/span><span class=\"nv\"> <\/span><span class=\"s\">-Xms256m\"<\/span>\n      <span class=\"na\">ELASTIC_PASSWORD<\/span><span class=\"pi\">:<\/span> <span class=\"s\">changeme<\/span>\n      <span class=\"s\">discovery.type<\/span><span class=\"pi\">:<\/span> <span class=\"s\">single-node<\/span>\n    <span class=\"na\">networks<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s\">elk<\/span>\n\n  <span class=\"na\">logstash<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">build<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">context<\/span><span class=\"pi\">:<\/span> <span class=\"s\">logstash\/<\/span>\n      <span class=\"na\">args<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">ELK_VERSION<\/span><span class=\"pi\">:<\/span> <span class=\"s\">$ELK_VERSION<\/span>\n    <span class=\"na\">volumes<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">type<\/span><span class=\"pi\">:<\/span> <span class=\"s\">bind<\/span>\n        <span class=\"na\">source<\/span><span class=\"pi\">:<\/span> <span class=\"s\">.\/logstash\/config\/logstash.yml<\/span>\n        <span class=\"na\">target<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/usr\/share\/logstash\/config\/logstash.yml<\/span>\n        <span class=\"na\">read_only<\/span><span class=\"pi\">:<\/span> <span class=\"no\">true<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">type<\/span><span class=\"pi\">:<\/span> <span class=\"s\">bind<\/span>\n        <span class=\"na\">source<\/span><span class=\"pi\">:<\/span> <span class=\"s\">.\/logstash\/pipeline<\/span>\n        <span class=\"na\">target<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/usr\/share\/logstash\/pipeline<\/span>\n        <span class=\"na\">read_only<\/span><span class=\"pi\">:<\/span> <span class=\"no\">true<\/span>\n    <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">5000:5000\/tcp\"<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">5000:5000\/udp\"<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">9600:9600\"<\/span>\n    <span class=\"na\">environment<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">LS_JAVA_OPTS<\/span><span class=\"pi\">:<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">-Xmx256m<\/span><span class=\"nv\"> <\/span><span class=\"s\">-Xms256m\"<\/span>\n    <span class=\"na\">networks<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s\">elk<\/span>\n    <span class=\"na\">depends_on<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s\">elasticsearch<\/span>\n\n  <span class=\"na\">kibana<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">build<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">context<\/span><span class=\"pi\">:<\/span> <span class=\"s\">kibana\/<\/span>\n      <span class=\"na\">args<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">ELK_VERSION<\/span><span class=\"pi\">:<\/span> <span class=\"s\">$ELK_VERSION<\/span>\n    <span class=\"na\">volumes<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">type<\/span><span class=\"pi\">:<\/span> <span class=\"s\">bind<\/span>\n        <span class=\"na\">source<\/span><span class=\"pi\">:<\/span> <span class=\"s\">.\/kibana\/config\/kibana.yml<\/span>\n        <span class=\"na\">target<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/usr\/share\/kibana\/config\/kibana.yml<\/span>\n        <span class=\"na\">read_only<\/span><span class=\"pi\">:<\/span> <span class=\"no\">true<\/span>\n    <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">5601:5601\"<\/span>\n    <span class=\"na\">networks<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s\">elk<\/span>\n    <span class=\"na\">depends_on<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"s\">elasticsearch<\/span>\n\n<span class=\"na\">networks<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">elk<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">driver<\/span><span class=\"pi\">:<\/span> <span class=\"s\">bridge<\/span>\n\n<span class=\"na\">volumes<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">elasticsearch<\/span><span class=\"pi\">:<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Isn't it a bit too hard?! Look, I just want to write strings into <code>stdout<\/code>. And then store it somewhere.<\/p>\n\n<p>But no. You need a database. And a separate web-service. You also have to monitor your logging subsystem. And periodically update it, you also have to make sure that it is secure. And has enough resources. And everyone has access to it. And so on and so forth.<\/p>\n\n<p>Of course, there are cloud providers just for logging. It might be a good idea to consider using them.<\/p>\n\n<h2>\n  \n  \n  Logging is hard to manage\n<\/h2>\n\n<p>In case you are still using logging after all my previous arguments, you will find out that it requires a lot of discipline and tooling. There are several well-known problems:<\/p>\n\n<ol>\n<li><p>Logging should be very strict about its format. Do you remember that we are probably going to store our logs in a NoSQL database? Our logs need to be indexable. You would probably end up using <a href=\"https:\/\/github.com\/hynek\/structlog\"><code>structlog<\/code><\/a> or a similar solution. In my opinion, this should be the default<\/p><\/li>\n<li><p>The next thing to fight is levels. All developers will use their own ideas on what is critical and what's not. Unless you (as an architect) will write a <a href=\"https:\/\/stackoverflow.com\/questions\/7839565\/logging-levels-logback-rule-of-thumb-to-assign-log-levels\">clear policy<\/a> that will cover most of the cases. You might also need to review it carefully. Otherwise, your logging database might blow up with useless data<\/p><\/li>\n<li>\n<p>Your logging usage should be consistent!<br>\nAll people tend to write in their own style. There's <a href=\"https:\/\/github.com\/globality-corp\/flake8-logging-format\">a linter<\/a> for that! It will enforce:<br>\n<\/p>\n<pre class=\"highlight python\"><code><span class=\"n\">logger<\/span><span class=\"p\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span>\n    <span class=\"s\">'Hello {world}'<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">extra<\/span><span class=\"o\">=<\/span><span class=\"p\">{<\/span><span class=\"s\">'world'<\/span><span class=\"p\">:<\/span> <span class=\"s\">'Earth'<\/span><span class=\"p\">},<\/span>\n<span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n\n<p>Instead of:<br>\n<\/p>\n<pre class=\"highlight python\"><code><span class=\"n\">logger<\/span><span class=\"p\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span>\n    <span class=\"s\">'Hello {world}'<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">world<\/span><span class=\"o\">=<\/span><span class=\"s\">'Earth'<\/span><span class=\"p\">),<\/span>\n<span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/li>\n<\/ol>\n\n<p>And many other edge cases.<\/p>\n\n<ol>\n<li>Logging should be business-oriented.\nI usually see people using logging with a minimal amount of useful information. For example, if you are logging an invalid current object state: it is not enough! You need to do more: you need to show how this object got into this invalid state. There are different approaches to this problem. Some use simple solutions like <a href=\"https:\/\/github.com\/etianen\/django-reversion\">version history<\/a>, some people use <a href=\"https:\/\/github.com\/johnbywater\/eventsourcing\">EventSourcing<\/a> to compose their objects from changes. And some libraries log the entire execution context, logical steps that were taken, and changes made to the object. Like <a href=\"https:\/\/github.com\/dry-python\/stories\"><code>dry-python\/stories<\/code><\/a> (<a href=\"https:\/\/stories.readthedocs.io\/en\/latest\/debugging\/\">docs on logging<\/a>). And here's how the context looks like:\n<\/li>\n<\/ol>\n\n<div class=\"highlight\"><pre class=\"highlight plaintext\"><code>   ApplyPromoCode.apply\n     find_category\n     find_promo_code\n     check_expiration\n     calculate_discount (errored: TypeError)\n\n   Context:\n     category_id = 1024                # Story argument\n     category = &lt;example.Category&gt;     # Set by ApplyPromoCode.find_category\n     promo_code = &lt;example.PromoCode&gt;  # Set by ApplyPromoCode.find_promo_code\n<\/code><\/pre><\/div>\n\n\n\n<p>See? It contains the full representation of what's happened and how to recreate this error. Not just some random state information here and there. And you don't even have to call logger yourself. It will handled for you. By the way, it even has a <a href=\"https:\/\/stories.readthedocs.io\/en\/latest\/contrib\/sentry\/\">native Sentry integraion<\/a>, which is better in my opinion.<\/p>\n\n<ol>\n<li>You should pay attention to what you log.\nThere are <a href=\"https:\/\/logdna.com\/best-practices-for-gdpr-logging\/\">GDPR rules on logging<\/a> and specialized security audits for your logs. Common sense dictates that logging passwords, credit cards, emails, etc is not secure. But, sadly, common sense is not enough. This is a complex process to follow.<\/li>\n<\/ol>\n\n<p>There are other problems to manage as well. My point here is to show that you will need senior people to work on that: by creating policies, writing down processes, and setting up your logging toolchain.<\/p>\n\n<h2>\n  \n  \n  What to do instead?\n<\/h2>\n\n<p>Let's do a quick recap:<\/p>\n\n<ol>\n<li>Logging does not make much sense in monitoring and error tracking. Use better tools instead: like error and business monitorings with alerts<\/li>\n<li>Logging adds significant complexity to your architecture. And it requires more testing. Use architecture patterns that will make logging an explicit part of your contracts<\/li>\n<li>Logging is a whole infrastructure subsystem on its own. And quite a complex one. You will have to maintain it or to outsource this job to existing logging services<\/li>\n<li>Logging should be done right. And it is hard. You will have to use a lot of tooling. And you will have to mentor developers that are unaware of the problems we have just discussed<\/li>\n<\/ol>\n\n<p>Is logging worth it? You should make an informed decision based on this knowledge and your project requirements. In my opinion, it is not required for the most of the regular web apps.<\/p>\n\n<p>Please, get this right. I understand that logging can be really useful (and sometimes even the only source of useful information). Like for on-premise software, for example. Or for initial steps when your app is not fully functioning yet. It would be hard to understand what is going on without logging. I am fighting an \"overlogging\" culture. When logs are used just for no good reason. Because developers just do it without analyzing costs and tradeoffs.<\/p>\n\n<p>Are you joining my side?<\/p>\n\n","category":["python","beginners","devops","webdev"]},{"title":"Cover the tricky parts of your code!","pubDate":"Thu, 05 Mar 2020 15:17:30 +0000","link":"https:\/\/dev.to\/wemake-services\/cover-the-tricky-parts-of-your-code-24m6","guid":"https:\/\/dev.to\/wemake-services\/cover-the-tricky-parts-of-your-code-24m6","description":"<p>Recently I had to add <code>python3.8<\/code> for our Python linter (the strictest one in existence): <a href=\"https:\/\/github.com\/wemake-services\/wemake-python-styleguide\"><code>wemake-python-styleguide<\/code><\/a>. And during this straight-forward (at first look) task, I have found several problems with test coverage that were not solved in Python community at all.<\/p>\n\n<p>Let's dig into it.<\/p>\n\n<h2>\n  \n  \n  Environment-based logic\n<\/h2>\n\n<p>The thing about this update was that <code>python3.8<\/code> introduced a new API.<br>\nPreviously we had <code>visit_Num<\/code>, <code>visit_Str<\/code>, <code>visit_NameConstant<\/code> methods (we don't really care about what they do, only names are important), but now we have to use <code>visit_Constant<\/code> method instead.<\/p>\n\n<p>Well, we will have to write some compatibility layer in our app to support both new <code>python3.8<\/code>, previous <code>python3.6<\/code>, and <code>python3.7<\/code> releases.<\/p>\n\n<p>My first attempt was to create a <code>route_visit<\/code> function to do the correct thing for all releases. To make it work I also had to define <code>PY38<\/code> constant like so:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">sys<\/span>\n\n<span class=\"n\">PY38<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sys<\/span><span class=\"p\">.<\/span><span class=\"n\">version_info<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"p\">(<\/span><span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">8<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">if<\/span> <span class=\"n\">PY38<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">route_visit<\/span> <span class=\"o\">=<\/span> <span class=\"p\">...<\/span>  <span class=\"c1\"># new logic: 3.8+\n<\/span><span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">route_visit<\/span> <span class=\"o\">=<\/span> <span class=\"p\">...<\/span>  <span class=\"c1\"># old logic: 3.6, 3.7\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>And it worked pretty well. I was able to run my tests successfully. The only thing broken was coverage. We use <code>pytest<\/code> and <code>pytest-cov<\/code> to measure coverage in our apps, we also enforce <code>--cov-branch<\/code> and <code>--cov-fail-under=100<\/code> policies. Which enforce us to cover all our code and all branches inside it.<\/p>\n\n<p>Here's the problem with my solution: I was either covering <code>if PY38:<\/code> branch on <code>python3.8<\/code> build or <code>else:<\/code> branch on other releases. I was never covering 100% of my program. Because it is literally impossible.<\/p>\n\n<h2>\n  \n  \n  Common pitfalls\n<\/h2>\n\n<p>Open-source libraries usually face this problem. They are required to work with different python versions, 3rd party API changes, backward compatibility, etc. Here are some examples that you probably have already seen somewhere:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n    <span class=\"kn\">import<\/span> <span class=\"nn\">django<\/span>\n    <span class=\"n\">HAS_DJANGO<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">True<\/span>\n<span class=\"k\">except<\/span> <span class=\"nb\">ImportError<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">HAS_DJANGO<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">False<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Or this was a popular hack during <code>2<\/code> \/ <code>3<\/code> days:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">range_<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">xrange<\/span>  <span class=\"c1\"># python2\n<\/span><span class=\"k\">except<\/span> <span class=\"nb\">NameError<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">range_<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">range<\/span>  <span class=\"c1\"># python3\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>With all these examples in mind, one can be sure that 100% of coverage is not possible.<br>\nThe common scenario to still achieve a feeling of 100% coverage for these cases was:<\/p>\n\n<ol>\n<li>Using <a href=\"https:\/\/coverage.readthedocs.io\/en\/stable\/excluding.html\"><code># pragma: no cover<\/code><\/a> magic comment to exclude a single line or a whole block from coverage<\/li>\n<li>Or writing every compatibility related check in a special <code>compat.py<\/code> that were later <a href=\"https:\/\/coverage.readthedocs.io\/en\/stable\/source.html#execution\">ommited<\/a> from coverage<\/li>\n<\/ol>\n\n<p>Here's how the first way looks like:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">try<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># pragma: no cover\n<\/span>    <span class=\"kn\">import<\/span> <span class=\"nn\">django<\/span>\n    <span class=\"n\">HAS_DJANGO<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">True<\/span>\n<span class=\"k\">except<\/span> <span class=\"nb\">ImportError<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># pragma: no cover\n<\/span>    <span class=\"n\">HAS_DJANGO<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">False<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Let's be honest: these solutions are dirty hacks. But, they do work. And I personally used both of them countless times in my life.<\/p>\n\n<p>Here's the interesting thing: aren't we supposed to test these complex integrational parts with the most precision and observability? Because that's where our application breaks the most: integration parts. And currently, we are just ignoring them from coverage and pretending that this problem does not exist.<\/p>\n\n<p>And for this reason, this time I felt like I am not going to simply exclude my compatibility logic. I got an idea for a new project.<\/p>\n\n<h2>\n  \n  \n  Conditional coverage\n<\/h2>\n\n<p>My idea was that <code># pragma<\/code> comments can have more information inside them. Not just <code>no cover<\/code>, but <code>no cover when?<\/code>. That's how <a href=\"https:\/\/github.com\/wemake-services\/coverage-conditional-plugin\"><code>coverage-conditional-plugin<\/code><\/a> was born. Let's use it and see how it works!<\/p>\n\n<p>First, we would need to install it:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight shell\"><code>pip <span class=\"nb\">install <\/span>coverage-conditional-plugin  <span class=\"c\"># pip works, but I prefer poetry<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>And then we would have to <a href=\"https:\/\/coverage.readthedocs.io\/en\/coverage-5.0.3\/config.html\">configure<\/a> <code>coverage<\/code> and the plugin itself:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight yaml\"><code><span class=\"pi\">[<\/span><span class=\"nv\">coverage<\/span><span class=\"pi\">:<\/span><span class=\"nv\">run<\/span><span class=\"pi\">]<\/span>\n<span class=\"c1\"># Here we specify plugins for coverage to be used:<\/span>\n<span class=\"s\">plugins =<\/span>\n  <span class=\"s\">coverage_conditional_plugin<\/span>\n\n<span class=\"pi\">[<\/span><span class=\"nv\">coverage<\/span><span class=\"pi\">:<\/span><span class=\"nv\">coverage_conditional_plugin<\/span><span class=\"pi\">]<\/span>\n<span class=\"c1\"># Here we specify our pragma rules:<\/span>\n<span class=\"s\">rules =<\/span> <span class=\"c1\"># we are going to define them later.<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Notice this <code>rules<\/code> key. It is the most important thing here. The rule (in this context) is some predicate that tells: should we include lines behind this specific <code>pragma<\/code> in our coverage or not. Here are some examples:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight yaml\"><code><span class=\"pi\">[<\/span><span class=\"nv\">coverage<\/span><span class=\"pi\">:<\/span><span class=\"nv\">coverage_conditional_plugin<\/span><span class=\"pi\">]<\/span>\n<span class=\"c1\"># Here we specify our pragma rules:<\/span>\n<span class=\"s\">rules =<\/span>\n  <span class=\"s\">\"sys_version_info &gt;= (3, 8)\"<\/span><span class=\"pi\">:<\/span> <span class=\"s\">py-gte-38<\/span>\n  <span class=\"s\">\"sys_version_info &lt; (3, 8)\"<\/span><span class=\"pi\">:<\/span> <span class=\"s\">py-lt-38<\/span>\n  <span class=\"s\">\"is_installed('django')\"<\/span><span class=\"pi\">:<\/span> <span class=\"s\">has-django<\/span>\n  <span class=\"s\">\"not is_installed('django')\"<\/span><span class=\"pi\">:<\/span> <span class=\"s\">has-no-django<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>It is pretty clear what we are doing here: we are defining pairs of predicates to include this code if some condition is true and another code in the opposite case.<\/p>\n\n<p>Here's how our previous examples would look like with these magic comments:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">sys<\/span>\n\n<span class=\"n\">PY38<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sys<\/span><span class=\"p\">.<\/span><span class=\"n\">version_info<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"p\">(<\/span><span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">8<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">if<\/span> <span class=\"n\">PY38<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># pragma: py-lt-38\n<\/span>    <span class=\"n\">route_visit<\/span> <span class=\"o\">=<\/span> <span class=\"p\">...<\/span>  <span class=\"c1\"># new logic: 3.8+\n<\/span><span class=\"k\">else<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># pragma: py-gte-38\n<\/span>    <span class=\"n\">route_visit<\/span> <span class=\"o\">=<\/span> <span class=\"p\">...<\/span>  <span class=\"c1\"># old logic: 3.6, 3.7\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>What does it say? If we are running on <code>py-lt-38<\/code> ignore <code>if PY38:<\/code> part. But, cover <code>else:<\/code> case. Because it is going to be executed and we know it. And we need to know how good did we cover it. On the other hand, if we are running on <code>py-gte-38<\/code> then cover <code>if PY38:<\/code> case and leave <code>else:<\/code> alone.<\/p>\n\n<p>And we can test that everything works correctly. Let's add some nonsense into our <code>PY38<\/code> branch to see if it is going to be covered by <code>python3.8<\/code> build:<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--KE9A1gP1--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/ul5t9zxjc5omzvu16qxn.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--KE9A1gP1--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/ul5t9zxjc5omzvu16qxn.png\" alt=\"PY38 Covered\"><\/a><\/p>\n\n<p>As we can see: green signs show which lines were fully covered, the yellow line indicates that branch coverage was not full, and the red line indicates that the line was not covered at all. And here's an example of grey or ignored lines under the opposed condition:<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--Vt86PLCJ--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/ebcok1i4wgmxftez0sjy.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--Vt86PLCJ--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/ebcok1i4wgmxftez0sjy.png\" alt=\"py-lt-38 ignored\"><\/a><\/p>\n\n<p><a href=\"https:\/\/github.com\/wemake-services\/wemake-python-styleguide\/blob\/master\/wemake_python_styleguide\/compat\/routing.py\">Here<\/a> you can find the full real-life source code for this sample.<\/p>\n\n<p>And here's one more example with <code>django<\/code> to show you how external packages can be handled:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">try<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># pragma: has-no-django\n<\/span>    <span class=\"kn\">import<\/span> <span class=\"nn\">django<\/span>\n    <span class=\"n\">HAS_DJANGO<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">True<\/span>\n<span class=\"k\">except<\/span> <span class=\"nb\">ImportError<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># pragma: has-django\n<\/span>    <span class=\"n\">HAS_DJANGO<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">False<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>We use the same logic here. Do we have <code>django<\/code> installed during tests (we have a little helper function <code>is_installed<\/code> to tell us)? If so, cover <code>try:<\/code>. If not, cover <code>except ImportError:<\/code> branch. But always cover <em>something<\/em>.<\/p>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>I hope you got the idea. Conditional coverage allows you to add or ignore lines based on predicates and collecting required bits of coverage from every run, not just ignoring complex conditions and keeping our eyes wide shut. Remember, that the code we need to cover the most!<\/p>\n\n<p>By the way, we have all kinds of helpers to query your environment:<\/p>\n\n<ul>\n<li>\n<code>os_environ<\/code> for env variables<\/li>\n<li>\n<code>patform_release<\/code> and <code>platform_version<\/code> for OS-based values<\/li>\n<li>\n<code>pkg_version<\/code> that returns package version information (as its name says)<\/li>\n<li>and many others!<\/li>\n<\/ul>\n\n<p>This little plugin is going to be really helpful for library authors that have to deal with compatibility and unfixed environments. And <code>coverage-conditional-plugin<\/code> will surely <em>cover<\/em> their backs! <a href=\"https:\/\/github.com\/wemake-services\/coverage-conditional-plugin\">Give it a star<\/a> on Github if you like this idea. And read the project docs to learn more.<\/p>\n\n","category":["python","showdev","testing","webdev"]},{"title":"Typed functional Dependency Injection in Python","pubDate":"Mon, 17 Feb 2020 08:57:07 +0000","link":"https:\/\/dev.to\/wemake-services\/typed-functional-dependency-injection-in-python-4e7b","guid":"https:\/\/dev.to\/wemake-services\/typed-functional-dependency-injection-in-python-4e7b","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2020\/02\/typed-functional-dependency-injection\">https:\/\/sobolevn.me\/2020\/02\/typed-functional-dependency-injection<\/a><\/p>\n\n<p>Dependency injection is a controversial topic. There are known problems, hacks, and even whole methodologies on how to work with DI frameworks.<\/p>\n\n<p>A lot of people asked me: how can you use a lot of typed functional concepts together with <a href=\"https:\/\/sobolevn.me\/2019\/03\/enforcing-srp\">traditional object-oriented dependency injection<\/a>?<\/p>\n\n<p>And this question makes a lot of sense. Because functional programming is all about composition. And dependency injection is just magic. You get some almost random objects injected into some places of your code. Moreover, the whole container to be injected later is magically assembled from the separate parts at runtime.<\/p>\n\n<p>This leads to a serious contradiction between functional declarative style and \"OMG, where did this class come from?\"<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--nIFoegRm--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/it0fk0jbqphcgecnpbq7.jpg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--nIFoegRm--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/it0fk0jbqphcgecnpbq7.jpg\" alt=\"DI meme\"><\/a><\/p>\n\n<p>Today we are going to solve this problem with a good-old functional way.<\/p>\n\n<h2>\n  \n  \n  Regular functions\n<\/h2>\n\n<p>Imagine that you have a <code>django<\/code> based game, where you award users with points for each guessed letter in a word (unguessed letters are marked as <code>'.'<\/code>):<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">django.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">,<\/span> <span class=\"n\">HttpResponse<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">words_app.logic<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">calculate_points<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">:<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">user_word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">=<\/span> <span class=\"n\">request<\/span><span class=\"p\">.<\/span><span class=\"n\">POST<\/span><span class=\"p\">[<\/span><span class=\"s\">'word'<\/span><span class=\"p\">]<\/span>  <span class=\"c1\"># just an example\n<\/span>    <span class=\"n\">points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">user_word<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">...<\/span>  <span class=\"c1\"># later you show the result to user somehow\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>And here's your business logic:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"c1\"># Somewhere in your `words_app\/logic.py`:\n<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">guessed_letters_count<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">len<\/span><span class=\"p\">([<\/span><span class=\"n\">letter<\/span> <span class=\"k\">for<\/span> <span class=\"n\">letter<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">word<\/span> <span class=\"k\">if<\/span> <span class=\"n\">letter<\/span> <span class=\"o\">!=<\/span> <span class=\"s\">'.'<\/span><span class=\"p\">])<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"mi\">0<\/span> <span class=\"k\">if<\/span> <span class=\"n\">guessed<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">5<\/span> <span class=\"k\">else<\/span> <span class=\"n\">guessed<\/span>  <span class=\"c1\"># minimum 6 points possible!\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>This is a pretty simple app: users try to guess some word and in the meantime, we award points for each guessed letter. But, we have a threshold. We only award 6 or more points for 6 or more guessed letters. That's it. We have our framework layer in place and our beautiful pure logic. This code is really simple to read and modify.<\/p>\n\n<p>Or is it? Let's try to modify it to find out. Let's say we want to make our game more challenging by making the points threshold configurable. How can we do that?<\/p>\n\n<h2>\n  \n  \n  Naive attempt\n<\/h2>\n\n<p>Ok, let's make <code>_award_points_for_letters<\/code> accept the second argument <code>threshold: int<\/code>:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">threshold<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"mi\">0<\/span> <span class=\"k\">if<\/span> <span class=\"n\">guessed<\/span> <span class=\"o\">&lt;<\/span> <span class=\"n\">threshold<\/span> <span class=\"k\">else<\/span> <span class=\"n\">guessed<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>And now your code won't type-check. Because that's how our caller looks like:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"c1\"># ...\n<\/span>    <span class=\"k\">return<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>To fix this <code>calculate_points<\/code> function (and all other upper caller functions) will have to accept <code>threshold: int<\/code> as a parameter and pass it to <code>_award_points_for_letters<\/code>, like so:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"n\">threshold<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"c1\"># ...\n<\/span>    <span class=\"k\">return<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">,<\/span> <span class=\"n\">threshold<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>It also affects our <code>views.py<\/code>:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">django.conf<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">settings<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">,<\/span> <span class=\"n\">HttpResponse<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">words_app.logic<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">calculate_points<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">:<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">user_word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">=<\/span> <span class=\"n\">request<\/span><span class=\"p\">.<\/span><span class=\"n\">POST<\/span><span class=\"p\">[<\/span><span class=\"s\">'word'<\/span><span class=\"p\">]<\/span>  <span class=\"c1\"># just an example\n<\/span>    <span class=\"n\">points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">user_word<\/span><span class=\"p\">,<\/span> <span class=\"n\">settings<\/span><span class=\"p\">.<\/span><span class=\"n\">WORD_THRESHOLD<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">...<\/span>  <span class=\"c1\"># later you show the result to user somehow\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>As we can see it is doable. But, it requires to change the whole call stack for a single parameter. Which leads us to some evident problems:<\/p>\n\n<ul>\n<li>If our call stack is big, we will have to modify lots of layers to achieve the place where we really need this value<\/li>\n<li>All our functions and classes will be infected with the values we pass. This would have several effects: you won't be able to understand which functions and classes really need passed values and which are just work as a transport layer (like <code>calculate_points<\/code> in our example). And of course, people would start using these passed values where they need them. And this will implicitly increase the coupling of your code and environment.<\/li>\n<li>It would be hard to compose your functions and classes together. This extra parameter will always get in the way. Arities of your functions would not match.<\/li>\n<\/ul>\n\n<p>All in all: this is working, but not a scalable solution. But, I would probably do this myself for a small app.<\/p>\n\n<h2>\n  \n  \n  Frameworks\n<\/h2>\n\n<p>A lot of Django people right now might ask: why not just importing settings and using it like <code>django<\/code> docs teaches us?<\/p>\n\n<p>Because this is <strong>ugly<\/strong>!<\/p>\n\n<p>For now, we have framework-independent business logic. It is a great thing. It is resistant to any framework-specific changes. And it also does not implicitly bring all the framework complexity with it. Let's remember how <code>django.conf.settings<\/code> works:<\/p>\n\n<ol>\n<li>We need to set <code>DJANGO_SETTINGS_MODULE<\/code> env variable to find the settings module<\/li>\n<li>Somewhere deep inside the framework <code>django.setup()<\/code> will happen<\/li>\n<li>\n<code>django<\/code> will import this module<\/li>\n<li>This module might contain <a href=\"https:\/\/sobolevn.me\/2017\/04\/managing-djangos-settings\">settings-dependent logic<\/a>\n<\/li>\n<li>Settings will also access environment, files, maybe even cloud providers for some specific values<\/li>\n<li>Then <code>django<\/code> will have almost-immutable singleton object that you can import from any place in your code<\/li>\n<li>You will have to mock settings in your tests to simply pass other value to your function. Or multiple (and many more) values if you heavily rely on settings module (which will probably be the case in any app)<\/li>\n<\/ol>\n\n<p>Is it worth it? Maybe. Sometimes <code>django.settings<\/code> is good enough. You can possibly have just a couple of primitive values to be injected. And this way allows to do it very fast.<\/p>\n\n<p>But, I would not recommend anyone to go this way if you really want to build a large app with clearly defined boundaries and comprehensive domain logic. I would even recommend using <a href=\"https:\/\/import-linter.readthedocs.io\/\"><code>import-linter<\/code><\/a> to forbid to import anything from <code>django<\/code> in your logic.<\/p>\n\n<p>So, how in a world I am going to pass this troublesome value into the function if nothing really works? Parameters are too noisy, containers are too magic and hard to work with, and global settings are just pure evil in the form of a singleton?<\/p>\n\n<h2>\n  \n  \n  Composition\n<\/h2>\n\n<p>Functional programmers are smart people. Really. They can do literally everything with just pure functions.<\/p>\n\n<p><a href=\"https:\/\/vimeo.com\/113588389\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--JHR7BdrI--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/dev-to-uploads.s3.amazonaws.com\/i\/e0kjtxmcsotbgj3plqok.gif\" alt=\"Functional patterns\"><\/a><\/p>\n\n<p>They have also solved the dependencies problem with elegance while maintaining simplicity. The core idea of dependency injection in a functional way is that we don't call things that rely on the context we don't have. Instead, we schedule them to be called later. Let's see how our original logic will change after adapting this idea:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing_extensions<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Protocol<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">_Deps<\/span><span class=\"p\">(<\/span><span class=\"n\">Protocol<\/span><span class=\"p\">):<\/span>  <span class=\"c1\"># we rely on abstractions, not direct values or types\n<\/span>    <span class=\"n\">WORD_THRESHOLD<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">],<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">guessed_letters_count<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">len<\/span><span class=\"p\">([<\/span><span class=\"n\">letter<\/span> <span class=\"k\">for<\/span> <span class=\"n\">letter<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">word<\/span> <span class=\"k\">if<\/span> <span class=\"n\">letter<\/span> <span class=\"o\">!=<\/span> <span class=\"s\">'.'<\/span><span class=\"p\">])<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">],<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">factory<\/span><span class=\"p\">(<\/span><span class=\"n\">deps<\/span><span class=\"p\">:<\/span> <span class=\"n\">_Deps<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"mi\">0<\/span> <span class=\"k\">if<\/span> <span class=\"n\">guessed<\/span> <span class=\"o\">&lt;<\/span> <span class=\"n\">deps<\/span><span class=\"p\">.<\/span><span class=\"n\">WORD_THRESHOLD<\/span> <span class=\"k\">else<\/span> <span class=\"n\">guessed<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">factory<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Please, notice how we are now passing <code>factory<\/code> function to the top level of our app. Also, pay extra attention to our new <code>_Deps<\/code> <a href=\"https:\/\/mypy.readthedocs.io\/en\/stable\/protocols.html\">protocol<\/a>: it allows us to use structural subtyping to define the required API. In other words, all objects with <code>WORD_THRESHOLD<\/code> <code>int<\/code> attribute would pass the check (to enforce the <code>django<\/code> settings typecheck use <a href=\"https:\/\/github.com\/typeddjango\/django-stubs\"><code>django-stubs<\/code><\/a>).<\/p>\n\n<p>The only thing left is to call our returned <code>factory<\/code> function from the very top:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">django.conf<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">settings<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">,<\/span> <span class=\"n\">HttpResponse<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">words_app.logic<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">calculate_points<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">:<\/span> <span class=\"n\">HttpRequest<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">user_word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">=<\/span> <span class=\"n\">request<\/span><span class=\"p\">.<\/span><span class=\"n\">POST<\/span><span class=\"p\">[<\/span><span class=\"s\">'word'<\/span><span class=\"p\">]<\/span>  <span class=\"c1\"># just an example\n<\/span>    <span class=\"n\">points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">user_words<\/span><span class=\"p\">)(<\/span><span class=\"n\">settings<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># passing the dependencies and calling\n<\/span>    <span class=\"p\">...<\/span>  <span class=\"c1\"># later you show the result to user somehow\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>Looks easy! All our requirements are satisfied:<\/p>\n\n<ol>\n<li>We don't have any magic, literally zero<\/li>\n<li>Everything is typed properly<\/li>\n<li>Our logic is still pure and is independent from the framework<\/li>\n<\/ol>\n\n<p>The only problem we now have is that our logic does not compose well. Let me illustrate my point with a new requirement. During the holiday season our game might award one extra point to the final score randomly. And here's how we can adapt our source code to meet the new requirement:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">],<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">guessed_letters_count<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">len<\/span><span class=\"p\">([<\/span><span class=\"n\">letter<\/span> <span class=\"k\">for<\/span> <span class=\"n\">letter<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">word<\/span> <span class=\"k\">if<\/span> <span class=\"n\">letter<\/span> <span class=\"o\">!=<\/span> <span class=\"s\">'.'<\/span><span class=\"p\">])<\/span>\n    <span class=\"n\">awarded_points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">_maybe_add_extra_holiday_point<\/span><span class=\"p\">(<\/span><span class=\"n\">awarded_points<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># won't work :(\n<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_maybe_add_extra_holiday_point<\/span><span class=\"p\">(<\/span><span class=\"n\">awarded_points<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">awarded_points<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span> <span class=\"k\">if<\/span> <span class=\"n\">random<\/span><span class=\"p\">.<\/span><span class=\"n\">choice<\/span><span class=\"p\">([<\/span><span class=\"bp\">True<\/span><span class=\"p\">,<\/span> <span class=\"bp\">False<\/span><span class=\"p\">])<\/span> <span class=\"k\">else<\/span> <span class=\"n\">awarded_points<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>But, oops, <code>awarded_points<\/code> has type <code>Callable[[_Deps], int]<\/code> it cannot be easily composed with this new function. Of course we can create a new function inside <code>_maybe_add_extra_holiday_point<\/code> just for the sake of composition:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">_maybe_add_extra_holiday_point<\/span><span class=\"p\">(<\/span><span class=\"n\">awarded_points<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">],<\/span> <span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">[[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">],<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">factory<\/span><span class=\"p\">(<\/span><span class=\"n\">deps<\/span><span class=\"p\">:<\/span> <span class=\"n\">_Deps<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">awarded_points<\/span><span class=\"p\">(<\/span><span class=\"n\">deps<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">points<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span> <span class=\"k\">if<\/span> <span class=\"n\">random<\/span><span class=\"p\">.<\/span><span class=\"n\">choice<\/span><span class=\"p\">([<\/span><span class=\"bp\">True<\/span><span class=\"p\">,<\/span> <span class=\"bp\">False<\/span><span class=\"p\">])<\/span> <span class=\"k\">else<\/span> <span class=\"n\">points<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">factory<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>But is it pleasant to work with? I hope most people will agree with me that it is not.<\/p>\n\n<p>Let's remember that functional programmers are smart people. They can do literally everything with just pure functions. And composition helpers. That's why they came up with the <code>Reader<\/code> monad (or as we call it <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/context.html\"><code>RequiresContext<\/code> container<\/a> to not scare people to death in Python land): it is a composition helper for this exact situation. Let's refactor our code once again to see how it works:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">random<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">typing_extensions<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Protocol<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">returns.context<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">RequiresContext<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">_Deps<\/span><span class=\"p\">(<\/span><span class=\"n\">Protocol<\/span><span class=\"p\">):<\/span>  <span class=\"c1\"># we rely on abstractions, not direct values or types\n<\/span>    <span class=\"n\">WORD_THRESHOLD<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">RequiresContext<\/span><span class=\"p\">[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">guessed_letters_count<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">len<\/span><span class=\"p\">([<\/span><span class=\"n\">letter<\/span> <span class=\"k\">for<\/span> <span class=\"n\">letter<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">word<\/span> <span class=\"k\">if<\/span> <span class=\"n\">letter<\/span> <span class=\"o\">!=<\/span> <span class=\"s\">'.'<\/span><span class=\"p\">])<\/span>\n    <span class=\"n\">awarded_points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">awarded_points<\/span><span class=\"p\">.<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span><span class=\"n\">_maybe_add_extra_holiday_point<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">RequiresContext<\/span><span class=\"p\">[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">factory<\/span><span class=\"p\">(<\/span><span class=\"n\">deps<\/span><span class=\"p\">:<\/span> <span class=\"n\">_Deps<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"mi\">0<\/span> <span class=\"k\">if<\/span> <span class=\"n\">guessed<\/span> <span class=\"o\">&lt;<\/span> <span class=\"n\">deps<\/span><span class=\"p\">.<\/span><span class=\"n\">WORD_THRESHOLD<\/span> <span class=\"k\">else<\/span> <span class=\"n\">guessed<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">RequiresContext<\/span><span class=\"p\">(<\/span><span class=\"n\">factory<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">_maybe_add_extra_holiday_point<\/span><span class=\"p\">(<\/span><span class=\"n\">awarded_points<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">awarded_points<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span> <span class=\"k\">if<\/span> <span class=\"n\">random<\/span><span class=\"p\">.<\/span><span class=\"n\">choice<\/span><span class=\"p\">([<\/span><span class=\"bp\">True<\/span><span class=\"p\">,<\/span> <span class=\"bp\">False<\/span><span class=\"p\">])<\/span> <span class=\"k\">else<\/span> <span class=\"n\">awarded_points<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>We have changed the return type of the functions and we have also added <code>awarded_points.map(_maybe_add_extra_holiday_point)<\/code>. Which is another way of saying \"compose <code>RequiresContext<\/code> container with this pure function <code>_maybe_add_extra_holiday_point<\/code>\". We don't change our framework layer at all.<\/p>\n\n<p>How does it work?<\/p>\n\n<ol>\n<li>When we call <code>calculate_points(user_words)<\/code> it does not actually start to do anything, it only returns <code>RequiresContext<\/code> container to be called later<\/li>\n<li>The container is smart enough to understand the <code>.map<\/code> method. It remembers that after its execution it will need to call <code>_maybe_add_extra_holiday_point<\/code> function<\/li>\n<li>When we add the context to the container in a form of <code>calculate_points(user_words)(settings)<\/code> our <code>def factory(deps: _Deps)<\/code> executes and returns the long-awaited value<\/li>\n<li>Then <code>_maybe_add_extra_holiday_point<\/code> actually executes and returns the final value<\/li>\n<\/ol>\n\n<p>That's it. No mess, no magic, no framework internals. But typing, composition, and simplicity.<\/p>\n\n<h2>\n  \n  \n  Transparent dependencies\n<\/h2>\n\n<p>What if you want to also change the symbol that indicates unguessed letter (currently <code>.<\/code>), to be configurable? Some users prefer <code>.<\/code>, some <code>_<\/code>. Ok, we can do that, cannot we?<\/p>\n\n<p>A little confusion can happen to functional newcomers at this step. Because we only have <code>deps<\/code> available inside <code>_award_points_for_letters<\/code> and not inside <code>calculate_points<\/code>. And composition again is the answer. We have a special composition helper for this case: <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/context.html#returns.context.requires_context.Context.ask\"><code>Context.ask()<\/code><\/a> which fetches the dependencies from the current context and allows us to explicitly use it whenever we want:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">returns.context<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Context<\/span><span class=\"p\">,<\/span> <span class=\"n\">RequiresContext<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">_Deps<\/span><span class=\"p\">(<\/span><span class=\"n\">Protocol<\/span><span class=\"p\">):<\/span>  <span class=\"c1\"># we rely on abstractions, not direct values or types\n<\/span>    <span class=\"n\">WORD_THRESSHOLD<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">UNGUESSED_CHAR<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>  <span class=\"c1\"># new value!\n<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">calculate_points<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">RequiresContext<\/span><span class=\"p\">[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">factory<\/span><span class=\"p\">(<\/span><span class=\"n\">deps<\/span><span class=\"p\">:<\/span> <span class=\"n\">_Deps<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">RequiresContext<\/span><span class=\"p\">[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n        <span class=\"n\">guessed_letters_count<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">len<\/span><span class=\"p\">([<\/span>\n            <span class=\"n\">letter<\/span> <span class=\"k\">for<\/span> <span class=\"n\">letter<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">word<\/span> <span class=\"k\">if<\/span> <span class=\"n\">letter<\/span> <span class=\"o\">!=<\/span> <span class=\"n\">deps<\/span><span class=\"p\">.<\/span><span class=\"n\">UNGUESSED_CHAR<\/span>\n        <span class=\"p\">])<\/span>\n        <span class=\"n\">awarded_points<\/span> <span class=\"o\">=<\/span> <span class=\"n\">_award_points_for_letters<\/span><span class=\"p\">(<\/span><span class=\"n\">guessed_letters_count<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">awarded_points<\/span><span class=\"p\">.<\/span><span class=\"nb\">map<\/span><span class=\"p\">(<\/span><span class=\"n\">_maybe_add_extra_holiday_point<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"n\">Context<\/span><span class=\"p\">[<\/span><span class=\"n\">_Deps<\/span><span class=\"p\">].<\/span><span class=\"n\">ask<\/span><span class=\"p\">().<\/span><span class=\"n\">bind<\/span><span class=\"p\">(<\/span><span class=\"n\">factory<\/span><span class=\"p\">)<\/span>\n\n<span class=\"c1\"># ...\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>Two things to mention here:<\/p>\n\n<ol>\n<li>\n<code>Context.ask()<\/code> requires to be explicitly annotated with <code>_Deps<\/code>, because <code>mypy<\/code> cannot infer the type here<\/li>\n<li>\n<code>.bind<\/code> method is also a composition utility. In contrast to <code>.map<\/code> which composes a container with a pure function, <code>.bind<\/code> allows us to compose a container with a function that also returns a container of the same type<\/li>\n<\/ol>\n\n<p>Now we can share the same immutable read-only context for all our code.<\/p>\n\n<h2>\n  \n  \n  Static typing\n<\/h2>\n\n<p>Static typing is much more than accidentally trying to add a string to an integer. It is about your architecture and rules that each piece of this complex system has.<\/p>\n\n<p>And many readers might already found one trap that I have left in this example. Let's reveal the ugly truth: <code>_maybe_add_extra_holiday_point<\/code> is not pure. We won't be able to test it without a mock on <code>random<\/code>. And because it is impure, its type is <code>Callable[[int], IO[int]]<\/code>.<\/p>\n\n<p>Wait, what is it? <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/io.html\"><code>IO<\/code><\/a> is your best friend when we talk about good architecture and high-quality composition rules. It is an explicit marker that your function is not pure. Without this type, we are free to write an ugly, untestable code where we want it. We don't have any contracts to respect. We can just pray for the reigning chaos.<\/p>\n\n<p>And also, don't forget that some functions may end up with successful execution and may also fail. <a href=\"https:\/\/sobolevn.me\/2019\/02\/python-exceptions-considered-an-antipattern\">Throwing exceptions<\/a> inside your business logic is ugly. And it also removes the explicit contract from your architecture. That's a barbarian practice. If something can go wrong, we have to be prepared for it. That's why many people use <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/result.html\"><code>Result<\/code><\/a> to indicate that things can (and will!) go wrong.<\/p>\n\n<p>All these two types are heavily related to our <code>RequiresContext<\/code> container. Because its return type can be tricky:<\/p>\n\n<ul>\n<li>\n<code>RequiresContext[EnvType, ReturnType]<\/code> indicates that we work with a pure function that cannot fail (like in our example without <code>random<\/code>)<\/li>\n<li>\n<code>RequiresContext[EnvType, Result[ValueType, ErrorType]]<\/code> indicates that we work with a pure function that can fail (for example when we use <code>\/<\/code> math operator and <code>0<\/code> as a delimiter or any logical failures)<\/li>\n<li>\n<code>RequiresContext[EnvType, IO[ReturnType]]<\/code> indicates that we work with an impure function that cannot fail (like our example with <code>random<\/code>)<\/li>\n<li>\n<code>RequiresContext[EnvType, IO[Result[ValueType, ErrorType]]]<\/code> indicates that we work with an impure function that can fail (like HTTP, filesystem, or database calls)<\/li>\n<\/ul>\n\n<p>It does not look fun to compose, yeah? That's why we also ship useful combinators with <a href=\"https:\/\/github.com\/dry-python\/returns\"><code>returns<\/code><\/a> like:<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/context.html#requirescontextresult-container\"><code>RequireContextResult<\/code><\/a> to easily work with <code>RequiresContext<\/code> which has <code>Result<\/code> as the return type<\/li>\n<li>\n<a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/pages\/context.html#requirescontextioresult-container\"><code>RequiresContextIOResult<\/code><\/a> to easily work with <code>RequiresContext<\/code> which has <code>IO[Result]<\/code> as the return type<\/li>\n<\/ul>\n\n<p>This way <code>returns<\/code> library takes all the hacky composition rules on itself and provides nice API for our end users.<\/p>\n\n<h2>\n  \n  \n  DI containers\n<\/h2>\n\n<p>\"So, are you saying that I should not use any DI frameworks at all?\" one may ask after reading this article. And it is an interesting question indeed.<\/p>\n\n<p>I have spent a lot of time thinking about this very topic myself. And my answer is: you can (and probably should) use a dependency injection container framework on the framework level.<\/p>\n\n<p>And here's why: in real-world apps your <code>_Deps<\/code> class with soon become really big. It would have lots of stuff inside:<\/p>\n\n<ul>\n<li>Repositories and database-related utilities<\/li>\n<li>HTTP services and API integrations<\/li>\n<li>Permission and authentication helpers<\/li>\n<li>Caching layer<\/li>\n<li>Async tasks and utilities to work with them<\/li>\n<li>And probably more!<\/li>\n<\/ul>\n\n<p>To handle all this stuff you would need to import a lot of other stuff.<br>\nAnd you will need to create this monstrous object somehow. That's where DI frameworks can help you. A lot.<\/p>\n\n<p>With their magic creating this context might be a lot easier and safer. You need complex tools to fight real complexity.<\/p>\n\n<p>That's why tools like <a href=\"https:\/\/github.com\/dry-python\/dependencies\"><code>dependencies<\/code><\/a> are required for every complex system: both functional and imperative.<\/p>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>Let's sum up things we have learned today:<\/p>\n\n<ol>\n<li>Different applications require different levels of architecture and have different requirements on dependency injection<\/li>\n<li>If you are scared from magic DI containers have, use typed functional composition: <code>RequiresContext<\/code> can help you to provide the required context from the top level to the very bottom and to define nice composition API<\/li>\n<li>When writing high-level code think about your architecture and explicitly define your contracts as types. Use <code>IO<\/code> and <code>Result<\/code> to indicate possible failure and impurity<\/li>\n<li>Use composition helpers when in doubt<\/li>\n<li>Use DI containers on the very top level of your app, when complexity gets out of your control<\/li>\n<\/ol>\n\n<p>Functional programming is fun and easy! Feel free to <a href=\"https:\/\/github.com\/dry-python\/returns\">star our repo<\/a> if you liked the concepts above. Or go to <a href=\"https:\/\/returns.readthedocs.io\/en\/latest\/index.html\">the docs<\/a> to learn more new things.<\/p>\n\n<p>Very special thanks to <a href=\"https:\/\/www.facebook.com\/nikolay.fominykh\">Nikolay Fominykh<\/a> and <a href=\"https:\/\/github.com\/supadrupa\">Artem<\/a> for reviewing this article.<\/p>\n\n","category":["python","webdev","beginners","django"]},{"title":"9 best open-source findings, November 2019","pubDate":"Wed, 18 Dec 2019 12:43:17 +0000","link":"https:\/\/dev.to\/sobolevn\/9-best-open-source-findings-november-2019-gi0","guid":"https:\/\/dev.to\/sobolevn\/9-best-open-source-findings-november-2019-gi0","description":"<p>Hello, everyone!<\/p>\n\n<p>Let me introduce a list of the best open-source findings for November 2019.<\/p>\n\n<p>If you want to have more awesomeness, including new and useful open-source tools, great articles, and excellent talks - you can join my telegram channel called <a href=\"https:\/\/t.me\/opensource_findings\">@OpensourceFindings<\/a> (<a href=\"https:\/\/tlg.name\/opensource_findings\">mirror link<\/a>).<\/p>\n\n<p>In this list we will discuss: Rust, TypeScript, JavaScript, Go, and Python.<br>\nThis includes web development, devops and QA tooling, and DX improvements.<\/p>\n\n<h2>\n  \n  \n  githistory\n<\/h2>\n\n<p>Your time machine for Github. Allows to see how each file was modified during its lifetime.<br>\nWritten in JavaScript.<\/p>\n\n<p><a href=\"https:\/\/github.com\/pomber\/git-history\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--0Oh9sF0q--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/user-images.githubusercontent.com\/1911623\/54575634-9b10b000-49d3-11e9-8a19-56e40636e45d.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--0Oh9sF0q--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/user-images.githubusercontent.com\/1911623\/54575634-9b10b000-49d3-11e9-8a19-56e40636e45d.gif\" alt=\"githistory\"><\/a><\/p>\n\n<h2>\n  \n  \n  qawolf\n<\/h2>\n\n<p>Just one command <code>npx qawolf record example.com<\/code> to record your E2E tests.<br>\nWritten in TypeScript.<\/p>\n\n<p><a href=\"https:\/\/github.com\/qawolf\/qawolf\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--uG5vrLOD--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--ACuSD0bS--\/c_limit%252Cf_auto%252Cfl_progressive%252Cq_66%252Cw_880\/https:\/\/thepracticaldev.s3.amazonaws.com\/i\/60b1lcpobe6wtobv24gs.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--uG5vrLOD--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--ACuSD0bS--\/c_limit%252Cf_auto%252Cfl_progressive%252Cq_66%252Cw_880\/https:\/\/thepracticaldev.s3.amazonaws.com\/i\/60b1lcpobe6wtobv24gs.gif\" alt=\"qawolf\"><\/a><\/p>\n\n<h2>\n  \n  \n  plynth\n<\/h2>\n\n<p>And now I will show you something really unusual!<br>\nYou can write thread-safe HTML\/CSS\/Python GUI framework. Wait, what? Yes, you heard it write. You can write your frontend in Python. It even supports Vue.js out of the box.<br>\nWritten in Python.<\/p>\n\n<p><a href=\"https:\/\/www.plynth.net\/\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--jBG7uMsS--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/rw\/7y\/ei\/rw7yeiu7mmpjivvezur7koyonnq.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--jBG7uMsS--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/rw\/7y\/ei\/rw7yeiu7mmpjivvezur7koyonnq.jpeg\" alt=\"plynth\"><\/a><\/p>\n\n<h2>\n  \n  \n  lazydocker\n<\/h2>\n\n<p>Minimalistic TUI to work with docker. Supports mouse.<br>\nWritten in Go.<\/p>\n\n<p><a href=\"https:\/\/github.com\/jesseduffield\/lazydocker\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--SAN_fftN--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/lu\/qw\/tc\/luqwtcisi1eyrmwahxfu1lqwbbu.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--SAN_fftN--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/lu\/qw\/tc\/luqwtcisi1eyrmwahxfu1lqwbbu.gif\" alt=\"Lazydocker\"><\/a><\/p>\n\n<h2>\n  \n  \n  tui-rs\n<\/h2>\n\n<p>One more thing about TUIs. Do you want to build your own one?<br>\nHere's the toolset you can use. Multiple components, easy setup.<br>\nWritten in Rust.<\/p>\n\n<p><a href=\"https:\/\/github.com\/fdehau\/tui-rs\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--DUg8TsYD--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/zp\/c7\/pb\/zpc7pbr95dpn16u1vfard-51bgc.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--DUg8TsYD--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/zp\/c7\/pb\/zpc7pbr95dpn16u1vfard-51bgc.gif\" alt=\"tui-rs\"><\/a><\/p>\n\n<h2>\n  \n  \n  easydb\n<\/h2>\n\n<p>Database in just one click.<br>\nThere are clients in Python, JavaScript, Ruby, Bash.<\/p>\n\n<p><a href=\"https:\/\/easydb.io\/\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--fCQ706rE--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/y0\/vf\/95\/y0vf959pyfokv8a4qys7yhzzcxe.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--fCQ706rE--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/y0\/vf\/95\/y0vf959pyfokv8a4qys7yhzzcxe.jpeg\" alt=\"easydb\"><\/a><\/p>\n\n<h2>\n  \n  \n  transform.tools\n<\/h2>\n\n<p>Very useful service to covert different formats in and out. Here are some examples:<\/p>\n\n<ul>\n<li>HTML and SVG into jsx<\/li>\n<li>json into Kotlin, TypeScript, or Go Struct<\/li>\n<li>Markdown into HTML, yaml int json, and etc<\/li>\n<\/ul>\n\n<p><a href=\"https:\/\/github.com\/ritz078\/transform\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--5JOHjSNT--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/-h\/dk\/0a\/-hdk0apb_qctvod9pg1pwaqw4oi.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--5JOHjSNT--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/-h\/dk\/0a\/-hdk0apb_qctvod9pg1pwaqw4oi.gif\" alt=\"transform.tools\"><\/a><\/p>\n\n<h2>\n  \n  \n  \u200b\u200bdovpanda\n<\/h2>\n\n<p>Linter for <code>pandas<\/code>. Finds errors and suggests improvements. Very helpful for ones who just start to work with data. Or teach how to work with it.<br>\nWritten in Python.<\/p>\n\n<p><a href=\"https:\/\/github.com\/dovpanda-dev\/dovpanda\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--afOneA1F--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/eu\/oz\/nh\/euoznhyc7c79aus6epadbm1tx8i.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--afOneA1F--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/eu\/oz\/nh\/euoznhyc7c79aus6epadbm1tx8i.jpeg\" alt=\"dovpanda\"><\/a><\/p>\n\n<h2>\n  \n  \n  flowy\n<\/h2>\n\n<p>Beautiful library to create flowcharts. <br>\nWritten in JavaScript.<\/p>\n\n<p><a href=\"https:\/\/github.com\/alyssaxuu\/flowy\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--uUa_dvCL--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/ic\/sa\/gd\/icsagdf7dom5b5tdjuyfiodld5q.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--uUa_dvCL--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/ic\/sa\/gd\/icsagdf7dom5b5tdjuyfiodld5q.gif\" alt=\"flowy\"><\/a><\/p>\n\n<h2>\n  \n  \n  Bonus!\n<\/h2>\n\n<p><a href=\"https:\/\/github.com\/wemake-services\/dotenv-linter\">dotenv-linter<\/a>: this is a linter for your <code>.env<\/code> files. Simple, protects you from erros in your configuration, forces you and your team to stick to the same style. Now with Github Actions support, it can (and will!) comment your pull requests like so:<\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--kLXE1bbC--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/gy\/hs\/eq\/gyhseqmyiygk77xgj3pscsmkjsg.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--kLXE1bbC--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/gy\/hs\/eq\/gyhseqmyiygk77xgj3pscsmkjsg.png\" alt=\"dotenv-linter\"><\/a><\/p>\n\n<p>That's it for today! Make sure to <a href=\"https:\/\/t.me\/opensource_findings\">subscribe<\/a> to my channel if you liked this list. I am going to post new ones each month. Also, feel free to post any cool projects you know in the comments. Feedback is always welcome.<\/p>\n\n<p>You can also <a href=\"https:\/\/github.com\/sobolevn\">follow me on Github<\/a> to see how new instruments are made, and what you can help with! It is a great start if you are new to open-source.<\/p>\n\n","category":["python","javascript","go","rust"]},{"title":"Testing Django Migrations","pubDate":"Mon, 02 Dec 2019 15:12:15 +0000","link":"https:\/\/dev.to\/wemake-services\/testing-django-migrations-3288","guid":"https:\/\/dev.to\/wemake-services\/testing-django-migrations-3288","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2019\/10\/testing-django-migrations\">https:\/\/sobolevn.me\/2019\/10\/testing-django-migrations<\/a><\/p>\n\n<p>Dear internet,<\/p>\n\n<p>Today we have screwed up by applying a broken migration to the running production service and causing a massive outage for several hours... Because the rollback function was terribly broken as well.<\/p>\n\n<p>As a result, we had to restore a backup that was made several hours ago, losing some new data.<\/p>\n\n<h2>\n  \n  \n  Why did it happen?\n<\/h2>\n\n<p>The easiest answer is just to say: \"Because it is X's fault! He is the author of this migration, he should learn how databases work\". But, it is counterproductive.<\/p>\n\n<p>Instead, as a part of our <a href=\"https:\/\/sobolevn.me\/2018\/12\/blameless-environment\">\"Blameless environment\"<\/a> culture, we tend to put all the guilt on the CI. It was the CI who put the broken code into the <code>master<\/code> branch. So, we need to improve it!<\/p>\n\n<p>We always <a href=\"https:\/\/sobolevn.me\/2019\/01\/how-to-fix-a-bug\">write post-mortems<\/a> for all massive incidents that we experience. And we write regression tests for all bugs, so they won't happen again. But, this situation was different, since it was a broken migration that worked during the CI process, and it was hard or impossible to test with the current set of instruments.<\/p>\n\n<p>So, let me explain the steps we took to solve this riddle.<\/p>\n\n<h2>\n  \n  \n  Existing setup\n<\/h2>\n\n<p>We use a very strict <a href=\"https:\/\/github.com\/wemake-services\/wemake-django-template\"><code>django<\/code> project setup<\/a> with several quality checks for our migrations:<\/p>\n\n<ol>\n<li>We write all data migration as typed functions in our main source code. Then we check everything with <code>mypy<\/code> and test as regular functions<\/li>\n<li>We lint migration files with <a href=\"https:\/\/github.com\/wemake-services\/wemake-python-styleguide\"><code>wemake-python-styleguide<\/code><\/a>, it drastically reduces the possibility of bad code inside the migration files<\/li>\n<li>We use tests that automatically set up the database by applying all migrations before each session<\/li>\n<li>We use <a href=\"https:\/\/github.com\/3YOURMIND\/django-migration-linter\"><code>django-migration-linter<\/code><\/a> to find migrations that are not suited for zero-time deployments<\/li>\n<li>And then we review the code by two senior people<\/li>\n<li>Then we test everything manually with the help of the review apps<\/li>\n<\/ol>\n\n<p>And somehow it is still not enough: our server was dead.<\/p>\n\n<p>When writing the post-mortem for this bug, I spotted that data in our staging and production services were different. And that's why our data migration crushed and left one of the core tables in the broken state.<\/p>\n\n<p>So, how can we test migrations on some existing data?<\/p>\n\n<h2>\n  \n  \n  django-test-migrations\n<\/h2>\n\n<p>That's where <a href=\"https:\/\/github.com\/wemake-services\/django-test-migrations\"><code>django-test-migrations<\/code><\/a> comes in handy.<\/p>\n\n<p>The idea of this project is simple:<\/p>\n\n<ol>\n<li>Set some migration as a starting point<\/li>\n<li>Create some model's data that you want to test<\/li>\n<li>Run the new migration that you are testing<\/li>\n<li>Assert the results!<\/li>\n<\/ol>\n\n<p>Let's illustrate it with some code samples.<br>\nFull source code is <a href=\"https:\/\/github.com\/wemake-services\/django-test-migrations\/tree\/master\/django_test_app\">available here<\/a>.<\/p>\n\n<p>Here's the latest version of our model:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">SomeItem<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"p\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"s\">\"\"\"We use this model for testing migrations.\"\"\"<\/span>\n\n    <span class=\"n\">string_field<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"p\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">50<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">is_clean<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"p\">.<\/span><span class=\"n\">BooleanField<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>This is a pretty simple model that serves only one purpose: to illustrate the problem. <code>is_clean<\/code> field is related to the contents of <code>string_field<\/code> in some manner.<br>\nWhile the <code>string_field<\/code> itself contains only regular text data.<\/p>\n\n<p>Imagine that you have a data migration that looks like so:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">_is_clean_item<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">:<\/span> <span class=\"s\">'SomeItem'<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n    <span class=\"s\">\"\"\"\n    Pure function to the actual migration.\n\n    Ideally, it should be moved to ``main_app\/logic\/migrations``.\n    But, as an example it is easier to read them together.\n    \"\"\"<\/span>\n    <span class=\"k\">return<\/span> <span class=\"s\">' '<\/span> <span class=\"ow\">not<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">string_field<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">_set_clean_flag<\/span><span class=\"p\">(<\/span><span class=\"n\">apps<\/span><span class=\"p\">,<\/span> <span class=\"n\">schema_editor<\/span><span class=\"p\">):<\/span>\n    <span class=\"s\">\"\"\"\n    Performs the data-migration.\n\n    We can't import the ``SomeItem`` model directly as it may be a newer\n    version than this migration expects.\n\n    We are using ``.all()`` because\n    we don't have a lot of ``SomeItem`` instances.\n    In real-life you should not do that.\n    \"\"\"<\/span>\n    <span class=\"n\">SomeItem<\/span> <span class=\"o\">=<\/span> <span class=\"n\">apps<\/span><span class=\"p\">.<\/span><span class=\"n\">get_model<\/span><span class=\"p\">(<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'SomeItem'<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">instance<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"nb\">all<\/span><span class=\"p\">():<\/span>\n        <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">is_clean<\/span> <span class=\"o\">=<\/span> <span class=\"n\">_is_clean_item<\/span><span class=\"p\">(<\/span><span class=\"n\">instance<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">instance<\/span><span class=\"p\">.<\/span><span class=\"n\">save<\/span><span class=\"p\">(<\/span><span class=\"n\">update_fields<\/span><span class=\"o\">=<\/span><span class=\"p\">[<\/span><span class=\"s\">'is_clean'<\/span><span class=\"p\">])<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">_remove_clean_flags<\/span><span class=\"p\">(<\/span><span class=\"n\">apps<\/span><span class=\"p\">,<\/span> <span class=\"n\">schema_editor<\/span><span class=\"p\">):<\/span>\n    <span class=\"s\">\"\"\"\n    This is just a noop example of a rollback function.\n\n    It is not used in our simple case,\n    but it should be implemented for more complex scenarios.\n    \"\"\"<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Migration<\/span><span class=\"p\">(<\/span><span class=\"n\">migrations<\/span><span class=\"p\">.<\/span><span class=\"n\">Migration<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">dependencies<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span>\n        <span class=\"p\">(<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'0002_someitem_is_clean'<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">]<\/span>\n\n    <span class=\"n\">operations<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span>\n        <span class=\"n\">migrations<\/span><span class=\"p\">.<\/span><span class=\"n\">RunPython<\/span><span class=\"p\">(<\/span><span class=\"n\">_set_clean_flag<\/span><span class=\"p\">,<\/span> <span class=\"n\">_remove_clean_flags<\/span><span class=\"p\">),<\/span>\n    <span class=\"p\">]<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>And here's how we are going to test this migration. At first, we will have to set some migration as a starting point:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"n\">old_state<\/span> <span class=\"o\">=<\/span> <span class=\"n\">migrator<\/span><span class=\"p\">.<\/span><span class=\"n\">before<\/span><span class=\"p\">((<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'0002_someitem_is_clean'<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Then we have to get the model class. We cannot use direct <code>import<\/code> from <code>models<\/code> because the model might be different, since migrations change them from our stored definition:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"n\">SomeItem<\/span> <span class=\"o\">=<\/span> <span class=\"n\">old_state<\/span><span class=\"p\">.<\/span><span class=\"n\">apps<\/span><span class=\"p\">.<\/span><span class=\"n\">get_model<\/span><span class=\"p\">(<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'SomeItem'<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>Then we need to create some data that we want to test:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"c1\"># One instance will be `clean`, the other won't be:\n<\/span><span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">create<\/span><span class=\"p\">(<\/span><span class=\"n\">string_field<\/span><span class=\"o\">=<\/span><span class=\"s\">'a'<\/span><span class=\"p\">)<\/span> <span class=\"c1\"># clean\n<\/span><span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">create<\/span><span class=\"p\">(<\/span><span class=\"n\">string_field<\/span><span class=\"o\">=<\/span><span class=\"s\">'a b'<\/span><span class=\"p\">)<\/span> <span class=\"c1\"># contains whitespace, is not clean\n<\/span><\/code><\/pre><\/div>\n\n\n\n<p>Then we will run the migration that we are testing and get the new project state:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"n\">new_state<\/span> <span class=\"o\">=<\/span> <span class=\"n\">migrator<\/span><span class=\"p\">.<\/span><span class=\"n\">after<\/span><span class=\"p\">((<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'0003_auto_20191119_2125'<\/span><span class=\"p\">))<\/span>\n<span class=\"n\">SomeItem<\/span> <span class=\"o\">=<\/span> <span class=\"n\">new_state<\/span><span class=\"p\">.<\/span><span class=\"n\">apps<\/span><span class=\"p\">.<\/span><span class=\"n\">get_model<\/span><span class=\"p\">(<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'SomeItem'<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>And the last step: we need to make some assertions on the resulting data.<br>\nWe have created two model instances before: one clean and one with the whitespace. So, let's check that:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">2<\/span>\n<span class=\"c1\"># One instance is clean, the other is not:\n<\/span><span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"nb\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">is_clean<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">).<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>\n<span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"nb\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">is_clean<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">).<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>And that's how it works! Now we have an ability to test our schema and data transformations with ease. Complete test example:<br>\n<\/p>\n\n<div class=\"highlight\"><pre class=\"highlight python\"><code><span class=\"o\">@<\/span><span class=\"n\">pytest<\/span><span class=\"p\">.<\/span><span class=\"n\">mark<\/span><span class=\"p\">.<\/span><span class=\"n\">django_db<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">test_main_migration0002<\/span><span class=\"p\">(<\/span><span class=\"n\">migrator<\/span><span class=\"p\">):<\/span>\n    <span class=\"s\">\"\"\"Ensures that the second migration works.\"\"\"<\/span>\n    <span class=\"n\">old_state<\/span> <span class=\"o\">=<\/span> <span class=\"n\">migrator<\/span><span class=\"p\">.<\/span><span class=\"n\">before<\/span><span class=\"p\">((<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'0002_someitem_is_clean'<\/span><span class=\"p\">))<\/span>\n    <span class=\"n\">SomeItem<\/span> <span class=\"o\">=<\/span> <span class=\"n\">old_state<\/span><span class=\"p\">.<\/span><span class=\"n\">apps<\/span><span class=\"p\">.<\/span><span class=\"n\">get_model<\/span><span class=\"p\">(<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'SomeItem'<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># One instance will be `clean`, the other won't be:\n<\/span>    <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">create<\/span><span class=\"p\">(<\/span><span class=\"n\">string_field<\/span><span class=\"o\">=<\/span><span class=\"s\">'a'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">create<\/span><span class=\"p\">(<\/span><span class=\"n\">string_field<\/span><span class=\"o\">=<\/span><span class=\"s\">'a b'<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">2<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"nb\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">is_clean<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">).<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">2<\/span>\n\n    <span class=\"n\">new_state<\/span> <span class=\"o\">=<\/span> <span class=\"n\">migrator<\/span><span class=\"p\">.<\/span><span class=\"n\">after<\/span><span class=\"p\">((<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'0003_auto_20191119_2125'<\/span><span class=\"p\">))<\/span>\n    <span class=\"n\">SomeItem<\/span> <span class=\"o\">=<\/span> <span class=\"n\">new_state<\/span><span class=\"p\">.<\/span><span class=\"n\">apps<\/span><span class=\"p\">.<\/span><span class=\"n\">get_model<\/span><span class=\"p\">(<\/span><span class=\"s\">'main_app'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'SomeItem'<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">2<\/span>\n    <span class=\"c1\"># One instance is clean, the other is not:\n<\/span>    <span class=\"k\">assert<\/span> <span class=\"n\">SomeItem<\/span><span class=\"p\">.<\/span><span class=\"n\">objects<\/span><span class=\"p\">.<\/span><span class=\"nb\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">is_clean<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">).<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>\n<\/code><\/pre><\/div>\n\n\n\n<p>By the way, we also support raw <a href=\"https:\/\/github.com\/wemake-services\/django-test-migrations#unittest\"><code>unittest<\/code> cases<\/a>.<\/p>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>Don't be sure about your migrations. Test them!<\/p>\n\n<p>You can test forward and rollback migrations and their <a href=\"https:\/\/github.com\/wemake-services\/django-test-migrations#testing-migrations-ordering\">ordering<\/a> with the help of <code>django-test-migrations<\/code>. It is simple, friendly, and already works with the test framework of your choice.<\/p>\n\n<p>I also want to say \"thank you\" to <a href=\"https:\/\/github.com\/wemake-services\/django-test-migrations#credits\">these awesome people<\/a>. Without their work it would take me much longer to come up with the working solution.<\/p>\n\n","category":["python","django","webdev","beginners"]},{"title":"9 best open-source findings, October 2019","pubDate":"Mon, 11 Nov 2019 16:13:49 +0000","link":"https:\/\/dev.to\/sobolevn\/9-best-open-source-findings-october-2019-1klb","guid":"https:\/\/dev.to\/sobolevn\/9-best-open-source-findings-october-2019-1klb","description":"<p>Hello, everyone!<\/p>\n\n<p>Let me introduce a list of the best open-source findings for October 2019.<\/p>\n\n<p>If you want to have more awesomeness, including new and useful open-source tools, great articles, and excellent talks - you can join my telegram channel called <a href=\"https:\/\/t.me\/opensource_findings\">@OpensourceFindings<\/a> (<a href=\"https:\/\/tlg.name\/opensource_findings\">mirror link<\/a>).<\/p>\n\n<p>In this list we will discuss: Rust, Swift, TypeScript, JavaScript, Go, Scala, and Python. <br>\nThis includes web and mobile development, developer tooling, and Big Data tooling.<\/p>\n\n<h2>\n  \n  \n  vue-interactive-paycard\n<\/h2>\n\n<p>Beautiful credit card component with smooth animations.<br>\nWritten in JavaScript + Vue.<\/p>\n\n<p><a href=\"https:\/\/github.com\/muhammederdem\/vue-interactive-paycard\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--WOOoWpGQ--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/no\/5o\/8d\/no5o8dcm1g_k5v4hci-t-5gmlx8.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--WOOoWpGQ--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/no\/5o\/8d\/no5o8dcm1g_k5v4hci-t-5gmlx8.gif\" alt=\"vue-interactive-paycard\"><\/a><\/p>\n\n<h2>\n  \n  \n  rx\n<\/h2>\n\n<p>Minimalistic pixel-editor, that feels like <code>vi<\/code>.<br>\nWritten in Rust.<\/p>\n\n<p><a href=\"https:\/\/github.com\/cloudhead\/rx\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--Ldytp1fB--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/pi\/t2\/d4\/pit2d4a0ommh-eouj0bvmqj41-i.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--Ldytp1fB--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/pi\/t2\/d4\/pit2d4a0ommh-eouj0bvmqj41-i.gif\" alt=\"rx\"><\/a><\/p>\n\n<h2>\n  \n  \n  Bow\n<\/h2>\n\n<p>Swift function programming! Supports \"emulated HKT\", algebraic data types, and ad-hoc polymorphism. <br>\nWritten in Swift.<\/p>\n\n<p><a href=\"https:\/\/bow-swift.io\/\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--tBXoR_iP--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/85\/qp\/46\/85qp46hhi8kcmz1mdrwojmdz8eo.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--tBXoR_iP--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/85\/qp\/46\/85qp46hhi8kcmz1mdrwojmdz8eo.png\" alt=\"Bow\"><\/a><\/p>\n\n<h2>\n  \n  \n  is-website-vulnerable\n<\/h2>\n\n<p>CLI tool to check for insecure javascript code and libraries on any website. Just type an address!<br>\nWritten in JavaScript.<\/p>\n\n<p><a href=\"https:\/\/github.com\/lirantal\/is-website-vulnerable\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--WsRCSwGD--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/gj\/r-\/tq\/gjr-tqeutnw9noinpqwx0nmgb4o.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--WsRCSwGD--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/gj\/r-\/tq\/gjr-tqeutnw9noinpqwx0nmgb4o.jpeg\" alt=\"is-website-vulnerable\"><\/a><\/p>\n\n<h2>\n  \n  \n  SandDance\n<\/h2>\n\n<p>Beautiful data visualisation library.<br>\nWritten in TypeScript.<\/p>\n\n<p><a href=\"https:\/\/github.com\/microsoft\/SandDance\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--nYsXUfID--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/el\/gr\/y-\/elgry-4mzbihwrrwn-fcmlozivi.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--nYsXUfID--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/el\/gr\/y-\/elgry-4mzbihwrrwn-fcmlozivi.gif\" alt=\"SandDance\"><\/a><\/p>\n\n<h2>\n  \n  \n  spleeter\n<\/h2>\n\n<p>spleeter makes it easy to train source separation model (assuming you have a dataset of isolated sources), and provides already trained state of the art model for performing various flavour of separation.<br>\nWritten in Python.<\/p>\n\n<p><a href=\"https:\/\/github.com\/deezer\/spleeter\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--No7qvSjV--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/co\/sp\/ra\/cosprakstfetgaflas_abayakoi.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--No7qvSjV--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/co\/sp\/ra\/cosprakstfetgaflas_abayakoi.jpeg\" alt=\"spleeter\"><\/a><\/p>\n\n<h2>\n  \n  \n  grpcui\n<\/h2>\n\n<p>An interactive web UI for gRPC, along the lines of Postman.<br>\nWritten in Go.<\/p>\n\n<p><a href=\"https:\/\/github.com\/fullstorydev\/grpcui\">Link<\/a><br>\n<a href=\"https:\/\/github.com\/fullstorydev\/grpcurl\">There's also a curl alternative<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--TJe1fyvi--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/is\/4b\/5n\/is4b5nk5kwab8wv4izphrpokgqu.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--TJe1fyvi--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/is\/4b\/5n\/is4b5nk5kwab8wv4izphrpokgqu.jpeg\" alt=\"grpcui\"><\/a><\/p>\n\n<h2>\n  \n  \n  polynote\n<\/h2>\n\n<p>Polynote is a different kind of notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Supports: Scala, Python, SQL, Vega.<br>\nWritten in Scala and Python.<\/p>\n\n<p><a href=\"https:\/\/polynote.org\/\">Link<\/a><\/p>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--1gFrcySc--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/xg\/07\/vq\/xg07vqse7gzl-osoyk9znd9v_wa.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--1gFrcySc--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880\/https:\/\/habrastorage.org\/webt\/xg\/07\/vq\/xg07vqse7gzl-osoyk9znd9v_wa.gif\" alt=\"polynote\"><\/a><\/p>\n\n<h2>\n  \n  \n  mermaid-js\n<\/h2>\n\n<p>Draw graphs and charts as text! This is a must for writing a great documentation.<br>\nWritten in JavaScript.<\/p>\n\n<p><a href=\"https:\/\/mermaidjs.github.io\">Link<\/a><br>\nHas lots of plugins for different tools, like:<\/p>\n\n<ul>\n<li><a href=\"https:\/\/github.com\/mgaitan\/sphinxcontrib-mermaid\">Sphinx<\/a><\/li>\n<li><a href=\"https:\/\/marketplace.atlassian.com\/apps\/1214124\/mermaid-plugin-for-confluence\">Confluence<\/a><\/li>\n<li> Lot's of others<\/li>\n<\/ul>\n\n<p><a href=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--EDt7BB1v--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/l1\/xa\/5f\/l1xa5f6mi3twbexsk1gfhewo83a.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/res.cloudinary.com\/practicaldev\/image\/fetch\/s--EDt7BB1v--\/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880\/https:\/\/habrastorage.org\/webt\/l1\/xa\/5f\/l1xa5f6mi3twbexsk1gfhewo83a.jpeg\" alt=\"mermaid-js\"><\/a><\/p>\n\n<h2>\n  \n  \n  Bonus!\n<\/h2>\n\n<p>Here is the <a href=\"https:\/\/github.com\/palash25\/best-practices-checklist\">list of the best practices<\/a> for mostly any programming language. Why \"mostly\"? Because it, obviously, does not cover all <a href=\"https:\/\/codelani.com\/lists\/languages.html\">~3600<\/a> existing languages. But, you can always add your favourite language in case it is missing. PRs are welcome! <\/p>\n\n<p>That's it for today! Make sure to <a href=\"https:\/\/t.me\/opensource_findings\">subscribe<\/a> to my channel if you liked this list. I am going to post new ones each month. Also, feel free to post any cool projects you know in the comments. Feedback is always welcome.<\/p>\n\n","category":["python","javascript","go","swift"]},{"title":"Complexity Waterfall","pubDate":"Mon, 28 Oct 2019 08:14:07 +0000","link":"https:\/\/dev.to\/wemake-services\/complexity-waterfall-n2d","guid":"https:\/\/dev.to\/wemake-services\/complexity-waterfall-n2d","description":"<p><strong>Originally published in my blog<\/strong>: <a href=\"https:\/\/sobolevn.me\/2019\/10\/complexity-waterfall\" rel=\"noopener noreferrer\">https:\/\/sobolevn.me\/2019\/10\/complexity-waterfall<\/a><\/p>\n\n<p>When talking about \"bad code\" people almost certainly mean \"complex code\" among other popular problems. The thing about complexity is that it comes out of nowhere. One day you start your fairly simple project, the other day you find it in ruins. And no one knows how and when did it happen.<\/p>\n\n<p>But, this ultimately happens for a reason! Code complexity enters your codebase in two possible ways: with big chunks and incremental additions. And people are bad at reviewing and finding both of them.<\/p>\n\n<p>When a big chunk of code comes in, the reviewer will be challenged to find the exact location where the code is complex and what to do about it. Then, the review will have to prove the point: why this code is complex in the first place. And other developers might disagree. We all know these kinds of code reviews!<\/p>\n\n<p><a href=\"https:\/\/media.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fimgur.com%2Fc4HYCAi.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fimgur.com%2Fc4HYCAi.png\" alt=\"Number of lines for review and comments ratio\"><\/a><\/p>\n\n<p>The second way of complexity getting into your code is incremental addition: when you submit one or two lines to the existing function. And it is extremely hard to notice that your function was alright one commit ago, but now it is too complex. It takes a good portion of concentration, reviewing skill, and good code navigation practice to actually spot it. Most people (like me!) lack these skills and allow complexity to enter the codebase regularly.<\/p>\n\n<p>So, what can be done to prevent your code from getting complex? We need to use automation! Let's make a deep dive into the code complexity and ways to find and finally solve it.<\/p>\n\n<p>In this article, I will guide you through places where complexity lives and how to fight it there. Then we will discuss how well written simple code and automation enable an opportunity of \"Continous Refactoring\" and \"Architecture on Demand\" development styles.<\/p>\n\n<h2>\n  \n  \n  Complexity explained\n<\/h2>\n\n<p>One may ask: what exactly \"code complexity\" is? And while it sounds familiar, there are hidden obstacles in understanding the exact complexity location. Let's start with the most primitive parts and then move to higher-level entities.<\/p>\n\n<p>Remember, that this article is named \"Complexity Waterfall\"? I will show you how complexity from the simplest primitives overflows into the highest abstractions.<\/p>\n\n<p>I will use <code>python<\/code> as the main language for my examples and <a href=\"https:\/\/github.com\/wemake-services\/wemake-python-styleguide\" rel=\"noopener noreferrer\"><code>wemake-python-styleguide<\/code><\/a> as the main linting tool to find the violations in my code and illustrate my point.<\/p>\n\n<h3>\n  \n  \n  Expressions\n<\/h3>\n\n<p>All your code consists of simple expressions like <code>a + 1<\/code> and <code>print(x)<\/code>. While expressions themself are simple, they might unnoticeably overflow your code with complexity at some point. Example: imagine that you have a dictionary that represents some <code>User<\/code> model and you use it like so:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">format_username<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">username<\/span><span class=\"sh\">'<\/span><span class=\"p\">]:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">email<\/span><span class=\"sh\">'<\/span><span class=\"p\">]<\/span>\n    <span class=\"k\">elif<\/span> <span class=\"nf\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">username<\/span><span class=\"sh\">'<\/span><span class=\"p\">])<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">12<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">username<\/span><span class=\"sh\">'<\/span><span class=\"p\">][:<\/span><span class=\"mi\">12<\/span><span class=\"p\">]<\/span> <span class=\"o\">+<\/span> <span class=\"sh\">'<\/span><span class=\"s\">...<\/span><span class=\"sh\">'<\/span>\n    <span class=\"k\">return<\/span> <span class=\"sh\">'<\/span><span class=\"s\">@<\/span><span class=\"sh\">'<\/span> <span class=\"o\">+<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">username<\/span><span class=\"sh\">'<\/span><span class=\"p\">]<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>It looks pretty simple, doesn't it? In fact, it contains two expression-based complexity issues. It <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.OverusedStringViolation\" rel=\"noopener noreferrer\">overuses <code>'username'<\/code> string<\/a>  and uses <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/best_practices.html#wemake_python_styleguide.violations.best_practices.MagicNumberViolation\" rel=\"noopener noreferrer\">magic number<\/a> <code>12<\/code> (why do we use this number in the first place, why not <code>13<\/code> or <code>10<\/code>?). It is hard to find these kinds of things all by yourself. Here's how the better version would look like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\">#: That's how many chars fit in the preview box.\n<\/span><span class=\"n\">LENGTH_LIMIT<\/span><span class=\"p\">:<\/span> <span class=\"n\">Final<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">12<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">format_username<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">username<\/span> <span class=\"o\">=<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">username<\/span><span class=\"sh\">'<\/span><span class=\"p\">]<\/span>\n\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">username<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">'<\/span><span class=\"s\">email<\/span><span class=\"sh\">'<\/span><span class=\"p\">]<\/span>\n    <span class=\"k\">elif<\/span> <span class=\"nf\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">username<\/span><span class=\"p\">)<\/span> <span class=\"o\">&gt;<\/span> <span class=\"n\">LENGTH_LIMIT<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># See? It is now documented\n<\/span>        <span class=\"k\">return<\/span> <span class=\"n\">username<\/span><span class=\"p\">[:<\/span><span class=\"n\">LENGTH_LIMIT<\/span><span class=\"p\">]<\/span> <span class=\"o\">+<\/span> <span class=\"sh\">'<\/span><span class=\"s\">...<\/span><span class=\"sh\">'<\/span>\n    <span class=\"k\">return<\/span> <span class=\"sh\">'<\/span><span class=\"s\">@<\/span><span class=\"sh\">'<\/span> <span class=\"o\">+<\/span> <span class=\"n\">username<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>There are different problems with expression as well. We can also have <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.OverusedExpressionViolation\" rel=\"noopener noreferrer\">overused expressions<\/a>: when you use <code>some_object.some_attr<\/code> attribute everywhere instead of creating a new local variable. We can also have <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.TooManyConditionsViolation\" rel=\"noopener noreferrer\">too complex logic conditions<\/a> or <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.TooDeepAccessViolation\" rel=\"noopener noreferrer\">too deep dot access<\/a>.<\/p>\n\n<p><strong>Solution<\/strong>: create new variables, arguments, or constants. Create and use new utility functions or methods if you have to.<\/p>\n\n<h3>\n  \n  \n  Lines\n<\/h3>\n\n<p>Expressions form code lines (please, do not confuse lines with statements: single statement can take multiple lines and multiple statements might be located on a single line).<\/p>\n\n<p>The first and the most obvious complexity metric for a line is its length. Yes, you heard it correctly. That's why we (programmers) prefer to stick to <code>80<\/code> chars-per-line rule and not because it was <a href=\"https:\/\/en.wikipedia.org\/wiki\/Characters_per_line\" rel=\"noopener noreferrer\">previously used<\/a> in the teletypewriters. There are a lot of rumors about it lately, saying that it does not make any sence to use <code>80<\/code> chars for your code in 2k19. But, that's obviously not true.<\/p>\n\n<p>The idea is simple. You can have twice as much logic in a line with <code>160<\/code> chars than in line with only <code>80<\/code> chars. That's why this limit should be set and enforced. Remember, this is <em>not a stylistic choice<\/em>. It is a complexity metric!<\/p>\n\n<p>The second main line complexity metric is less known and less used. It is called <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.LineComplexityViolation\" rel=\"noopener noreferrer\">Jones Complexity<\/a>. The idea behind it is simple: we count code (or <code>ast<\/code>) nodes in a single line to get its complexity. Let's have a look at the example. These two lines are fundamentally different in terms of complexity but have the exact same width in chars:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">first_long_name_with_meaning<\/span><span class=\"p\">,<\/span> <span class=\"n\">second_very_long_name_with_meaning<\/span><span class=\"p\">,<\/span> <span class=\"n\">third<\/span><span class=\"p\">)<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">first<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">5<\/span> <span class=\"o\">+<\/span> <span class=\"n\">math<\/span><span class=\"p\">.<\/span><span class=\"n\">pi<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">matrix<\/span><span class=\"p\">.<\/span><span class=\"nf\">trans<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">matrix<\/span><span class=\"p\">),<\/span> <span class=\"n\">display<\/span><span class=\"p\">.<\/span><span class=\"nf\">show<\/span><span class=\"p\">(<\/span><span class=\"n\">matrix<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's count the nodes in the first one: one call, three names. Four nodes totally. The second one has twenty-one <code>ast<\/code> nodes. Well, the difference is clear. That's why we use Jones Complexity metric to allow the first long line and disallow the second one based on an internal complexity, not on just raw length.<\/p>\n\n<p>What to do with lines with a high Jones Complexity score?<\/p>\n\n<p><strong>Solution<\/strong>: Split them into several lines or create new intermediate variables, utility functions, new classes, etc.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"nf\">print<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">first<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">5<\/span> <span class=\"o\">+<\/span> <span class=\"n\">math<\/span><span class=\"p\">.<\/span><span class=\"n\">pi<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">matrix<\/span><span class=\"p\">.<\/span><span class=\"nf\">trans<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">matrix<\/span><span class=\"p\">),<\/span>\n    <span class=\"n\">display<\/span><span class=\"p\">.<\/span><span class=\"nf\">show<\/span><span class=\"p\">(<\/span><span class=\"n\">matrix<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">),<\/span>\n<span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Now it is way more readable!<\/p>\n\n<h3>\n  \n  \n  Structures\n<\/h3>\n\n<p>The next step is analyzing language structures like <code>if<\/code>, <code>for<\/code>, <code>with<\/code>, etc that are formed from lines and expressions. I have to say that this point is very language-specific. I'll showcase several rules from this category using <code>python<\/code> as well.<\/p>\n\n<p>We'll start with <code>if<\/code>. What can be easier than a good-old <code>if<\/code>? Actually, <code>if<\/code> starts to get tricky really fast. Here's an example of how one can <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.TooManyElifsViolation\" rel=\"noopener noreferrer\">reimplement <code>switch<\/code><\/a> with <code>if<\/code>:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">if<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">some<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n<span class=\"k\">elif<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">some<\/span><span class=\"p\">,<\/span> <span class=\"nb\">float<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n<span class=\"k\">elif<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">some<\/span><span class=\"p\">,<\/span> <span class=\"nb\">complex<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n<span class=\"k\">elif<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">some<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n<span class=\"k\">elif<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">some<\/span><span class=\"p\">,<\/span> <span class=\"nb\">bytes<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n<span class=\"k\">elif<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">some<\/span><span class=\"p\">,<\/span> <span class=\"nb\">list<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>What's the problem with this code? Well, imagine that we have tens of data types that should be covered including customs ones that we are not aware of yet. Then this complex code is an indicator that we are choosing a wrong pattern here. We need to refactor our code to fix this problem. For example, one can use <a href=\"https:\/\/github.com\/thejohnfreeman\/python-typeclasses\" rel=\"noopener noreferrer\"><code>typeclass<\/code>es<\/a> or <a href=\"https:\/\/docs.python.org\/3\/library\/functools.html#functools.singledispatch\" rel=\"noopener noreferrer\"><code>singledispatch<\/code><\/a>. They the same job, but nicer.<\/p>\n\n<p><code>python<\/code> never stops to amuse us. For example, you can write <code>with<\/code> with <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/consistency.html#wemake_python_styleguide.violations.consistency.MultipleContextManagerAssignmentsViolation\" rel=\"noopener noreferrer\">an arbitrary number of cases<\/a>, which is too mentally complex and confusing:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">with<\/span> <span class=\"nf\">first<\/span><span class=\"p\">(),<\/span> <span class=\"nf\">second<\/span><span class=\"p\">(),<\/span> <span class=\"nf\">third<\/span><span class=\"p\">(),<\/span> <span class=\"nf\">fourth<\/span><span class=\"p\">():<\/span>\n    <span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>You can also write comprehensions with any number of <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/consistency.html#wemake_python_styleguide.violations.consistency.MultipleIfsInComprehensionViolation\" rel=\"noopener noreferrer\"><code>if<\/code><\/a> and <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.TooManyForsInComprehensionViolation\" rel=\"noopener noreferrer\"><code>for<\/code><\/a> expressions, which can lead to complex, unreadable code:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"p\">[<\/span>\n    <span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">,<\/span> <span class=\"n\">z<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">x<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">x_coords<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">y<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">y_coords<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">z<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">z_coords<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">0<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">y<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">0<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">z<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">0<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">+<\/span> <span class=\"n\">y<\/span> <span class=\"o\">&lt;=<\/span> <span class=\"n\">z<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">+<\/span> <span class=\"n\">z<\/span> <span class=\"o\">&lt;=<\/span> <span class=\"n\">y<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">y<\/span> <span class=\"o\">+<\/span> <span class=\"n\">z<\/span> <span class=\"o\">&lt;=<\/span> <span class=\"n\">x<\/span>\n<span class=\"p\">]<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Compare it with the simple and readable version:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"p\">[<\/span>\n    <span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">,<\/span> <span class=\"n\">z<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">itertools<\/span><span class=\"p\">.<\/span><span class=\"nf\">product<\/span><span class=\"p\">(<\/span><span class=\"n\">x_coords<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_coords<\/span><span class=\"p\">,<\/span> <span class=\"n\">z_coords<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">if<\/span> <span class=\"nf\">valid_coordinates<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">,<\/span> <span class=\"n\">z<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">]<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>You can also accidentally include <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#wemake_python_styleguide.violations.complexity.TooLongTryBodyViolation\" rel=\"noopener noreferrer\">multiple statements inside a <code>try<\/code><\/a> case, which is unsafe because it can raise and handle an exception in an expected place:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">user<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">fetch_user<\/span><span class=\"p\">()<\/span>  <span class=\"c1\"># Can also fail, but don't expect that\n<\/span>    <span class=\"n\">log<\/span><span class=\"p\">.<\/span><span class=\"nf\">save_user_operation<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"n\">email<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># Can fail, and we know it\n<\/span><span class=\"k\">except<\/span> <span class=\"n\">MyCustomException<\/span> <span class=\"k\">as<\/span> <span class=\"n\">exc<\/span><span class=\"p\">:<\/span>\n    <span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And that's not even 10% of cases that can and will go wrong with your <code>python<\/code> code. There are many, many <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/violations\/complexity.html#summary\" rel=\"noopener noreferrer\">more edge cases<\/a> that should be tracked and analyzed.<\/p>\n\n<p><strong>Solution<\/strong>: The only possible solution is to use <a href=\"https:\/\/wemake-python-stylegui.de\" rel=\"noopener noreferrer\">a good linter<\/a> for the language of your choice. And refactor complex places that this linter highlights. Otherwise, you will have to reinvent the wheel and set custom policies for the exact same problems.<\/p>\n\n<h3>\n  \n  \n  Functions\n<\/h3>\n\n<p>Expressions, statements, and structures form functions. Complexity from these entities flows into functions. And that's where things start to get intriguing. Because functions have literally dozens of complexity metrics: both good and bad.<\/p>\n\n<p>We will start with the most known ones: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cyclomatic_complexity\" rel=\"noopener noreferrer\">cyclomatic complexity<\/a> and function's length measured in code lines. Cyclomatic complexity indicates how many turns your execution flow can take: it is almost equal to the number of unit tests that are required to fully cover the source code. It is a good metric because it respects the semantic and helps the developer to do the refactoring. On the other hand, a function's length is a bad metric. It does not coop with the previously explained Jones Complexity metric since we already know: multiple lines are easier to read than one big line with everything inside. We will concentrate on good metrics only and ignore bad ones.<\/p>\n\n<p>Based on my experience multiple useful complexity metrics should be counted instead of regular function's length:<\/p>\n\n<ul>\n<li>Number of function decorators; lower is better<\/li>\n<li>Number of arguments; lower is better<\/li>\n<li>Number of annotations; higher is better<\/li>\n<li>Number of local variables; lower is better<\/li>\n<li>Number of returns, yields, awaits; lower is better<\/li>\n<li>Number of statements and expressions; lower is better<\/li>\n<\/ul>\n\n<p>The combination of all these checks really allows you to write simple functions (all rules are also applied to methods as well).<\/p>\n\n<p>When you will try to do some nasty things with your function, you will surely break at least one metric. And this will disappoint our linter and blow your build. As a result, your function will be saved.<\/p>\n\n<p><strong>Solution<\/strong>: when one function is too complex, the only solution you have is to split this function into multiple ones.<\/p>\n\n<h3>\n  \n  \n  Classes\n<\/h3>\n\n<p>The next level of abstraction after functions are classes. And as you already guessed they are even more complex and fluid than functions. Because classes might contain multiple functions inside (that are called method) and have other unique features like inheritance and mixins, class-level attributes and class-level decorators. So, we have to check all methods as functions and the class body itself.<\/p>\n\n<p>For classes we have to measure the following metrics:<\/p>\n\n<ul>\n<li>Number of class-level decorators; lower is better<\/li>\n<li>Number of base classes; lower is better<\/li>\n<li>Number of class-level public attributes; lower is better<\/li>\n<li>Number of instance-level public attributes; lower is better<\/li>\n<li>Number of methods; lower is better<\/li>\n<\/ul>\n\n<p>When any of these is overly complicated - we have to ring the alarm and fail the build!<\/p>\n\n<p><strong>Solution<\/strong>: refactor your failed class! Split one existing complex class into several simple ones or create new utility functions and use composition.<\/p>\n\n<p>Notable mention: one can also track <a href=\"https:\/\/github.com\/mschwager\/cohesion\" rel=\"noopener noreferrer\">cohesion<\/a> and coupling <a href=\"https:\/\/stackoverflow.com\/questions\/3085285\/difference-between-cohesion-and-coupling\" rel=\"noopener noreferrer\">metrics<\/a> to validate the complexity of your OOP design.<\/p>\n\n<h3>\n  \n  \n  Modules\n<\/h3>\n\n<p>Modules do contain multiple statements, functions, and classes. And as you might have already mentioned we usually advise to split functions and classes into new ones. That's why we have to keep and eye on module complexity: it literally flows into modules from classes and functions.<\/p>\n\n<p>To analyze the complexity of the module we have to check:<\/p>\n\n<ul>\n<li>The number of imports and imported names; lower is better<\/li>\n<li>The number of classes and functions; lower is better<\/li>\n<li>The average complexity of functions and classes inside; lower is better<\/li>\n<\/ul>\n\n<p>What do we do in the case of a complex module?<\/p>\n\n<p><strong>Solution<\/strong>: yes, you got it right. We split one module into several ones.<\/p>\n\n<h3>\n  \n  \n  Packages\n<\/h3>\n\n<p>Packages contain multiple modules. Luckily, that's all they do.<\/p>\n\n<p>So, he number of modules in a package can soon start to be too large, so you will end up with too many of them. And it is the only complexity that can be found with packages.<\/p>\n\n<p><strong>Solution<\/strong>: you have to split packages into sub-packages and packages of different levels.<\/p>\n\n<h2>\n  \n  \n  Complexity Waterfall effect\n<\/h2>\n\n<p>We now have covered almost all possible types of abstractions in your codebase. What have we learned from it? The main takeaway, for now, is that most problems can be solved with ejecting complexity to the same or upper abstraction level.<\/p>\n\n<p><a href=\"https:\/\/media.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fimgur.com%2FOQLTbjV.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fimgur.com%2FOQLTbjV.png\" alt=\"Complexity Waterfall\"><\/a><\/p>\n\n<p>This leads us to the most important idea of this article: do not let your code be overflowed with the complexity. I will give several examples of how it usually happens.<\/p>\n\n<p>Imagine that you are implementing a new feature. And that's the only change you make:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight diff\"><code><span class=\"gi\">+++ if user.is_active and user.has_sub() and sub.is_due(tz.now() + delta):\n<\/span><span class=\"gd\">--- if user.is_active and user.has_sub():\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Looks ok, I would pass this code on review. And nothing bad would happen. But, the point I am missing is that complexity overflowed this line! That's what <code>wemake-python-styleguide<\/code> will report:<\/p>\n\n<p><a href=\"https:\/\/media.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fimgur.com%2FqjPTkH5.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fimgur.com%2FqjPTkH5.png\" alt=\"wemake-python-styleguide-output\"><\/a><\/p>\n\n<p>Ok, we now have to solve this complexity. Let's make a new variable:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">Product<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">can_be_purchased<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">user_id<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">...<\/span>\n\n        <span class=\"n\">is_sub_paid<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sub<\/span><span class=\"p\">.<\/span><span class=\"nf\">is_due<\/span><span class=\"p\">(<\/span><span class=\"n\">tz<\/span><span class=\"p\">.<\/span><span class=\"nf\">now<\/span><span class=\"p\">()<\/span> <span class=\"o\">+<\/span> <span class=\"n\">delta<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"n\">is_active<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"nf\">has_sub<\/span><span class=\"p\">()<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">is_sub_paid<\/span><span class=\"p\">:<\/span>\n            <span class=\"bp\">...<\/span>\n\n        <span class=\"bp\">...<\/span>\n\n<span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Now, the line complexity is solved. But, wait a minute. What if our function has too many variables now? Because we have created a new variable without checking their number inside the function first. In this case we will have to split this method into several ones like so:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">Product<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">can_be_purchased<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">user_id<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">...<\/span>\n\n        <span class=\"k\">if<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"nf\">_has_paid_sub<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">,<\/span> <span class=\"n\">sub<\/span><span class=\"p\">,<\/span> <span class=\"n\">delta<\/span><span class=\"p\">):<\/span>\n            <span class=\"bp\">...<\/span>\n\n        <span class=\"bp\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">_has_paid_sub<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">user<\/span><span class=\"p\">,<\/span> <span class=\"n\">sub<\/span><span class=\"p\">,<\/span> <span class=\"n\">delta<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">is_sub_paid<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sub<\/span><span class=\"p\">.<\/span><span class=\"nf\">is_due<\/span><span class=\"p\">(<\/span><span class=\"n\">tz<\/span><span class=\"p\">.<\/span><span class=\"nf\">now<\/span><span class=\"p\">()<\/span> <span class=\"o\">+<\/span> <span class=\"n\">delta<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"n\">is_active<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"nf\">has_sub<\/span><span class=\"p\">()<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">is_sub_paid<\/span>\n\n<span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Now we are done! Right? No, because we now have to check the complexity of the <code>Product<\/code> class. Imagine, that it now has too many methods since we have created a new <code>_has_paid_sub<\/code> one.<\/p>\n\n<p>Ok, we run our linter to check the complexity again. And turns out our <code>Product<\/code> class is indeed too complex right now. Our actions? We split it into several classes!<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">Policy<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">SubcsriptionPolicy<\/span><span class=\"p\">(<\/span><span class=\"n\">Policy<\/span><span class=\"p\">):<\/span>\n    <span class=\"bp\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">can_be_purchased<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">user_id<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">...<\/span>\n\n        <span class=\"k\">if<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"nf\">_has_paid_sub<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"p\">,<\/span> <span class=\"n\">sub<\/span><span class=\"p\">,<\/span> <span class=\"n\">delta<\/span><span class=\"p\">):<\/span>\n            <span class=\"bp\">...<\/span>\n\n        <span class=\"bp\">...<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">_has_paid_sub<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">user<\/span><span class=\"p\">,<\/span> <span class=\"n\">sub<\/span><span class=\"p\">,<\/span> <span class=\"n\">delta<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">is_sub_paid<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sub<\/span><span class=\"p\">.<\/span><span class=\"nf\">is_due<\/span><span class=\"p\">(<\/span><span class=\"n\">tz<\/span><span class=\"p\">.<\/span><span class=\"nf\">now<\/span><span class=\"p\">()<\/span> <span class=\"o\">+<\/span> <span class=\"n\">delta<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"n\">is_active<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">user<\/span><span class=\"p\">.<\/span><span class=\"nf\">has_sub<\/span><span class=\"p\">()<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">is_sub_paid<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Product<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">_purchasing_policy<\/span><span class=\"p\">:<\/span> <span class=\"n\">Policy<\/span>\n\n    <span class=\"bp\">...<\/span>\n\n<span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Please, tell me that it is the last iteration! Well, I am sorry, but we now have to check the module complexity. And guess what? We now have too many module members. So, we have to split modules into separate ones! Then we check the package complexity. And also possibly split it into several sub-packages.<\/p>\n\n<p>Have you seen it? Because of the well-defined complexity rules our single-line modification turned out to be a huge refactoring session with several new modules and classes. And we haven't made a single decision ourselves: all our refactoring goals were driven by the internal complexity and the linter that reveals it.<\/p>\n\n<p>That's what I call a \"Continuous Refactoring\" process. You are forced to do the refactoring. Always.<\/p>\n\n<p>This process also has one interesting consequence. It allows you to have \"Architecture on Demand\". Let me explain. With \"Architecture on Demand\" philosophy you always start small. For example with a single <code>logic\/domains\/user.py<\/code> file. And you start to put everything <code>User<\/code> related there. Because at this moment you probably don't know what your architecture will look like. And you don't care. You only have like three functions.<\/p>\n\n<p>Some people fall into architecture vs code complexity trap. They can overly-complicate their architecture from the very start with the full repository\/service\/domain layers. Or they can overly-complicate the source code with no clear separation. Struggle and live like this for years (if they will be able to live for years with the code like this!).<\/p>\n\n<p>\"Architecture on Demand\" concept solves these problems. You start small, when the time comes - you split and refactor things:<\/p>\n\n<ol>\n<li>You start with <code>logic\/domains\/user.py<\/code> and put everything in there<\/li>\n<li>Later you create <code>logic\/domains\/user\/repository.py<\/code> when you have enough database related stuff<\/li>\n<li>Then you split it into <code>logic\/domains\/user\/repository\/queries.py<\/code> and <code>logic\/domains\/user\/repository\/commands.py<\/code> when the complexity tells you to do so<\/li>\n<li>Then you create <code>logic\/domains\/user\/services.py<\/code> with <code>http<\/code> related stuff<\/li>\n<li>Then you create a new module called <code>logic\/domains\/order.py<\/code>\n<\/li>\n<li>And so on and so on<\/li>\n<\/ol>\n\n<p>That's it. It is a perfect tool to balance your architecture and code complexity. And get as much architecture as you truly need at the moment.<\/p>\n\n<h2>\n  \n  \n  Conclusion\n<\/h2>\n\n<p>Good linter does much more than finding missing commas and bad quotes. Good linter allows you to rely on it with architecture decisions and help you with the refactoring process.<\/p>\n\n<p>For example, <code>wemake-python-styleguide<\/code> might help you with the <code>python<\/code> source code complexity, it allows you to:<\/p>\n\n<ul>\n<li>Successfully fight the complexity at all levels<\/li>\n<li>Enforce the enormous amount of naming standards, best practices, and consistency checks<\/li>\n<li>Easily integrate it into a legacy code base with the help of <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/integrations\/legacy.html\" rel=\"noopener noreferrer\"><code>diff<\/code> option<\/a> or <a href=\"https:\/\/wemake-python-stylegui.de\/en\/latest\/pages\/usage\/integrations\/flakehell.html\" rel=\"noopener noreferrer\"><code>flakehell<\/code><\/a> tool, so old violation will be forgiven, but new ones won't be allowed<\/li>\n<li>Enable it into your <a href=\"\">CI<\/a>, even as a <a href=\"https:\/\/github.com\/marketplace\/actions\/wemake-python-styleguide\" rel=\"noopener noreferrer\">Github Action<\/a>\n<\/li>\n<\/ul>\n\n<p>Do not let the complexity to overflow your code, <a href=\"https:\/\/github.com\/wemake-services\/wemake-python-styleguide\" rel=\"noopener noreferrer\">use a good linter<\/a>!<\/p>\n\n","category":["python","beginners","webdev","productivity"]}]}}