{"@attributes":{"version":"2.0"},"channel":{"title":"Taylor Manderson","description":"Engineering leader with a decade plus experience in the infrastructure, platform, and data spaces\n","link":"https:\/\/trm.io\/","pubDate":"Fri, 25 Sep 2026 12:27:10 +0000","lastBuildDate":"Fri, 25 Sep 2026 12:27:10 +0000","generator":"Jekyll v3.9.0","item":[{"title":"\"The History of UQCS\" on The Router","description":"<p>I was interviewed this long weekend for the UQCS podcast, The Router, as they did a special episode on the history of UQCS, interviewing 6 past presidents.<\/p>\n\n<div id=\"video-container\" style=\"position:relative;width:100%;padding-top:56.25%;\">\n<iframe style=\"position: absolute; top:0;left:0;width:100%;height:100%;\" src=\"https:\/\/www.youtube.com\/embed\/6iyjP2covcs\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe>\n<\/div>\n\n<p>I\u2019ve put a comment on the video that includes times to skip to for each speaker, but I\u2019m at 5:37 if you just want to skip to my part.<\/p>\n","pubDate":"Mon, 04 Oct 2021 00:45:00 +0000","link":"https:\/\/trm.io\/2021\/10\/04\/uqcs-the-router.html","guid":"https:\/\/trm.io\/2021\/10\/04\/uqcs-the-router.html","category":["uqcs","Tech"]},{"title":"Task Queues: A Celery Story - PyConAU 2018","description":{"0":"\n"},"pubDate":"Sun, 26 Aug 2018 03:30:00 +0000","link":"https:\/\/trm.io\/2018\/08\/26\/pycon-task-queues.html","guid":"https:\/\/trm.io\/2018\/08\/26\/pycon-task-queues.html","category":["python","queues","types","Tech"]},{"title":"Gradual Typing in Python - PyConAU 2017","description":"<p>This weekend at PyCon I presented a talk on gradual typing in Python, specifically talking about the <code class=\"language-plaintext highlighter-rouge\">typing<\/code> library from Python 3.5 onwards, and the <code class=\"language-plaintext highlighter-rouge\">mypy<\/code> typechecker. The slides are <a href=\"https:\/\/docs.google.com\/presentation\/d\/1_LKkvn7Xta2w2f-eX3mB4j_zHhH_0IV7yq9_O90boMU\/\">available on Google Docs<\/a>.<\/p>\n\n<div id=\"video-container\" style=\"position:relative;width:100%;padding-top:56.25%;\">\n<iframe style=\"position: absolute; top:0;left:0;width:100%;height:100%;\" src=\"https:\/\/www.youtube.com\/embed\/mXfsMDM3LwQ\" frameborder=\"0\" gesture=\"media\" allow=\"encrypted-media\" allowfullscreen=\"\"><\/iframe>\n<\/div>\n\n<p>The conference on the whole was absolutely fantastic, and I had a great experience as both a speaker and attendee. Looking forward to submitting a proposal for PyCon AU 2018 in Sydney!<\/p>\n","pubDate":"Sun, 06 Aug 2017 00:45:00 +0000","link":"https:\/\/trm.io\/2017\/08\/06\/gradual-typing-talk.html","guid":"https:\/\/trm.io\/2017\/08\/06\/gradual-typing-talk.html","category":["python","types","pycon","Tech"]},{"title":"Extending Celery to Flexibly Support Custom Data Types","description":"<h2 id=\"background\">Background<\/h2>\n\n<p>Celery is Python\u2019s industrial-strength distributed and asynchronous task queue solution, and provides a convenient API for defining tasks that can be run on asynchronous worker nodes. At <a href=\"https:\/\/polymathian.com\">my work<\/a> we make heavy use of Celery for running our mathematical optimisation engines, and our web framework for building data visualisation apps, <a href=\"https:\/\/tropofy.com\/\">Tropofy<\/a>, has built-in support for Celery.<\/p>\n\n<h2 id=\"the-problem\">The Problem<\/h2>\n\n<p>By default, Celery uses JSON to serialise arguments passed to asynchronous function calls. Unfortunately, this means that any custom classes must use Pickle (<a href=\"https:\/\/blog.nelhage.com\/2011\/03\/exploiting-pickle\/\">which is considered insecure<\/a>), or define a custom serialiser. The latter locks you into a particular serialisation format, just for one extra data type, which I personally don\u2019t think is worth the trade off.<\/p>\n\n<h2 id=\"the-solution\">The Solution<\/h2>\n\n<p>The standard way to create Celery tasks is using the <code class=\"language-plaintext highlighter-rouge\">@app.task<\/code> decorator, which converts a function into an instance of a Celery <code class=\"language-plaintext highlighter-rouge\">Task<\/code> object. By subclassing <code class=\"language-plaintext highlighter-rouge\">Task<\/code> to alter the <code class=\"language-plaintext highlighter-rouge\">__call__<\/code> and <code class=\"language-plaintext highlighter-rouge\">apply_async<\/code> methods, we can provide enough extra information to help Celery out a little, at least where the thing we want to convert is passed in directly as an argument.<\/p>\n\n<p>Say I have a Celery app <code class=\"language-plaintext highlighter-rouge\">app<\/code> and a class <code class=\"language-plaintext highlighter-rouge\">A<\/code> that I want to serialise. I\u2019m using Python 3.6 syntax so I can use <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0526\/\">variable annotations<\/a>.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">celery<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Celery<\/span>\n\n<span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Celery<\/span><span class=\"p\">()<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">A<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">y<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">):<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">x<\/span> <span class=\"o\">=<\/span> <span class=\"n\">x<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">y<\/span> <span class=\"o\">=<\/span> <span class=\"n\">y<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>I can create a <code class=\"language-plaintext highlighter-rouge\">Task<\/code> subclass that will convert an instance of <code class=\"language-plaintext highlighter-rouge\">A<\/code> to something serialisable as follows:<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">celery<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Task<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">SerialisableAsyncCallTask<\/span><span class=\"p\">(<\/span><span class=\"n\">Task<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">apply_async<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">args<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">,<\/span> <span class=\"n\">kwargs<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args_<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs_<\/span><span class=\"p\">):<\/span>\n        <span class=\"c1\"># Convert every arg\n<\/span>        <span class=\"n\">args<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">list<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">[:])<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">idx<\/span><span class=\"p\">,<\/span> <span class=\"n\">arg<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">):<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">,<\/span> <span class=\"n\">A<\/span><span class=\"p\">):<\/span>\n                <span class=\"n\">args<\/span><span class=\"p\">[<\/span><span class=\"n\">idx<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"s\">'x'<\/span><span class=\"p\">:<\/span> <span class=\"n\">arg<\/span><span class=\"p\">.<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"s\">'y'<\/span><span class=\"p\">:<\/span> <span class=\"n\">arg<\/span><span class=\"p\">.<\/span><span class=\"n\">y<\/span><span class=\"p\">}<\/span>\n\n        <span class=\"k\">for<\/span> <span class=\"n\">key<\/span><span class=\"p\">,<\/span> <span class=\"n\">val<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">kwargs<\/span><span class=\"p\">.<\/span><span class=\"n\">items<\/span><span class=\"p\">():<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">,<\/span> <span class=\"n\">A<\/span><span class=\"p\">):<\/span>\n                <span class=\"n\">kwargs<\/span><span class=\"p\">[<\/span><span class=\"n\">key<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"s\">'x'<\/span><span class=\"p\">:<\/span> <span class=\"n\">val<\/span><span class=\"p\">.<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"s\">'y'<\/span><span class=\"p\">:<\/span> <span class=\"n\">val<\/span><span class=\"p\">.<\/span><span class=\"n\">y<\/span><span class=\"p\">}<\/span>\n\n        <span class=\"nb\">super<\/span><span class=\"p\">().<\/span><span class=\"n\">apply_async<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"o\">=<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"n\">kwargs<\/span><span class=\"o\">=<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args_<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs_<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>It\u2019s not crazy complex, it\u2019s just doing an instance check on every argument, then converting if needed. This will totally break my code every time a remote task attempts to run, because it\u2019s only converting one way. The above code is just to give you the gist, here\u2019s how I\u2019d actually approach doing this generically (including a full implementation).<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">importlib<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">itertools<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">abc<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">ABCMeta<\/span><span class=\"p\">,<\/span> <span class=\"n\">abstractmethod<\/span><span class=\"p\">,<\/span> <span class=\"n\">abstractclassmethod<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">celery<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Celery<\/span><span class=\"p\">,<\/span> <span class=\"n\">Task<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">AsSerialisable<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">,<\/span> <span class=\"n\">metaclass<\/span><span class=\"o\">=<\/span><span class=\"n\">ABCMeta<\/span><span class=\"p\">):<\/span>\n    <span class=\"o\">@<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">to_serialisable<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">):<\/span> <span class=\"k\">pass<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"n\">abstractclassmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">from_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">):<\/span> <span class=\"k\">pass<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">ExtendedSerialisableTask<\/span><span class=\"p\">(<\/span><span class=\"n\">Task<\/span><span class=\"p\">):<\/span>\n    <span class=\"o\">@<\/span><span class=\"nb\">staticmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">_convert_arg_to_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"p\">{<\/span>\n            <span class=\"s\">'__as_serialisable__'<\/span><span class=\"p\">:<\/span> <span class=\"bp\">True<\/span><span class=\"p\">,<\/span>\n            <span class=\"s\">'data'<\/span><span class=\"p\">:<\/span> <span class=\"n\">arg<\/span><span class=\"p\">.<\/span><span class=\"n\">to_serialisable<\/span><span class=\"p\">(),<\/span>\n            <span class=\"s\">'class'<\/span><span class=\"p\">:<\/span> <span class=\"n\">arg<\/span><span class=\"p\">.<\/span><span class=\"n\">__class__<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span><span class=\"p\">,<\/span>\n            <span class=\"s\">'module'<\/span><span class=\"p\">:<\/span> <span class=\"n\">arg<\/span><span class=\"p\">.<\/span><span class=\"n\">__class__<\/span><span class=\"p\">.<\/span><span class=\"n\">__module__<\/span><span class=\"p\">,<\/span>\n        <span class=\"p\">}<\/span>\n\n    <span class=\"o\">@<\/span><span class=\"nb\">staticmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">_convert_arg_from_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">module<\/span> <span class=\"o\">=<\/span> <span class=\"n\">importlib<\/span><span class=\"p\">.<\/span><span class=\"n\">import_module<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">[<\/span><span class=\"s\">'module'<\/span><span class=\"p\">])<\/span>\n        <span class=\"n\">cls<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">getattr<\/span><span class=\"p\">(<\/span><span class=\"n\">module<\/span><span class=\"p\">,<\/span> <span class=\"n\">arg<\/span><span class=\"p\">[<\/span><span class=\"s\">'class'<\/span><span class=\"p\">])<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">cls<\/span><span class=\"p\">.<\/span><span class=\"n\">from_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">[<\/span><span class=\"s\">'data'<\/span><span class=\"p\">])<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">apply_async<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">args<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">,<\/span> <span class=\"n\">kwargs<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args_<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs_<\/span><span class=\"p\">):<\/span>\n        <span class=\"c1\"># Convert every arg using `_convert_arg_to_serialisable`\n<\/span>        <span class=\"n\">args<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">list<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">[:])<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">idx<\/span><span class=\"p\">,<\/span> <span class=\"n\">arg<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">):<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">,<\/span> <span class=\"n\">AsSerialisable<\/span><span class=\"p\">):<\/span>\n                <span class=\"n\">args<\/span><span class=\"p\">[<\/span><span class=\"n\">idx<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_convert_arg_to_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"k\">for<\/span> <span class=\"n\">key<\/span><span class=\"p\">,<\/span> <span class=\"n\">val<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">kwargs<\/span><span class=\"p\">.<\/span><span class=\"n\">items<\/span><span class=\"p\">():<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">,<\/span> <span class=\"n\">A<\/span><span class=\"p\">):<\/span>\n                <span class=\"n\">kwargs<\/span><span class=\"p\">[<\/span><span class=\"n\">key<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_convert_arg_to_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"nb\">super<\/span><span class=\"p\">().<\/span><span class=\"n\">apply_async<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"o\">=<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"n\">kwargs<\/span><span class=\"o\">=<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args_<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs_<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__call__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n        <span class=\"c1\"># Unconvert every arg using `_convert_arg_from_serialisable`\n<\/span>        <span class=\"n\">args<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">list<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">idx<\/span><span class=\"p\">,<\/span> <span class=\"n\">arg<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">):<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">,<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">)<\/span> <span class=\"ow\">and<\/span> <span class=\"s\">'__as_serialisable__'<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">arg<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">args<\/span><span class=\"p\">[<\/span><span class=\"n\">idx<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_convert_arg_from_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">arg<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"k\">for<\/span> <span class=\"n\">key<\/span><span class=\"p\">,<\/span> <span class=\"n\">val<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">kwargs<\/span><span class=\"p\">.<\/span><span class=\"n\">items<\/span><span class=\"p\">():<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">,<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">)<\/span> <span class=\"ow\">and<\/span> <span class=\"s\">'__as_serialisable__'<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">val<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">kwargs<\/span><span class=\"p\">[<\/span><span class=\"n\">key<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_convert_arg_from_serialisable<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"k\">return<\/span> <span class=\"nb\">super<\/span><span class=\"p\">().<\/span><span class=\"n\">__call__<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">ExtendedCelery<\/span><span class=\"p\">(<\/span><span class=\"n\">Celery<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">task_cls<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ExtendedSerialisableTask<\/span>\n\n<span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ExtendedCelery<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>We\u2019re now using an abstract base class to register things we can convert to something serialisable, which means adding new serialisable classes is just a matter of subclassing (or metaclass registration). We now use class methods on the <code class=\"language-plaintext highlighter-rouge\">Task<\/code> subclass to convert to and from the serialisable representations, which means we can cleanly extend the serialised information with enough data to get the class object (which has the <code class=\"language-plaintext highlighter-rouge\">from_serialisable<\/code> method).<\/p>\n\n<p>Other than that, it\u2019s basically just arg checking and calling the conversion functions as per the second code block.<\/p>\n\n<h2 id=\"drawbacks\">Drawbacks<\/h2>\n\n<p>You can pass in as many <code class=\"language-plaintext highlighter-rouge\">AsSerialisable<\/code> objects as arguments as you want, and everything should work nicely. This solution was perfect for my use case and is fairly flexible going forwards, so I\u2019m more than happy to share it, but it\u2019s no panacea.<\/p>\n\n<p>Unfortunately, this doesn\u2019t traverse lists\/dictionaries\/etc to make any references to <code class=\"language-plaintext highlighter-rouge\">AsSerialisable<\/code>s serialisable. An extensible and generic API for serialisation could solve this problem to at least some degree, but that\u2019s more than a little out of the scope of this post.<\/p>\n","pubDate":"Sun, 26 Feb 2017 11:55:00 +0000","link":"https:\/\/trm.io\/2017\/02\/26\/celery-extend-serialisation.html","guid":"https:\/\/trm.io\/2017\/02\/26\/celery-extend-serialisation.html","category":["python","celery","serialisation","Tech"]},{"title":"Testing Out New Metaclass Support in mypy","description":{"0":"\n"},"pubDate":"Tue, 07 Feb 2017 06:30:00 +0000","link":"https:\/\/trm.io\/2017\/02\/07\/testing-new-mypy-metaclass.html","guid":"https:\/\/trm.io\/2017\/02\/07\/testing-new-mypy-metaclass.html","category":["python","mypy","types","Tech"]},{"title":"Lessons Learned from Not Documenting","description":"<p>In this post I\u2019m discussing lessons I\u2019ve learned about documentation where I really should have been doing something, but haven\u2019t been. Every single lesson is, at the time of writing, something I am about to implement, or have implemented within the last week. Every single one is something I see immense value in. Every single section has concrete implementation steps at the bottom.<\/p>\n\n<p><img src=\"\/uploads\/2017\/02\/deco-commit.png\" alt=\"Commit message\" \/><\/p>\n\n<p><em>Quality commit messages are part of documentation<\/em><\/p>\n\n<h2 id=\"documenting-processes\">Documenting Processes<\/h2>\n\n<p>Until recently, my role at work was primarily a devops one, and thus involved a whole lot of processes. Being young and reckless, I pretty consistently took the \u201cjust play with it until it works\u201d approach, which was honestly good enough at the time. Now, after stepping back and handing a lot of the devops workload to someone else, the importance of documenting processes has dawned on me.<\/p>\n\n<p>This lesson only really hit me with the recent <a href=\"https:\/\/docs.google.com\/document\/d\/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik\/pub\">Gitlab database incident<\/a>, where they lost their production database to a stray <code class=\"language-plaintext highlighter-rouge\">rm -rf<\/code> that was accidentally run on prod instead of staging. The incident itself wasn\u2019t what really drove the point home, it was <a href=\"https:\/\/news.ycombinator.com\/item?id=13537177\">this comment by gizmo<\/a> on the <a href=\"https:\/\/news.ycombinator.com\/item?id=13537052\">Hacker News thread<\/a>.<\/p>\n\n<blockquote>\n  <p>This is painful to read. It\u2019s easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that\u2019s far more important and easily missed.<\/p>\n\n  <p>When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize the steps. If there is no coworker, talk to your rubber ducky or stapler on your desk. This will help you catch mistakes. Then when the entire plan looks sensible, go through the steps one by one. Don\u2019t deviate from the plan. Don\u2019t get distracted and start switching between terminal windows. While making the checklist ask yourself if what you\u2019re doing is A) absolutely necessary and B) risks making things worse. Even when the angry emails are piling up you can\u2019t allow that pressure to cloud your judgment.<\/p>\n\n  <p>Every startup has moments when last-minute panic-patching of a critical part of the server infrastructure is needed, but if you use a checklist you\u2019re not likely to mess up badly, even when tired.<\/p>\n<\/blockquote>\n\n<p>This really seemed remarkably sane to me.<\/p>\n\n<p>When setting up a new machine, you should have a documented process, and ideally it should be automated. If it\u2019s automated, you\u2019ll still probably be invoking some arcane script that does some magic, and that magic should be documented. You should know exactly what each command does before you run it. When starting development on a new project, you should be able to follow a checklist to get everything up to speed, or automate it. When doing a production release, checklist or automate.<\/p>\n\n<p>Up until now I\u2019ve been basically of the mind that \u201cconfiguration management is all the documentation I need\u201d, which really isn\u2019t true at all. We\u2019re dealing with interacting components, not some immutable building block.<\/p>\n\n<p>My plan from now when developing a process:<\/p>\n<ul>\n  <li>Write down every action before I take it<\/li>\n  <li>Talk each action over with a colleague \u2013 they seem to be pretty good at spotting my stupid mistakes<\/li>\n  <li>Write down the result after I take each action<\/li>\n  <li>When I\u2019m done writing down and trying every step of the process, automate it<\/li>\n<\/ul>\n\n<p>Especially important:<\/p>\n<ul>\n  <li>I\u2019ll add human checks to the automated process, to be removed at a later date when confidence has been established.<\/li>\n<\/ul>\n\n<p>UPDATE 2017-02-06 11:30:00+10:00:<\/p>\n\n<p>In response to some discussion about this post on Facebook, <a href=\"http:\/\/blog.tmorris.net\/\">Tony Morris<\/a> had this to say:<\/p>\n\n<blockquote>\n  <p>As to checklists, this is taught heavily in aviation under the subject Human Factors. I regularly talk to myself when flying. In fact, I did it just this morning. However, I disagree with the advice of deviating from a plan. That just doesn\u2019t work. Be prepared for the change in plan.<\/p>\n\n  <p>I was given my landing clearance at 1500ft AGL, &lt;1 mile out this morning. I was planning to land, but you know, I have to deviate from that plan, or I\u2019d be sitting in a twisted aeroplane right now. I also had an unwell passenger. The skill is learning to adapt to changing plans, including unanticipated changes.<\/p>\n<\/blockquote>\n\n<p>This emphasises something very important that I didn\u2019t: things go wrong, and you have to be able to deal with that.<\/p>\n\n<h2 id=\"documenting-changes-and-versions\">Documenting Changes (and versions)<\/h2>\n\n<p>Up until recently, everyone at the office would participate in a daily standup, where everyone would hear what was going on in each project in the business \u2013 this included our shared data visualisation framework (where the bulk of my work is). The company has now grown large enough that everyone hearing everything is infeasible, so we\u2019ve broken the daily down into team meetings. The very first thing that was said after we first tried this was basically as follows:<\/p>\n\n<blockquote>\n  <p>Hey, seeing as we don\u2019t get to hear what you guys are working on day to day, it\u2019d be really good to get like an email or something to let us know what you\u2019ve been working on.<\/p>\n<\/blockquote>\n\n<p>So basically a changelog\u2026<\/p>\n\n<p>Being able to see what\u2019s changed in a version upgrade of a project, or even between commits (without having to read commit messages), is a really useful debugging tool, a nice way to communicate progress, and even an opportunity to <a href=\"https:\/\/api.slack.com\/changelog\">have a bit of fun<\/a>.<\/p>\n\n<p>The worst part is, <a href=\"http:\/\/semver.org\/\">versioning releases<\/a> and documenting changes is really such a trivial and useful thing. It\u2019s as simple as:<\/p>\n<ul>\n  <li>Keeping a \u201cCHANGELOG.md\u201d in your git repo, and making sure every pull request you do updates it (Haven\u2019t got a pull request workflow? Just start doing it and never merge your changes until people review them in a PR)<\/li>\n  <li>After a release, start a new heading for the next version, add your changes there<\/li>\n  <li>Need to do hotfixes? Start a branch from where you released last. Super simple stuff<\/li>\n  <li>Want to step it up a notch? Have a dev blog where you do reader-friendly writeups of large changes (or even writeups of challenging bugfixes)<\/li>\n<\/ul>\n\n<h2 id=\"high-level-documentation\">High-level documentation<\/h2>\n\n<p>I have a tendency to document my work at the API level, and no higher. I can plainly say that this is a recipe for disaster. When all you\u2019ve got are type annotations and a brief description of what a function does, it becomes very unclear how to actually interact with an API until you have a play with it and trigger a few errors. I think everyone has dealt with a poorly documented library at some stage, and I think this is a large part of what causes it.<\/p>\n\n<p><img src=\"\/uploads\/2017\/02\/documentation.png\" alt=\"Tropofy Fileframe\" title=\"One of the most commonly used APIs in work's web framework. Well documented, huh?\" \/><\/p>\n\n<p><em>One of the most commonly used APIs in work\u2019s web framework. Well documented, huh? Those types are it.<\/em><\/p>\n\n<p>Some of the best documentation I\u2019ve seen is the <a href=\"http:\/\/docs.sqlalchemy.org\/en\/rel_1_1\/\">SQLAlchemy reference<\/a>, which starts from the high level with basic usage, then drills down section by section about how to use various parts of the tool \u2013 ORM mappers, the Session API (not just API documentation, but how to use it, and how components interact). On top of this, there\u2019s <em>really<\/em> in-depth API documentation too.<\/p>\n\n<p>Pretty much everyone agrees that documentation is good, but pretty universally, people under-document. I have a slight suspicion that this is basically due to two things: underestimating <a href=\"https:\/\/wiki.lesswrong.com\/wiki\/Inferential_distance\">inferential distance<\/a> on work you\u2019re familiar with, and high cycle-time for documentation.<\/p>\n\n<p>The first is pretty concrete: people are familiar with the things they work on, and due to bog-standard human biases, we forget that other people aren\u2019t as familiar. On documentation cycle times, we frequently test our code, but it\u2019s far less common to do similar things with our documentation. In my workplace, we do documentation builds on our CI server but nobody even considers building documentation before pushing code, and nobody really reads it once it\u2019s out there (besides our poor, poor users).<\/p>\n\n<p>Both these issues are really hard to address, but I think there\u2019s a few things we can do at the very least.<\/p>\n<ul>\n  <li>Make sure every pull request that adds a new feature also adds new documentation<\/li>\n  <li>Ensure a minimum level of API documentation<\/li>\n  <li>Try and encourage code reviewers to also review your documentation<\/li>\n  <li>Force every changelog item (you\u2019re doing those now, right?) to include a link to documentation<\/li>\n<\/ul>\n\n<p>If I knew of any tools like this, I\u2019d no doubt be suggesting some sort of linter for documentation. I don\u2019t think a spelling and grammar check would do the job, mainly due to the large quantities of non-standard language that gets used in documentation.<\/p>\n\n<h2 id=\"final-thoughts\">Final Thoughts<\/h2>\n\n<p>I\u2019m really starting to see documentation as part of taking pride in my work, and part of being able to actually call my work \u201cfinished\u201d. It would be hypocritical of me to call out developers for writing unreliable software if I\u2019m here writing unmaintainable software, or establishing unsustainable processes due to sheer laziness. Getting documentation right really isn\u2019t too hard, and I\u2019ve really come to see that really is worth the effort.<\/p>\n","pubDate":"Sun, 05 Feb 2017 11:55:00 +0000","link":"https:\/\/trm.io\/2017\/02\/05\/documentation-lessons.html","guid":"https:\/\/trm.io\/2017\/02\/05\/documentation-lessons.html","category":["documentation","processes","Tech"]},{"title":"A Python-esque Type System for Python: Duck Typing Statically","description":"<p>I think the mypy static type checker is a fantastic initiative, and absolutely love it. My one complaint is that it relies a little too much on subclassing for determining compatibility. This post discusses nominal vs. structural subtyping, duck typing and how it relates to structural subtyping, subtyping in mypy, and using abstract base classes in lieu of a structural subtyping system. (Can you say \u201cstructural subtyping\u201d?)<\/p>\n\n<h2 id=\"nominative-and-structural-subtyping\">Nominative and Structural Subtyping<\/h2>\n\n<p>In OOP languages with type systems, we\u2019ve got this notion of \u201csubtyping\u201d or \u201csupertyping\u201d, that\u2019s pretty much exclusively based on inheritance. If you\u2019re comfortable with inheritance in OOP, you\u2019re comfortable with the idea of a \u201csubclass\u201d (pretty much by definition). I\u2019ll use Java as an example for this. In the following code example, <code class=\"language-plaintext highlighter-rouge\">B<\/code> is a subclass, and thus a subtype, of <code class=\"language-plaintext highlighter-rouge\">A<\/code>.<\/p>\n\n<div class=\"language-java highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kd\">class<\/span> <span class=\"nc\">A<\/span> <span class=\"o\">{<\/span>\n    <span class=\"kt\">int<\/span> <span class=\"nf\">x<\/span><span class=\"o\">(<\/span><span class=\"kt\">int<\/span> <span class=\"n\">y<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">y<\/span><span class=\"o\">;<\/span>\n    <span class=\"o\">}<\/span>\n<span class=\"o\">}<\/span>\n\n<span class=\"kd\">class<\/span> <span class=\"nc\">B<\/span> <span class=\"kd\">extends<\/span> <span class=\"no\">A<\/span><span class=\"o\">{<\/span>\n\n<span class=\"o\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>The idea of a subtyping relation has to deal with all the constructs in a type system though, which (in Java at least) also includes interfaces and abstract classes. For abstract classes, subtyping is still exactly subclassing. Whether or not you can instantiate your abstract class, it\u2019s still a class, and at type level, you can still subclass it. With interfaces, it\u2019s still fairly intuitive. A class <code class=\"language-plaintext highlighter-rouge\">A<\/code> is a subtype of interface <code class=\"language-plaintext highlighter-rouge\">I<\/code> if <code class=\"language-plaintext highlighter-rouge\">A implements I<\/code>. Nothing particularly complex.<\/p>\n\n<div class=\"language-java highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kd\">interface<\/span> <span class=\"nc\">I<\/span> <span class=\"o\">{<\/span>\n    <span class=\"kt\">int<\/span> <span class=\"nf\">x<\/span><span class=\"o\">(<\/span><span class=\"kt\">int<\/span><span class=\"o\">);<\/span>\n<span class=\"o\">}<\/span>\n\n<span class=\"kd\">class<\/span> <span class=\"nc\">A<\/span> <span class=\"kd\">implements<\/span> <span class=\"no\">I<\/span><span class=\"o\">{<\/span>\n    <span class=\"kt\">int<\/span> <span class=\"nf\">x<\/span><span class=\"o\">(<\/span><span class=\"kt\">int<\/span> <span class=\"n\">y<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">y<\/span><span class=\"o\">;<\/span>\n    <span class=\"o\">}<\/span>\n<span class=\"o\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>The subtyping relation in Java is what\u2019s described as \u201cnominative\u201d (name-based) subtyping, forming a \u201cnominal type system\u201d. Subtypes are determined by names and explicit declarations of compatibility or inheritance, but it\u2019s not the only option. There\u2019s also \u201cstructural\u201d subtyping, where <code class=\"language-plaintext highlighter-rouge\">A<\/code> being a subtype of <code class=\"language-plaintext highlighter-rouge\">B<\/code> is determined by the structure of both <code class=\"language-plaintext highlighter-rouge\">A<\/code> and <code class=\"language-plaintext highlighter-rouge\">B<\/code>. The best example I could come up with of a language with structural subtyping is Go, and I\u2019ve shamelessly stolen the following example from <a href=\"https:\/\/gobyexample.com\/interfaces\">Go By Example<\/a>.<\/p>\n\n<div class=\"language-go highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">import<\/span> <span class=\"s\">\"fmt\"<\/span>\n<span class=\"k\">import<\/span> <span class=\"s\">\"math\"<\/span>\n\n\n<span class=\"k\">type<\/span> <span class=\"n\">geometry<\/span> <span class=\"k\">interface<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">area<\/span><span class=\"p\">()<\/span> <span class=\"kt\">float64<\/span>\n    <span class=\"n\">perim<\/span><span class=\"p\">()<\/span> <span class=\"kt\">float64<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">type<\/span> <span class=\"n\">rect<\/span> <span class=\"k\">struct<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">width<\/span><span class=\"p\">,<\/span> <span class=\"n\">height<\/span> <span class=\"kt\">float64<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"k\">type<\/span> <span class=\"n\">circle<\/span> <span class=\"k\">struct<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">radius<\/span> <span class=\"kt\">float64<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">func<\/span> <span class=\"p\">(<\/span><span class=\"n\">r<\/span> <span class=\"n\">rect<\/span><span class=\"p\">)<\/span> <span class=\"n\">area<\/span><span class=\"p\">()<\/span> <span class=\"kt\">float64<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">width<\/span> <span class=\"o\">*<\/span> <span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">height<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"k\">func<\/span> <span class=\"p\">(<\/span><span class=\"n\">r<\/span> <span class=\"n\">rect<\/span><span class=\"p\">)<\/span> <span class=\"n\">perim<\/span><span class=\"p\">()<\/span> <span class=\"kt\">float64<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">return<\/span> <span class=\"m\">2<\/span><span class=\"o\">*<\/span><span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">width<\/span> <span class=\"o\">+<\/span> <span class=\"m\">2<\/span><span class=\"o\">*<\/span><span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">height<\/span>\n<span class=\"p\">}<\/span>\n\n\n<span class=\"k\">func<\/span> <span class=\"p\">(<\/span><span class=\"n\">c<\/span> <span class=\"n\">circle<\/span><span class=\"p\">)<\/span> <span class=\"n\">area<\/span><span class=\"p\">()<\/span> <span class=\"kt\">float64<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">math<\/span><span class=\"o\">.<\/span><span class=\"n\">Pi<\/span> <span class=\"o\">*<\/span> <span class=\"n\">c<\/span><span class=\"o\">.<\/span><span class=\"n\">radius<\/span> <span class=\"o\">*<\/span> <span class=\"n\">c<\/span><span class=\"o\">.<\/span><span class=\"n\">radius<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"k\">func<\/span> <span class=\"p\">(<\/span><span class=\"n\">c<\/span> <span class=\"n\">circle<\/span><span class=\"p\">)<\/span> <span class=\"n\">perim<\/span><span class=\"p\">()<\/span> <span class=\"kt\">float64<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">return<\/span> <span class=\"m\">2<\/span> <span class=\"o\">*<\/span> <span class=\"n\">math<\/span><span class=\"o\">.<\/span><span class=\"n\">Pi<\/span> <span class=\"o\">*<\/span> <span class=\"n\">c<\/span><span class=\"o\">.<\/span><span class=\"n\">radius<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">func<\/span> <span class=\"n\">measure<\/span><span class=\"p\">(<\/span><span class=\"n\">g<\/span> <span class=\"n\">geometry<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">fmt<\/span><span class=\"o\">.<\/span><span class=\"n\">Println<\/span><span class=\"p\">(<\/span><span class=\"n\">g<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">fmt<\/span><span class=\"o\">.<\/span><span class=\"n\">Println<\/span><span class=\"p\">(<\/span><span class=\"n\">g<\/span><span class=\"o\">.<\/span><span class=\"n\">area<\/span><span class=\"p\">())<\/span>\n    <span class=\"n\">fmt<\/span><span class=\"o\">.<\/span><span class=\"n\">Println<\/span><span class=\"p\">(<\/span><span class=\"n\">g<\/span><span class=\"o\">.<\/span><span class=\"n\">perim<\/span><span class=\"p\">())<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">func<\/span> <span class=\"n\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">r<\/span> <span class=\"o\">:=<\/span> <span class=\"n\">rect<\/span><span class=\"p\">{<\/span><span class=\"n\">width<\/span><span class=\"o\">:<\/span> <span class=\"m\">3<\/span><span class=\"p\">,<\/span> <span class=\"n\">height<\/span><span class=\"o\">:<\/span> <span class=\"m\">4<\/span><span class=\"p\">}<\/span>\n    <span class=\"n\">c<\/span> <span class=\"o\">:=<\/span> <span class=\"n\">circle<\/span><span class=\"p\">{<\/span><span class=\"n\">radius<\/span><span class=\"o\">:<\/span> <span class=\"m\">5<\/span><span class=\"p\">}<\/span>\n    <span class=\"n\">measure<\/span><span class=\"p\">(<\/span><span class=\"n\">r<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">measure<\/span><span class=\"p\">(<\/span><span class=\"n\">c<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>In the above code snippet, we\u2019ve defined an interface <code class=\"language-plaintext highlighter-rouge\">geometry<\/code>, and two types <code class=\"language-plaintext highlighter-rouge\">rect<\/code> and <code class=\"language-plaintext highlighter-rouge\">circle<\/code>. While Go likes to claim it doesn\u2019t actually have a subtype relation, for all intents and purposes <code class=\"language-plaintext highlighter-rouge\">rect<\/code> and <code class=\"language-plaintext highlighter-rouge\">circle<\/code> are subtypes of <code class=\"language-plaintext highlighter-rouge\">geometry<\/code> because there is a defined <code class=\"language-plaintext highlighter-rouge\">area<\/code> and <code class=\"language-plaintext highlighter-rouge\">perim<\/code> method for both of them, both of which match the type signature of <code class=\"language-plaintext highlighter-rouge\">geometry<\/code> for their respective types. Because <code class=\"language-plaintext highlighter-rouge\">rect<\/code> and <code class=\"language-plaintext highlighter-rouge\">circle<\/code> implement the necessary methods to match the <em>structure<\/em> of <code class=\"language-plaintext highlighter-rouge\">geometry<\/code>, they can be used in the <code class=\"language-plaintext highlighter-rouge\">measure<\/code> method, which accepts <code class=\"language-plaintext highlighter-rouge\">geometry<\/code> values.<\/p>\n\n<p>Put simply, because <code class=\"language-plaintext highlighter-rouge\">rect<\/code> and <code class=\"language-plaintext highlighter-rouge\">circle<\/code> match the structure of <code class=\"language-plaintext highlighter-rouge\">geometry<\/code>, they are a subtype of it, and this is structural subtyping. You might find it worthwhile to read more about structural subtyping on the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Structural_type_system\">Wikipedia page about \u201cStructural Type Systems\u201d<\/a>.<\/p>\n\n<p>Because Go doesn\u2019t provide generics, people make use of Go\u2019s structural type system to do nifty generic-like tricks. The most common such trick you\u2019ll see is <code class=\"language-plaintext highlighter-rouge\">type interface {}<\/code> which is sometimes called the \u201ctop type\u201d in Go, because everything is a structural subtype of it. When there\u2019s no necessary structure for a type, everything implements the necessary structure.<\/p>\n\n<h2 id=\"duck-typing\">Duck Typing<\/h2>\n\n<p>Duck typing is relatively poorly defined, but there\u2019s a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Duck_test\">common saying<\/a> used to describe how this works.<\/p>\n<blockquote>\n  <p>\u201cIf it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.\u201d<\/p>\n<\/blockquote>\n\n<p>My interpretation of duck typing is that you shouldn\u2019t check the type of what\u2019s passed into your functions \u2013 at most check that what\u2019s passed in has the methods and attributes that are necessary to do what you need, but you probably don\u2019t need to do that, as trying to use those things will error anyway if they\u2019re not there.<\/p>\n\n<p>Due to the informal definition of duck typing, a lot of people mix it up with structural subtyping. Both are talking about the structure of a type, checking the necessary methods and attributes are present, etc, but duck typing tends to have connotations of dynamic types (\u201cshouldn\u2019t check the type\u201d), whereas structural type systems specifically talk about subtyping relations, which implies static typing.<\/p>\n\n<p>The philosophy of duck typing is fairly heavily encouraged in Python-land. The most clear-cut example I could find of this is when looking for tips on \u201cchecking if an object is file-like\u201d. If you go to StackOverflow or the official Python docs, you\u2019ll see something along the lines of\u2026<\/p>\n\n<blockquote>\n  <p>Don\u2019t check if an object is file-like, check if it has methods like <code class=\"language-plaintext highlighter-rouge\">read<\/code> or <code class=\"language-plaintext highlighter-rouge\">write<\/code> using <code class=\"language-plaintext highlighter-rouge\">hasattr(obj, \"read\")<\/code>.<\/p>\n<\/blockquote>\n\n<p>If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.<\/p>\n\n<h2 id=\"structural-subtyping-in-the-mypy-type-checker\">Structural Subtyping in the mypy Type-checker<\/h2>\n<p>The subtype relation in mypy as it stands is basically just a nominative checker. You have to explicitly inherit from a superclass to be considered a subtype of it, pretty much like Java. However, there\u2019s been an <a href=\"https:\/\/github.com\/python\/typing\/issues\/11\">outstanding issue<\/a> discussing approaches to structural subtyping (which they call \u201cProtocols\u201d) since the early days of mypy. The meat of the proposed approach is in <a href=\"https:\/\/github.com\/python\/typing\/issues\/11#issuecomment-138133867\">this comment<\/a>, but I\u2019ll summarise here.<\/p>\n\n<p>Protocols will be declared as separate classes, inheriting from a magic <code class=\"language-plaintext highlighter-rouge\">Protocol<\/code> base class. Rather than using <code class=\"language-plaintext highlighter-rouge\">isinstance<\/code> to check if something implements a protocol (read: \u201cis a structural subtype of\u201d), there\u2019ll be some separate checker function like <code class=\"language-plaintext highlighter-rouge\">implements<\/code>. Runtime implementation checks are performance expensive, and Protocols are intended to make static checking easier, not necessarily as a runtime representation of an API (though that functionality is still useful), hence the separate method.<\/p>\n\n<p>Protocols will be extensible via subclassing (though you\u2019ll have to redeclare that you\u2019re subclassing the <code class=\"language-plaintext highlighter-rouge\">Protocol<\/code> base class, or it\u2019ll be assumed you\u2019re just implementing the given protocol), will be allowed to be generic, will support declaring both methods and attributes to implement. Here\u2019s an example of the syntax:<\/p>\n\n<p>Protocols:<\/p>\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Protocol<\/span> <span class=\"c1\"># not actually there yet\n<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">FooProtocol<\/span><span class=\"p\">(<\/span><span class=\"n\">Protocol<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">attr<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">foo<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span> <span class=\"p\">...<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">FooConcrete<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">attr<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">attr<\/span> <span class=\"o\">=<\/span> <span class=\"n\">attr<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">foo<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">x<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">fn_on_foo<\/span><span class=\"p\">(<\/span><span class=\"n\">foo_obj<\/span><span class=\"p\">:<\/span> <span class=\"n\">FooProtocol<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">foo_obj<\/span><span class=\"p\">.<\/span><span class=\"n\">foo<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">foo<\/span> <span class=\"o\">=<\/span> <span class=\"n\">FooConcrete<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">fn_on_foo<\/span><span class=\"p\">(<\/span><span class=\"n\">foo<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Unfortunately, none of this is concrete and certain, so the above syntax is a combination of my best guess and what I\u2019d like the syntax to end up like.<\/p>\n\n<h2 id=\"the-stop-gap-measure\">The Stop-gap Measure<\/h2>\n\n<p>While I wait for protocols to be firmed up in the form of a PEP, I\u2019ve come up with a stop-gap measure that I\u2019m relatively happy with. I\u2019ve been making use of <a href=\"https:\/\/docs.python.org\/3\/library\/abc.html\">abstract base classes<\/a>, which provide the ability to register classes that don\u2019t directly inherit from them (better than nominative subtyping, not quite as good as structural subtyping). As an example, I\u2019ve included below an example of how I\u2019d write the protocol from above as an abstract base class.<\/p>\n\n<p>Abstract Base Classes:<\/p>\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">abc<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">ABC<\/span><span class=\"p\">,<\/span> <span class=\"n\">abstractmethod<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">FooBase<\/span><span class=\"p\">(<\/span><span class=\"n\">ABC<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">attr<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">None<\/span> <span class=\"c1\"># type: int\n<\/span>    <span class=\"o\">@<\/span><span class=\"n\">abstractmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">foo<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span> <span class=\"p\">...<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">FooConcrete<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">attr<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">attr<\/span> <span class=\"o\">=<\/span> <span class=\"n\">attr<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">foo<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">x<\/span>\n\n<span class=\"n\">FooBase<\/span><span class=\"p\">.<\/span><span class=\"n\">register<\/span><span class=\"p\">(<\/span><span class=\"n\">FooConcrete<\/span><span class=\"p\">)<\/span> <span class=\"c1\"># error: \"FooBase\" has no attribute \"register\"\n<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">fn_on_foo<\/span><span class=\"p\">(<\/span><span class=\"n\">foo_obj<\/span><span class=\"p\">:<\/span> <span class=\"n\">FooBase<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">foo_obj<\/span><span class=\"p\">.<\/span><span class=\"n\">foo<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">foo<\/span> <span class=\"o\">=<\/span> <span class=\"n\">FooConcrete<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">fn_on_foo<\/span><span class=\"p\">(<\/span><span class=\"n\">foo<\/span><span class=\"p\">)<\/span>  <span class=\"c1\"># error: Argument 1 to \"fn_on_foo\" has incompatible type \"FooConcrete\"; expected \"FooBase\"\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>Very similar code, just with a different base class, an extra decorator, and explicit registration. While these work just as well as runtime, I\u2019ve included the error output from mypy as comments. Not quite as type-checker-friendly. This case is a little contrived, seeing as I could\u2019ve just inherited from <code class=\"language-plaintext highlighter-rouge\">FooBase<\/code> to start with and everything would\u2019ve been lovely, but you can see why I\u2019m calling this a stop-gap solution and nothing more.<\/p>\n\n<h2 id=\"final-thoughts\">Final Thoughts<\/h2>\n\n<p>In general, you can pretty comfortably get away with the nominal subtyping that mypy happily type checks right now, but it does push you in a direction that\u2019s slightly less Pythonic. Given Python\u2019s emphasis on duck typing, it makes a lot of sense to provide some sort of structural subtyping too. There\u2019s been a little bit of interest lately on the <a href=\"https:\/\/gitter.im\/python\/mypy\">mypy gitter chat<\/a> to do with structural subtyping, so hopefully the situation will start improving soon.<\/p>\n","pubDate":"Sun, 29 Jan 2017 10:55:00 +0000","link":"https:\/\/trm.io\/2017\/01\/29\/structural-subtyping-python.html","guid":"https:\/\/trm.io\/2017\/01\/29\/structural-subtyping-python.html","category":["python","types","mypy","Tech"]},{"title":"Why Proving that Addition is Commutative is Worth Your Time","description":"<p>I\u2019ve recently been working my way through the fantastic book <a href=\"https:\/\/www.cis.upenn.edu\/~bcpierce\/sf\/current\/index.html\">Software Foundations<\/a>, primarily written by Benjamin C. Pierce, which focuses on how to build reliable software. To illustrate the content, here\u2019s an excerpt from the preface:<\/p>\n\n<blockquote>\n  <p>Building reliable software is hard. The scale and complexity of modern systems, the number of people involved in building them, and the range of demands placed on them render it extremely difficult to build software that is even more-or-less correct, much less 100% correct. At the same time, the increasing degree to which information processing is woven into every aspect of society continually amplifies the cost of bugs and insecurities.<\/p>\n\n  <p>Computer scientists and software engineers have responded to these challenges by developing a whole host of techniques for improving software reliability, ranging from recommendations about managing software projects and organizing programming teams (e.g., extreme programming) to design philosophies for libraries (e.g., model-view-controller, publish-subscribe, etc.) and programming languages (e.g., object-oriented programming, aspect-oriented programming, functional programming, \u2026) to mathematical techniques for specifying and reasoning about properties of software and tools for helping validate these properties.<\/p>\n<\/blockquote>\n\n<h2 id=\"the-question\">The Question<\/h2>\n\n<p>I haven\u2019t made it that far through the book yet, but what I been working through is the basics of <a href=\"https:\/\/coq.inria.fr\/\">Coq, a formal proof assistant<\/a>, which is the tool the book uses to teach concepts. I made my start during a recent work trip, where I made it through the first few chapters. Discussion about what I was up to pretty commonly went as follows:<\/p>\n\n<blockquote>\n  <p>Colleague: What are you up to, Tom?<\/p>\n\n  <p>Me: Right now? Proving that addition is <a href=\"https:\/\/en.wikipedia.org\/wiki\/Commutative_property\">commutative<\/a>.<\/p>\n\n  <p>C: I would\u2019ve figured that\u2019s true by definition<\/p>\n\n  <p>M: Uh, yeah, but, uh, I guess I still have to prove it?<\/p>\n<\/blockquote>\n\n<p>I started Software Foundations for fun, without really having a goal in sight, but discussions like that made me question what I was doing. Why on earth was I bothering to prove something that\u2019s so obvious?<\/p>\n\n<h2 id=\"the-answer\">The Answer<\/h2>\n\n<p>In order to use something in a program, you have to implement it. We all know how addition works (I hope), but there are some inherent assumptions about that understanding (addition on real numbers? on 32 bit integers? signed or unsigned?). When using formal methods to verify our program (like proofs), we are verifying that our implementation is congruent with those inherent assumptions.<\/p>\n\n<p>Sure, I know that, by definition, addition is commutative, but is my specific implementation of addition on inductively defined natural numbers <em>really<\/em> commutative? The standard approach to being confident about something like this is to come up with test cases. This gives you a lot of benefits, but the only way to be sure that what you\u2019re trying to verify works for 100% of cases is to use formal methods.<\/p>\n\n<p>Do we really need 100% certainty for addition being commutative? I\u2019d wager that it\u2019s not really the case for most uses, but in order to prove that your webapp does what you think it does, you\u2019ll almost definitely build your functionality from smaller functions. Correspondingly, you\u2019d also be building up your proofs from smaller proofs (like a proof of the commutativity of addition, for example). While you\u2019ll likely have the proof of commutativity of addition provided to you, it\u2019s a valuable learning exercise to work through it yourself.<\/p>\n\n<p>Often, it\u2019s about as much time to write a proof as it is to write enough test cases to cover a reasonable amount of your code, and you can take a lot more certainty from it. As your software gets more complex, it becomes both harder to test, and harder to formally verify, but unlike your test cases which cover less and less of your functionality, proofs are always 100%. When reliability is key, you can\u2019t trump formal methods.<\/p>\n\n<h2 id=\"the-real-world-answer\">The Real-World Answer<\/h2>\n\n<p>There\u2019s currently an effort underway to <a href=\"https:\/\/github.com\/idris-hackers\/software-foundations\">rewrite Software Foundations<\/a> for the <a href=\"http:\/\/www.idris-lang.org\/\">Idris programming language<\/a>. Idris is a Haskell-like, dependently typed language with the ability to <a href=\"http:\/\/docs.idris-lang.org\/en\/latest\/tutorial\/theorems.html\">state and prove theorems<\/a>. I think it\u2019s fantastic that we\u2019ve got a general purpose programming language with formal methods built right in, but what I think is more immediately useful for the average programmer is dependent types.<\/p>\n\n<blockquote>\n  <p>Dependent types allow types to be predicated on values, meaning that some aspects of a program\u2019s behaviour can be specified precisely in the type.<\/p>\n<\/blockquote>\n\n<p>Sounds a lot like some sort of method for verifying program behaviour, doesn\u2019t it? Funnily enough, there\u2019s a proof that proofs of program behaviour are mathematically equivalent to dependent types, called the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Curry%E2%80%93Howard_correspondence\">Curry-Howard Isomorphism<\/a>. In practice, however, you can be a little bit more piecemeal with dependent types, and slowly add more details as you go. You can be 100% certain that whatever your dependent type says is going to hold at runtime, and it\u2019s generally easier and more concise to do verify aspects of your program with your type system than with formal methods or tests.<\/p>\n\n<p>The reason Idris is worth mentioning is because it allows you to make use of proofs when it <em>is<\/em> useful, and in other situations you can just rely on type system features like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Kind_(type_theory)#Kinds_in_Haskell\">higher-kinded types<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Algebraic_data_type\">algebraic datatypes<\/a>, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Dependent_type\">dependent types<\/a>, which all let you get just a little bit extra certainty in your program. While I\u2019d love to go into more detail on each of the above, they\u2019d probably need a post of their own, for now I\u2019ve linked the respective Wikipedia pages.<\/p>\n\n<p>When reliability is a concern in your software, consider using formal methods before jumping right to tests. Surprisingly enough, you\u2019ll probably get more reliability from less code. If you\u2019re interested in learning more, I wholeheartedly recommend Software Foundations, and feel free to send me an email or get in touch via twitter.<\/p>\n","pubDate":"Sun, 22 Jan 2017 11:05:00 +0000","link":"https:\/\/trm.io\/2017\/01\/22\/proving-things-worth-your-time.html","guid":"https:\/\/trm.io\/2017\/01\/22\/proving-things-worth-your-time.html","category":["proofs","coq","mathematics","Tech"]},{"title":"An Overview of Types in Python","description":"<p>While Python is a dynamically typed language, from Python 3.5 onwards there\u2019s been an effort to add the ability to use types within the language. Python 3.4 added the ability to add any arbitrary Python expression to the arguments in a function with <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-3107\/\">PEP 3107 (Function Annotations)<\/a>, and this was expanded upon in Python 3.5 with <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0484\/\">PEP 484 (Type Hints)<\/a>, which aimed to provide a standard syntax for declaring types in Python. In the \u201cRationale\u201d section of PEP 484, it is made very clear that \u201cPython will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention\u201d.<\/p>\n\n<p>This standardised language for types in Python made the <a href=\"http:\/\/mypy-lang.org\/\">Mypy project<\/a> possible - a static typechecker for Python programs with type annotations. At runtime, these type annotations do absolutely nothing (as checking and coercing have a huge runtime overhead), but allow programs like <code class=\"language-plaintext highlighter-rouge\">mypy<\/code> to function.<\/p>\n\n<h2 id=\"why-types\">Why Types?<\/h2>\n\n<p>Types give us the ability to reason about our programs more easily, by restricting the inputs and outputs of our functions to only those things our code actually knows how to deal with. The more sophisticated your type system, the more nuanced you can make these restrictions, eventually reaching the point where you can do cool things like type-level programming.<\/p>\n\n<p>Type checking can eliminate whole classes of type-related bugs, to the point where languages with sophisticated type systems like Haskell can, very much validly, make claims like \u201cif it compiles, it probably runs\u201d. Silly mistakes and typos are almost always caught by the typechecker.<\/p>\n\n<p>The other advantage of types in languages that are statically typed is efficiency gains - if you know what type something is before you use it, you can eliminate a lot of code just to do with getting at the values you need. Unfortunately, this will never be the case for Python, as PEP 484 (Type Hints) explicitly states \u201cPython will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention\u201d. If you\u2019re desperate for the runtime performance gains that static types bring you, <a href=\"http:\/\/cython.org\/\">Cython<\/a> is the way to go - it allows you to provide specialised annotations that lets some of your Python be compiled into C.<\/p>\n\n<h2 id=\"lets-look-at-some-code\">Let\u2019s look at some code<\/h2>\n\n<p>If you\u2019ve used a language like Java, you pretty much know what to expect when it comes to the basics. Generics, classes, etc, etc. Here\u2019s two code samples to compare<\/p>\n\n<p>Java:<\/p>\n<div class=\"language-java highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kd\">class<\/span> <span class=\"nc\">A<\/span> <span class=\"o\">{<\/span>\n    <span class=\"kd\">private<\/span> <span class=\"kt\">int<\/span> <span class=\"n\">x<\/span><span class=\"o\">;<\/span>\n    <span class=\"no\">A<\/span><span class=\"o\">(<\/span><span class=\"kt\">int<\/span> <span class=\"n\">x<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">this<\/span><span class=\"o\">.<\/span><span class=\"na\">x<\/span> <span class=\"o\">=<\/span> <span class=\"n\">x<\/span><span class=\"o\">;<\/span>\n    <span class=\"o\">}<\/span>\n<span class=\"o\">}<\/span>\n\n<span class=\"kd\">class<\/span> <span class=\"nc\">B<\/span> <span class=\"kd\">extends<\/span> <span class=\"no\">A<\/span><span class=\"o\">{<\/span>\n<span class=\"o\">}<\/span>\n\n<span class=\"kd\">class<\/span> <span class=\"nc\">C<\/span> <span class=\"o\">{<\/span>\n    <span class=\"o\">&lt;<\/span><span class=\"no\">T<\/span><span class=\"o\">&gt;<\/span> <span class=\"kd\">static<\/span> <span class=\"no\">T<\/span> <span class=\"nf\">method<\/span><span class=\"o\">(<\/span><span class=\"no\">T<\/span> <span class=\"n\">a<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">a<\/span><span class=\"o\">;<\/span>\n    <span class=\"o\">}<\/span>\n\n    <span class=\"o\">&lt;<\/span><span class=\"no\">T<\/span><span class=\"o\">&gt;<\/span> <span class=\"kd\">static<\/span> <span class=\"nc\">List<\/span><span class=\"o\">&lt;<\/span><span class=\"no\">T<\/span><span class=\"o\">&gt;<\/span> <span class=\"nf\">method<\/span><span class=\"o\">(<\/span><span class=\"no\">T<\/span> <span class=\"n\">a<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">return<\/span> <span class=\"nc\">Arrays<\/span><span class=\"o\">.<\/span><span class=\"na\">asList<\/span><span class=\"o\">(<\/span><span class=\"n\">a<\/span><span class=\"o\">);<\/span>\n    <span class=\"o\">}<\/span>\n<span class=\"o\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Python:<\/p>\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">,<\/span> <span class=\"n\">List<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">A<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">x<\/span> <span class=\"c1\"># type: int\n<\/span>    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">x<\/span> <span class=\"o\">=<\/span> <span class=\"n\">x<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">B<\/span><span class=\"p\">(<\/span><span class=\"n\">A<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">pass<\/span>\n\n<span class=\"n\">T<\/span> <span class=\"o\">=<\/span> <span class=\"n\">TypeVar<\/span><span class=\"p\">(<\/span><span class=\"s\">'T'<\/span><span class=\"p\">,<\/span> <span class=\"n\">covariant<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">C<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span>\n    <span class=\"o\">@<\/span><span class=\"nb\">staticmethod<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">method<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span><span class=\"p\">:<\/span> <span class=\"n\">T<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">T<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">a<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">method2<\/span><span class=\"p\">(<\/span><span class=\"n\">a<\/span><span class=\"p\">:<\/span> <span class=\"n\">T<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">List<\/span><span class=\"p\">[<\/span><span class=\"n\">T<\/span><span class=\"p\">]:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"p\">[<\/span><span class=\"n\">a<\/span><span class=\"p\">]<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>I\u2019ve snuck in a bit of Python weirdness without mentioning. As PEP 3107 (Function Annotations) declares that function annotations must be valid Python expressions, we can\u2019t use undefined variables etc, so generics become a little bit more verbose. The <code class=\"language-plaintext highlighter-rouge\">T = TypeVar('T')<\/code> declares a \u201ctype variable\u201d, and when used in a method, all usages of <code class=\"language-plaintext highlighter-rouge\">T<\/code> as a type annotation are forced to represent the same type.<\/p>\n\n<p>Type variables have a concept called \u201cvariance\u201d, which limits what can be substituted in for them. A \u201ccovariant\u201d type variable lets any usages of it be either the same type as, or a \u201csubtype\u201d (subclass) of the type that the variable is standing in for. A \u201ccontravariant\u201d type variable makes sure that the type it is standing in for is a subtype\/subclass of every usage. An \u201cinvariant\u201d type variable must have every usage being exactly the same type as whatever it\u2019s standing in for.<\/p>\n\n<p>Generics in Python, like list, are declared using <code class=\"language-plaintext highlighter-rouge\">Generic[T]<\/code> as a base class instead of <code class=\"language-plaintext highlighter-rouge\">object<\/code>, where the <code class=\"language-plaintext highlighter-rouge\">T<\/code> is a <code class=\"language-plaintext highlighter-rouge\">TypeVar<\/code> declared before the class is, and this also limits what a generic type can contain according to the <code class=\"language-plaintext highlighter-rouge\">TypeVar<\/code>\u2019s variance.<\/p>\n\n<p>There\u2019s way more info on the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Covariance_and_contravariance_(computer_science)\">wikipedia page about type variance<\/a>, but for now, here\u2019s an example for lists, which have an<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">List<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">A<\/span><span class=\"p\">(<\/span><span class=\"nb\">object<\/span><span class=\"p\">):<\/span> <span class=\"k\">pass<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">B<\/span><span class=\"p\">(<\/span><span class=\"n\">A<\/span><span class=\"p\">):<\/span> <span class=\"k\">pass<\/span>\n\n<span class=\"n\">x<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[]<\/span> <span class=\"c1\"># type: List[A]\n<\/span>\n<span class=\"c1\"># valid, appending an A to a List[A]\n<\/span><span class=\"n\">x<\/span><span class=\"p\">.<\/span><span class=\"n\">append<\/span><span class=\"p\">(<\/span><span class=\"n\">A<\/span><span class=\"p\">())<\/span>\n\n<span class=\"c1\"># errors, lists are invariant over the type they contain, so a subclass of A is\n# not valid, even though it would be in Java.\n<\/span><span class=\"n\">x<\/span><span class=\"p\">.<\/span><span class=\"n\">append<\/span><span class=\"p\">(<\/span><span class=\"n\">B<\/span><span class=\"p\">())<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h3 id=\"other-differences\">Other Differences<\/h3>\n\n<p>Because Python is dynamically typed, you get functions that accept inputs with all kinds of weird features, and you get functions that the type system isn\u2019t sophisticated enough to accurately represent. For situations like that, Python has two extra type system features you won\u2019t find in normal statically typed languages: <code class=\"language-plaintext highlighter-rouge\">Union<\/code> and <code class=\"language-plaintext highlighter-rouge\">Any<\/code>.<\/p>\n\n<p>Releatively unsurprisingly, <code class=\"language-plaintext highlighter-rouge\">Any<\/code> is considered valid a valid type for absolutely any. It\u2019s kind of the active opt-out of the type system, and lets you get out of type checking. When you\u2019re starting to add type annotations to a code base, or coming across those weird unrepresentable situations, <code class=\"language-plaintext highlighter-rouge\">Any<\/code> gives you an out.<\/p>\n\n<p><code class=\"language-plaintext highlighter-rouge\">Union<\/code> allows a value to be any one of a given set of types, and type check correctly for any of them. For example both <code class=\"language-plaintext highlighter-rouge\">\"abcd\"<\/code> and <code class=\"language-plaintext highlighter-rouge\">1234<\/code> are valid inputs for a function with an argument that takes a type of <code class=\"language-plaintext highlighter-rouge\">Union[int, str]<\/code>. I\u2019ve seen <code class=\"language-plaintext highlighter-rouge\">Union<\/code> used to represent types that have a common property before, which is often not the right approach, as there are a bunch of abstract base classes provided by the <code class=\"language-plaintext highlighter-rouge\">typing<\/code> module that can be used instead if it\u2019s common enough. The main other situation I see <code class=\"language-plaintext highlighter-rouge\">Union<\/code> used for is a form of user-controlled dynamic dispatch which you\u2019d get for free in Java. Here\u2019s some more code samples to demonstate the difference:<\/p>\n\n<p>Java:<\/p>\n<div class=\"language-java highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kd\">class<\/span> <span class=\"nc\">A<\/span><span class=\"o\">{<\/span>\n    <span class=\"kd\">static<\/span> <span class=\"kt\">float<\/span> <span class=\"nf\">halve<\/span><span class=\"o\">(<\/span><span class=\"kt\">int<\/span> <span class=\"n\">x<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">x<\/span> <span class=\"o\">\/<\/span> <span class=\"mf\">2.0<\/span><span class=\"o\">;<\/span>\n    <span class=\"o\">}<\/span>\n    <span class=\"kd\">static<\/span> <span class=\"kt\">float<\/span> <span class=\"nf\">halve<\/span><span class=\"o\">(<\/span><span class=\"nc\">String<\/span> <span class=\"n\">x<\/span><span class=\"o\">){<\/span>\n        <span class=\"k\">return<\/span> <span class=\"nf\">halve<\/span><span class=\"o\">(<\/span><span class=\"n\">parseInt<\/span><span class=\"o\">(<\/span><span class=\"n\">x<\/span><span class=\"o\">));<\/span>\n    <span class=\"o\">}<\/span>\n<span class=\"o\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Python:<\/p>\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Union<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">halve<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"n\">Union<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">if<\/span> <span class=\"nb\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">x<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">int<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">x<\/span> <span class=\"o\">\/<\/span> <span class=\"mf\">2.0<\/span><span class=\"p\">;<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>So while Java relies on the language to do type-based dynamic dispatch, Python does the conversion explicitly, by allowing more input types.<\/p>\n\n<h2 id=\"final-thoughts\">Final Thoughts<\/h2>\n\n<p>While throughout this post I\u2019ve been using Python 3 examples, this all works on Python 2 (though advanced metaprogramming features aren\u2019t going to work out as expected). If you want to start using <code class=\"language-plaintext highlighter-rouge\">typing<\/code> in your Python code, go check out <a href=\"https:\/\/docs.python.org\/3\/library\/typing.html\">the module documentation<\/a>, and the <a href=\"http:\/\/mypy.readthedocs.io\/en\/latest\/\"><code class=\"language-plaintext highlighter-rouge\">mypy<\/code> documentation<\/a>.<\/p>\n\n<p>Also, if you want to know more about type systems, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Type_system\">the Wikipedia article on them<\/a> is a great start. The maths behind type systems gets pretty crazy interesting, you\u2019ll likely see some more about it on here in the future.<\/p>\n\n<h3 id=\"ps\">(PS)<\/h3>\n\n<p>This is my first ever published blog post. I would absolutely love any feedback, be it positive or horribly negative. Feel free to email me via <a href=\"mailto:me@trm.io\">me@trm.io<\/a>, or hit me up on <a href=\"https:\/\/twitter.com\/TRManderson\">my insufficiently used twitter<\/a>.<\/p>\n","pubDate":"Sun, 18 Dec 2016 11:00:00 +0000","link":"https:\/\/trm.io\/2016\/12\/18\/overview-of-python-types.html","guid":"https:\/\/trm.io\/2016\/12\/18\/overview-of-python-types.html","category":["python","types","Tech"]}]}}