{"title":{},"link":[{"@attributes":{"rel":"self","type":"application\/atom+xml","href":"https:\/\/ddejohn.dev\/atom.xml"}},{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/ddejohn.dev"}}],"generator":"Zola","updated":"2024-09-12T00:00:00+00:00","id":"https:\/\/ddejohn.dev\/atom.xml","entry":[{"title":"Distinct string types with Python","published":"2024-09-12T00:00:00+00:00","updated":"2024-09-12T00:00:00+00:00","author":{"name":"\n \n Unknown\n \n "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/ddejohn.dev\/distinct-types\/"}},"id":"https:\/\/ddejohn.dev\/distinct-types\/","content":"

Writing type-safe code in Python<\/h2>\n

This post is one of many I plan to write detailing techniques you can use to write \"type-safe\" code in Python. I use quotes there because types in Python are not enforced. Despite this, they can make your code much<\/em> safer and easier to develop against, reason about, test, and maintain.<\/p>\n

Everything is a string<\/h2>\n

If you've ever developed or maintained an HTTP\/JSON API, you've likely experienced this issue: you've got some POST<\/code> endpoint that accepts a large JSON payload which gets dumped into your lap as an arbitrarily nested dict<\/code> of strings. This isn't particularly useful to us, the maintainer of this endpoint. Most web frameworks, I'm assuming, will marshall the data into native Python objects, but this is only half the battle:<\/p>\n

\n

What is the difference between \"123-456-7890\"<\/code> and \"first.last@email.com\"<\/code>?<\/p>\n<\/blockquote>\n

In Python (virtually any programming language, really), these two pieces of information are merely strings. Of course, you, the reader know that one is a phone number and the other is an email address -- but Python doesn't<\/em>. All Python sees is two strings. Python wouldn't think you were crazy for trying to store that email address in a database table called phone_numbers<\/code>.<\/p>\n

# my_database.py\n<\/span>from <\/span>typing <\/span>import <\/span>Protocol\n<\/span>\n<\/span>\n<\/span>class <\/span>Database<\/span>(<\/span>Protocol<\/span>):\n<\/span>    <\/span>def <\/span>persist<\/span>(<\/span>self<\/span>, <\/span>value<\/span>: str, <\/span>table<\/span>: str) -> <\/span>None<\/span>: <\/span>...\n<\/span>\n<\/span>\n<\/span>def <\/span>persist_phone_number<\/span>(<\/span>phone<\/span>: str, <\/span>db<\/span>: Database) -> <\/span>None<\/span>:\n<\/span>    db.<\/span>persist<\/span>(phone, <\/span>table<\/span>=<\/span>"phone_number"<\/span>)\n<\/span>\n<\/span>\n<\/span>def <\/span>persist_email<\/span>(<\/span>email<\/span>: str, <\/span>db<\/span>: Database) -> <\/span>None<\/span>:\n<\/span>    db.<\/span>persist<\/span>(email, <\/span>table<\/span>=<\/span>"email_address"<\/span>)\n<\/span><\/code><\/pre>\n
# my_service.py\n<\/span>import <\/span>my_database\n<\/span>from <\/span>typing <\/span>import <\/span>Tuple\n<\/span>\n<\/span>\n<\/span># UserInfo is a tuple consisting of a phone number and an email address\n<\/span>UserInfo <\/span>= <\/span>Tuple[str, str]\n<\/span>\n<\/span>\n<\/span>def <\/span>customer_info<\/span>(<\/span>data<\/span>: UserInfo, <\/span>db<\/span>: my_database.Database) -> <\/span>None<\/span>:\n<\/span>    email, phone <\/span>= <\/span>data\n<\/span>\n<\/span>    <\/span>if not <\/span>isinstance<\/span>(phone, str) <\/span>or <\/span>phone <\/span>== <\/span>""<\/span>:\n<\/span>        <\/span>raise <\/span>ValueError<\/span>(<\/span>"Phone number must be a non-empty string"<\/span>)\n<\/span>\n<\/span>    <\/span>if not <\/span>isinstance<\/span>(email, str) <\/span>or <\/span>email <\/span>== <\/span>""<\/span>:\n<\/span>        <\/span>raise <\/span>ValueError<\/span>(<\/span>"Email address must be a non-empty string"<\/span>)\n<\/span>\n<\/span>    my_database.<\/span>persist_phone_number<\/span>(phone, db)\n<\/span>    my_database.<\/span>persist_email<\/span>(email, db)\n<\/span><\/code><\/pre>\n
Can you spot the problem with this code? It's a little bit of a trick question; this code would run just fine -- there are no syntax errors, we are validating that phone<\/code> and email<\/code> are both non-empty strings, which means we're calling the persistence methods with the correct types -- so what's wrong?<\/p>\n
We've unpacked the UserInfo<\/code> tuple incorrectly. That line of code should read phone, email = data<\/code>.<\/p>\n
Best case scenario: your data team has some mechanism of validating the input at the database layer, and returns a DatabaseError<\/code>, crashing your program. Worst case scenario: there is no further validation and you've just stuffed an email into the phone_number<\/code> table, and a phone number into the email_address<\/code> table -- everything worked fine because you didn't violate the API contract anywhere along the line<\/em>. You performed a perfectly valid task in your Python program, but now you have a data integrity problem!<\/p>\n
Everything is a string, and that's kind of bad<\/h2>\n
How can we prevent this? We could<\/em> implement custom classes for EmailAddress<\/code> and PhoneNumber<\/code>, but that'll get old real quick when you also need to implement classes for UserName<\/code>, Password<\/code>, GivenName<\/code>, FamilyName<\/code>, PreferredName<\/code>, Pronouns<\/code>, etc. Another issue is that not all of these inputs can even be validated. This also wouldn't solve the issue of having to deal with two inputs that are the same \"type\" (e.g., mailing addresses -- you definitely don't want to mix up sending and receiving addresses!).<\/p>\n
You would also need to construct instances of each of those classes whenever you receive a payload, and deal with all the overhead of maintaining a custom class definition. You may have also taken already care of data validation in the frontend, or at the webserver ingress or proxy layer, so writing another<\/em> step of data validation is both wasteful and adds technical debt.<\/p>\n
Different types of string<\/h2>\n
This is where typing.NewType<\/code> comes in! It's only a type definition so it'll be completely ignored at runtime, but it allows you to define different \"variants\" of a given data type -- in our case, str<\/code> -- which our type checker will treat as completely different types:<\/p>\n
# my_database.py\n<\/span>from <\/span>typing <\/span>import <\/span>NewType, Tuple, Protocol\n<\/span>\n<\/span>\n<\/span>EmailAddress <\/span>= <\/span>NewType<\/span>(<\/span>"EmailAddress"<\/span>, str)\n<\/span>PhoneNumber <\/span>= <\/span>NewType<\/span>(<\/span>"PhoneNumber"<\/span>, str)\n<\/span>UserInfo <\/span>= <\/span>Tuple[PhoneNumber, EmailAddress]\n<\/span>\n<\/span>\n<\/span>class <\/span>Database<\/span>(<\/span>Protocol<\/span>):\n<\/span>    <\/span>def <\/span>persist<\/span>(<\/span>self<\/span>, <\/span>value<\/span>: PhoneNumber <\/span>| <\/span>EmailAddress, <\/span>table<\/span>: str) -> <\/span>None<\/span>: <\/span>...\n<\/span>\n<\/span>\n<\/span>def <\/span>persist_phone_number<\/span>(<\/span>phone<\/span>: PhoneNumber, <\/span>db<\/span>: Database) -> <\/span>None<\/span>:\n<\/span>    db.<\/span>persist<\/span>(phone, <\/span>table<\/span>=<\/span>"phone_number"<\/span>)\n<\/span>\n<\/span>\n<\/span>def <\/span>persist_email<\/span>(<\/span>email<\/span>: EmailAddress, <\/span>db<\/span>: Database) -> <\/span>None<\/span>:\n<\/span>    db.<\/span>persist<\/span>(email, <\/span>table<\/span>=<\/span>"email_address"<\/span>)\n<\/span><\/code><\/pre>\n
Using the same customer_info<\/code> function, but with a UserInfo<\/code> tuple that uses our NewType<\/code> types, the editor now shows a type error with our previous code:<\/p>\n
<\/p>\n
Note that you cannot get this behavior with just type aliases!<\/p>\n
This is obviously a contrived example, but using NewType<\/code> can really help you and the other developers reading and maintaining your code, or your users if you're developing a library! Remember that you really need to be validating your data somewhere along the line -- Python's types are not enforced, they are merely a tool for keeping developers from making simple mistakes.<\/p>\n
My next post will go into more detail on using typing.Protocol<\/code>. I'm going to try to keep these posts short and sweet for now -- there's tons more to say about NewType<\/code>, but you can also just go RTFM!<\/p>\n"},{"title":"String Shenanigans","published":"2021-09-26T00:00:00+00:00","updated":"2021-09-26T00:00:00+00:00","author":{"name":"\n            \n              Unknown\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/ddejohn.dev\/overlapping-strings\/"}},"id":"https:\/\/ddejohn.dev\/overlapping-strings\/","content":"
This post is an extension of one of my recent Stack Overflow answers<\/a>.<\/p>\n
I'll show two pure Python solutions for concatenating an arbitrary list of unique strings based on suffix-prefix pairs, e.g.: \"helloworld\"<\/code> and \"worldfoo\"<\/code> would combine to \"helloworldfoo\"<\/code>. Here's an example of a (short) list of strings for which we can start working on a solution:<\/p>\n
strings <\/span>= <\/span>{<\/span>"lazydog"<\/span>,\n<\/span>           <\/span>"Thequick"<\/span>,\n<\/span>           <\/span>"thelazy"<\/span>,\n<\/span>           <\/span>"overthe"<\/span>,\n<\/span>           <\/span>"foxjumps"<\/span>,\n<\/span>           <\/span>"jumpsover"<\/span>,\n<\/span>           <\/span>"brownfox"<\/span>,\n<\/span>           <\/span>"quickbrown"<\/span>}\n<\/span><\/code><\/pre>\n
Notice that if the strings were in order, e.g., \"Thequick\", \"quickbrown\", \"brownfox\", ...<\/code>, then it'd be pretty straight forward to join the strings and remove the overlaps to get the desired result \"Thequickbrownfox...\"<\/code>. Of course, we can't make that assumption about an ordered list, so we'll have to figure out the ordering manually.<\/p>\n
Since we need to check every pair of words, and since we can't assume any<\/em> pair is ordered, we also have to take the reverse of every pair:<\/p>\n
pairs <\/span>= <\/span>set<\/span>()\n<\/span>for <\/span>a <\/span>in <\/span>strings:\n<\/span>    <\/span>for <\/span>b <\/span>in <\/span>strings:\n<\/span>        <\/span>if <\/span>a <\/span>!= <\/span>b:\n<\/span>            pairs.<\/span>add<\/span>((a, b))\n<\/span>            pairs.<\/span>add<\/span>((b, a))\n<\/span><\/code><\/pre>\n
Output:<\/p>\n
{(<\/span>'Thequick'<\/span>, <\/span>'brownfox'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'foxjumps'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'jumpsover'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'lazydog'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'overthe'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'quickbrown'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'thelazy'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'Thequick'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'foxjumps'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'jumpsover'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'lazydog'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'overthe'<\/span>),\n<\/span> .\n<\/span> .\n<\/span> .\n<\/span><\/code><\/pre>\n
For this task, we can use permutations instead, which tends to be a bit quicker than the cartesian product:<\/p>\n
import <\/span>itertools\n<\/span>\n<\/span>set<\/span>(itertools.<\/span>permutations<\/span>(strings, <\/span>r<\/span>=<\/span>2<\/span>))\n<\/span><\/code><\/pre>\n
Output:<\/p>\n
{(<\/span>'Thequick'<\/span>, <\/span>'brownfox'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'foxjumps'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'jumpsover'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'lazydog'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'overthe'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'quickbrown'<\/span>),\n<\/span> (<\/span>'Thequick'<\/span>, <\/span>'thelazy'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'Thequick'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'foxjumps'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'jumpsover'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'lazydog'<\/span>),\n<\/span> (<\/span>'brownfox'<\/span>, <\/span>'overthe'<\/span>),\n<\/span> .\n<\/span> .\n<\/span> .\n<\/span><\/code><\/pre>\n
Now for every pair, we need to find whether or not they overlap. Take, for example, the pair ('brownfox', 'foxjumps')<\/code>. Let's \"slide\" the string 'foxjumps'<\/code> under the string \"brownfox\"<\/code> one character at a time to see if we get any overlaps:<\/p>\n
brownfox\n<\/span>       foxjumps\n<\/span>\n<\/span>brownfox\n<\/span>      foxjumps\n<\/span>\n<\/span>brownfox\n<\/span>     foxjumps  <\/span># Found an overlap!\n<\/span>\n<\/span>brownfox\n<\/span>    foxjumps\n<\/span>\n<\/span>brownfox\n<\/span>   foxjumps\n<\/span>\n<\/span>brownfox\n<\/span>  foxjumps\n<\/span>\n<\/span>brownfox\n<\/span> foxjumps\n<\/span><\/code><\/pre>\n
Note that we do not continue further than this because we are assuming that all words in the original list are unique, so there cannot be an overlap equal to the length of either string in a given pair.<\/p>\n
Now let's repeat the process with another pair, ('brownfox', 'jumpsover')<\/code>:<\/p>\n
brownfox\n<\/span>       jumpsover\n<\/span>\n<\/span>brownfox\n<\/span>      jumpsover\n<\/span>\n<\/span>brownfox\n<\/span>     jumpsover\n<\/span>\n<\/span>brownfox\n<\/span>    jumpsover\n<\/span>\n<\/span>brownfox\n<\/span>   jumpsover\n<\/span>\n<\/span>brownfox\n<\/span>  jumpsover\n<\/span>\n<\/span>brownfox\n<\/span> jumpsover  <\/span># No point in checking any further than this\n<\/span><\/code><\/pre>\n
We'll use this function to find a suffix-prefix match:<\/p>\n
def <\/span>get_match<\/span>(<\/span>pair<\/span>: Tuple[str, str], <\/span>min_overlap<\/span>: int <\/span>= <\/span>3<\/span>) -> str:\n<\/span>    a, b <\/span>= <\/span>pair\n<\/span>    <\/span>for <\/span>i <\/span>in <\/span>range<\/span>(min_overlap, <\/span>min<\/span>(<\/span>map<\/span>(<\/span>len<\/span>, pair)) <\/span>+ <\/span>1<\/span>):\n<\/span>        <\/span>if <\/span>a[<\/span>-<\/span>i:] <\/span>== <\/span>b[:i]:\n<\/span>            <\/span>return <\/span>b[:i]\n<\/span>    <\/span>return <\/span>""\n<\/span><\/code><\/pre>\n
What this function does is essentially what is shown in the diagrams above: the second string b<\/code> from pair<\/code> is \"slid\" over the \"top\" of the first string a<\/code> from pair<\/code> until a match is found at which point the function returns immediately. The min_overlap<\/code> parameter is optional and allows us to enforce that a match must be at least min_overlap<\/code> number of characters, e.g., if min_overlap == 2<\/code>, then \"hello\", \"one\"<\/code> shouldn't match since the only suffix-prefix match is \"o\"<\/code>.<\/p>\n
Now, we need to iterate over every possible pairing, and record all pairs which match:<\/p>\n
for <\/span>pair <\/span>in <\/span>itertools.<\/span>permutations<\/span>(strings, <\/span>r<\/span>=<\/span>2<\/span>):\n<\/span>    <\/span>if <\/span>(match <\/span>:= <\/span>get_match<\/span>(pair)):\n<\/span>        <\/span>print<\/span>(pair)\n<\/span><\/code><\/pre>\n
Output:<\/p>\n
(<\/span>'Thequick'<\/span>, <\/span>'quickbrown'<\/span>)\n<\/span>(<\/span>'thelazy'<\/span>, <\/span>'lazydog'<\/span>)\n<\/span>(<\/span>'overthe'<\/span>, <\/span>'thelazy'<\/span>)\n<\/span>(<\/span>'foxjumps'<\/span>, <\/span>'jumpsover'<\/span>)\n<\/span>(<\/span>'jumpsover'<\/span>, <\/span>'overthe'<\/span>)\n<\/span>(<\/span>'brownfox'<\/span>, <\/span>'foxjumps'<\/span>)\n<\/span>(<\/span>'quickbrown'<\/span>, <\/span>'brownfox'<\/span>)\n<\/span><\/code><\/pre>\n
Now here's where the two solutions diverge.<\/p>\n
First solution: constructing a DAG<\/h2>\n
So we know that the solution for our particular example should be Thequickbrownfoxjumpsoverthelazydog<\/code>. This can be represented by a (boring) Directed Acyclic Graph<\/a>:<\/p>\n
Thequick <\/span>-> <\/span>quickbrown <\/span>-> <\/span>brownfox <\/span>-> <\/span>foxjumps <\/span>-> <\/span>jumpsover <\/span>-> <\/span>overthe <\/span>-> <\/span>thelazy <\/span>-> <\/span>lazydog\n<\/span><\/code><\/pre>\n
This first solution uses a dictionary to track the adjacencies:<\/p>\n
def <\/span>links_joiners<\/span>(<\/span>strings<\/span>: List[str]) -> Tuple[Dict[str, str], Set[str]]:\n<\/span>    links <\/span>= <\/span>dict<\/span>()\n<\/span>    joiners <\/span>= <\/span>set<\/span>()\n<\/span>    <\/span>for <\/span>pair <\/span>in <\/span>itertools.<\/span>permutations<\/span>(strings, <\/span>r<\/span>=<\/span>2<\/span>):\n<\/span>        <\/span>if <\/span>(match <\/span>:= <\/span>get_match<\/span>(pair)):\n<\/span>            joiners.<\/span>add<\/span>(match)\n<\/span>            links.<\/span>update<\/span>((pair,))\n<\/span>    <\/span>return <\/span>links, joiners\n<\/span><\/code><\/pre>\n
Output:<\/p>\n
>>> <\/span>links, joiners <\/span>= <\/span>links_joiners<\/span>(strings)\n<\/span>>>> <\/span>links\n<\/span>{<\/span>'Thequick'<\/span>: <\/span>'quickbrown'<\/span>,\n<\/span> <\/span>'thelazy'<\/span>: <\/span>'lazydog'<\/span>,\n<\/span> <\/span>'overthe'<\/span>: <\/span>'thelazy'<\/span>,\n<\/span> <\/span>'foxjumps'<\/span>: <\/span>'jumpsover'<\/span>,\n<\/span> <\/span>'jumpsover'<\/span>: <\/span>'overthe'<\/span>,\n<\/span> <\/span>'brownfox'<\/span>: <\/span>'foxjumps'<\/span>,\n<\/span> <\/span>'quickbrown'<\/span>: <\/span>'brownfox'<\/span>}\n<\/span>\n<\/span>>>> <\/span>joiners\n<\/span>{<\/span>'brown'<\/span>, <\/span>'fox'<\/span>, <\/span>'jumps'<\/span>, <\/span>'lazy'<\/span>, <\/span>'over'<\/span>, <\/span>'quick'<\/span>, <\/span>'the'<\/span>}\n<\/span><\/code><\/pre>\n
For now, we'll focus on the links<\/code> dictionary, which shows us our adjacencies. Recall our target:<\/p>\n
Thequick <\/span>-> <\/span>quickbrown <\/span>-> <\/span>brownfox <\/span>-> <\/span>foxjumps <\/span>-> <\/span>jumpsover <\/span>-> <\/span>overthe <\/span>-> <\/span>thelazy <\/span>-> <\/span>lazydog\n<\/span><\/code><\/pre>\n
Now pick a random key in links<\/code> and take its associated value. If that value is also<\/em> a key, repeat that last step with the new key. If you get a value that isn't<\/em> also a key in links<\/code>, you've reached the end of the DAG. For example, consider an intial starting key, \"jumpsover\"<\/code>.<\/p>\n