Hey Matt. I am not getting reply emails for case ABC123 Jane Doe<\/p>\n<\/blockquote>\n
Ideally, with a solid monitoring stack<\/a>, you will be alerted of bugs and crashes as they happen, but some may still slip through the cracks. In any case, you\u2019ve got to find and fix these issues promptly or your users will learn to distrust you and your software, kicking off a feedback loop of negative perception. Best to nip this in the bud.<\/p>\n So a user has told you about a bug in production, and you\u2019ve gotta fix it - how do you figure out what went wrong? Where do you start? In this post I\u2019ll walk you through an illustrative example of hunting down a bug in our email system.<\/p>\n So this was the message I got over Slack from a user of my website:<\/p>\n Hey Matt. I am not getting reply emails for case ABC123 Jane Doe<\/p>\n<\/blockquote>\n A user was not receiving an email, despite their client insisting that they had sent the email. That\u2019s all I know so far...<\/p>\n ... and it\u2019s not quite enough. I know the case number but that\u2019s not enough to track any error messages efficiently. I followed up with my user to check:<\/p>\n With this info in hand I can focus my search on a particular time range and sender address.<\/p>\n There\u2019s one more piece of info you need to have before you start digging into log files and such: what are the components of the email-receiving system? I assembled this one myself, but under other circumstances, in a team setting, I might ask around to build a complete picture of the system. In this case it looks like this:<\/p>\n In brief:<\/p>\n Inside the web server there\u2019s a pretty standard \u201c3 tier\u201d setup:<\/p>\n So, the hunt begins for evidence of this missing email, but where to start looking? One needs a search strategy. In this case, my intuition is to check the \u201cstart\u201d and \u201cend\u201d points of this system and work my way inwards. My reasoning is:<\/p>\n So do you start upstream or downstream? I think you do whatever\u2019s most convenient and practical. <\/p>\n Of course you may have a special system-specific knowledge that leads you towards checking one particular component first (eg. \u201cour code is garbage it\u2019s probably our code, let\u2019s check that first\u201d), which is a cool and smart thing to do. Gotta exploit that domain knowledge.<\/p>\n In this case it seemed easiest to check SendGrid\u2019s fancy web UI for evidence of an email failing to be received or something. I had a click around and found their reporting on this matter to be... pretty fucking useless to be honest.<\/p>\n This is all I could find - so I\u2019ve learned that we usually get emails. Reassuring but not very helpful in this case. They have good reporting on email sending, but this dashboard was disappointingly vague.<\/p>\n After checking SendGrid (most upstream) I then checked to see if the the database (most downstream) had received the email content.<\/p>\n As an aside, I also checked if the email was showing up in the web UI, which it wasn\u2019t (maybe my user got confused and looked at the wrong case?). It\u2019s good to quickly check for stupid obvious things just in case.<\/p>\n Since we don\u2019t have a high volume of emails I was able to check the db by just eyeballing the Django admin page. If we were getting many emails per day I would have instead run a query in the Django shell via the ORM (or run an SQL query directly on the db).<\/p>\n It wasn\u2019t there >:(<\/p>\n So far we know that maybe<\/em> SendGrid got the email and it\u2019s definitely not in the database. Since it was easy to do I quickly scanned my error monitoring logs (using Sentry<\/a>) for any relevant errors. Nothing. No relevant application errors during the expected time period found.<\/p>\n Aside<\/strong>: yes my Sentry issue inbox is a mess. I know, it's bad. Think of it like an email in box with 200 unread emails, most of them spam, but maybe a few important ones in the pile. For both emails and error reports, it's best to have a clean inbox.<\/p>\n Aside<\/strong>: ideally I would get Slack notifications for any production errors and investigate them as they happen but Sentry recently made Slack integration a paid feature and I haven\u2019t decided whether to upgrade or move.<\/p>\n Looking back upstream, I wanted to know if I could find anything interesting in the NGINX logs. If you\u2019re not familiar with webserver logfiles I give a rundown in this article<\/a> covering a typical Django stack.<\/p>\n All my server logs get sent to SumoLogic, a log aggregator (explained in the \u201clog aggregation\u201d section of this article<\/a>), where I can search through them in a web UI.<\/p>\n I checked the NGINX access logs for all incoming requests to the email webhook path in the relevant timeframe and found nothing interesting. This shows NGINX is receiving email data in general, which is good.<\/p>\n Next I checked the NGINX error logs... and found a clue!<\/p>\n For those who don\u2019t want to squint at the screenshot above this was the error log:<\/p>\n 2022\/04\/30 02:38:40 [error] 30616#30616: *129401 client intended to send too large body: 21770024 bytes, client: 172.70.135.74, server: www.mysite.com, request: \"POST \/email\/receive\/ HTTP\/1.1\", host: \"www.mysite.com\u201d<\/p>\n<\/blockquote>\n This error, which occurs when in receiving a POST request to the webhook URL, lines up with the time that the client apparently sent the email. So it seems likely that this is related to the email problem.<\/p>\n I googled the error message and found this StackOverflow post<\/a>. It seems that NGINX limits the size of requests that it will receive (which is configurable via the nginx.conf file). I checked my NGINX config and I had a limit of 20MB set. Checking my email ingestion code, it seems like all the file attachments are included in the HTTP request body. So... my guess was that the client sending the email attached more than 20MB of attachments (an uncompressed phone camera image is ~5MB) and NGINX refused to receive that request. Most email providers (eg Gmail) offer ~25MB of attachments per email.<\/p>\n I actually didn\u2019t do this because I got a little over-exicted and immediately wrote and pushed a fix. <\/p>\n What I should have done is verified that the problem I had in mind actually exists. I should have tried to send a 21MB email to our staging server to see if I could reproduce the error, plus asked my user to ask the client if she was sending large files in her email.<\/p>\n Oops. A small fuckup given I think the error message is pretty clear about what the problem is.<\/p>\n The fix was pretty simple, as it often is in these cases, I bumped up the NGINX request size limit ( We\u2019ve asked the client to re-send her email. Hopefully it comes through and all is well.<\/p>\n I checked further back in the SumoLogic and this is not the first time this error has happened, meaning we\u2019ve dropped a few emails. I\u2019ll need to notify the team about this. <\/p>\n If I had more time to spend on this project and I\u2019d consider adding some kind of alert to NGINX error logs so that we\u2019d see them pop up in Slack - maybe SumoLogic offers this, I haven\u2019t checked. <\/p>\n Another option would be going with an alternative to SendGrid that had more useful reporting on failed webhook delivery attempts.<\/p>\n Although it can sometimes be stressful, finding and fixing these problems can also be a lot of fun. It\u2019s like a detective game where you are searching for clues to crack the case.<\/p>\n In summary my advice for productively hunting down errors in production are:<\/p>\n Importantly I was only able to solve this issue because I had access to my server log files. A good server monitoring setup makes these issues much quicker and less painful to crack. If you want to know what montioring tools I like to use in my projects, check out my Django montioring stack<\/a>.<\/p>","category":{"@attributes":{"term":"Programming"}}},{"title":"DevOps in academic research","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/devops-academic-research.html","rel":"alternate"}},"published":"2021-11-21T12:00:00+11:00","updated":"2021-11-21T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2021-11-21:\/devops-academic-research.html","summary":" I'd like to share some things I've learned and done in the 18 months I worked as a \"Research DevOps Specialist\" for a team of infectious disease epidemiologists<\/a>.\nPrior to this job I'd worked as a web developer for four years and I'd found that the day-to-day had become quite \u2026<\/p>","content":" I'd like to share some things I've learned and done in the 18 months I worked as a \"Research DevOps Specialist\" for a team of infectious disease epidemiologists<\/a>.\nPrior to this job I'd worked as a web developer for four years and I'd found that the day-to-day had become quite routine. Web dev is a mature field where most of the hard problems have been solved. Looking for something new, I started a new job at a local university in early 2020. The job was created when my colleagues wrote ~20k lines of Python code and then found out what a pain in the ass it is to maintain a medium-sized codebase. It's the usual story: the code is fragile, it's slow, it's easy to break things, changes are hard to make. I don't think this situation is anyone's fault per-se: it arises naturally whenever you write a big pile of code.<\/p>\n In the remainder of this post I'll talk about the application we were working on and the awesome, transformative, <superlative> power of:<\/p>\n If you're a web developer, you might be interested to see how familar practices can be applied in different contexts. If you're an academic who uses computers in your work, then you might be interested to learn how some ideas from software development can help you be more effective.<\/p>\n We were working on a compartmental<\/a> infectious disease model to simulate the spread of tuberculosis. Around March 2020 the team quickly pivoted to modelling COVID-19 as well (surprise!). There's documentation here<\/a> with examples<\/a> if you want to poke around.<\/p>\n In brief, it works like this: you feed the model some data for a target region (population, demographics, disease attributes) and then you simulate what's going to happen in the future (infections, deaths, etc). This kind of modelling is useful for exploring different scenarios, such as \"what would happen if we closed all the schools?\" or \"how should we roll out our vaccine?\". These results are presented to stakeholders, usually from some national health department, via a PowerBI dashboard. Alternatively the results are included in a fancy academic paper as graphs and tables.<\/p>\n (Note: \"notifications\" are the infected cases that we know about)<\/p>\n A big part of our workflow was model calibration. This is where we would build a disease model with variable input parameters, such as the \"contact rate\" (proportional to how infectious the disease is), and then try to learn the best value of those parameters given some historical data (such as a timeseries of the number of cases). We did this calibration using a technique called Markov chain Monte Carlo<\/a> (MCMC). MCMC has many nice statistical properties, but requires running the model 1000 to 10,000 times - which is quite computationally expensive.<\/p>\n This all sounds cool, right? It was! The problem is that when I started. the codebase just hadn't been getting the care it needed given its size and complexity. It was becoming unruly and unmanageable. Trying to read and understand the code was stressing me out.<\/p>\n Furthermore, running calibrations was slow<\/em>. It could take days or weeks. There was a lot of manual toil where someone needed to upload the application to the university computer cluster, babysit the run and download the outputs, and then post-process the results on their laptop. The execution of the code itself took days or weeks. This time-sink is a problem when you're trying to submit an academic paper and a reviewer is like \"hey can you just re-run everything with this one small change\" and that means re-running days or weeks of computation.<\/p>\n So there were definitely some pain points and room for improvement when I started.<\/p>\n The team knew that there were problems and everybody wanted to improve the way we worked. If I could point to any key factor in our later succeses it would be their willingness to change and openness to new things.<\/p>\n I took a \"DevOps\" approach to my role (it was in the job title after all). What do I mean by DevOps? This article<\/a> sums it up well:<\/p>\n a set of practices that works to automate and integrate the processes between [different teams], so they can build, test, and release software faster and more reliably<\/p>\n<\/blockquote>\n Traditionally this refers to work done by Software Dev<\/strong>elopers and IT Op<\/strong>erations<\/strong>, but I think it can be applied more broadly. In this case we had a software developer, a mathematician, an epidemiologist and a data visualisation expert working on a common codebase.<\/p>\n A key technique of DevOps is to think about the entire system that produces finished work. You want to conceive of it as a kind of pipeline to be optimised end-to-end, rather than focusing on any efficiencies achieved by individuals in isolation. One is encouraged to explicitly map the flow of work through the system. Where does work come from? What stages does it need to flow through to be completed? Where are the bottlenecks? Importantly: what is the goal of the system?<\/p>\n In this case, I determined that our goal was to produce robust academic research, in the form of published papers or reports. My key metric was to minimise \"time to produce a new piece of research\", since I believed that our team's biggest constraint was time, rather than materials or money or ideas or something else. Another key metric was \"number of errors\", which should be zero: it's bad to publish incorrect research.<\/p>\n If you want to read more about DevOps I recommend checking out The Phoenix Project<\/a> and\/or The Goal<\/a> (the audiobooks are decent).<\/p>\n As I mentioned, you want to conceive of your team's work as a kind of pipeline. So what was our pipeline? After chatting with my colleagues I came up with something like this:<\/p>\n It took several discussions to nail this process down. People typically have decent models of how they work floating around in their heads, but it's not common to write it out explicitly like this. Getting this workflow on paper gave us some clear targets for improvement. For example:<\/p>\n My first concern was testing. When I started there were no automated tests for the code. There were a few little scripts and \"test functions\" which you could run manually, but nothing that could be run as a part of continuous integration<\/a>.<\/p>\n This was a problem. Without tests, errors will inevitably creep into the code. As the complexity of the codebase increases, it becomes infeasible to manually check that everything is working since there are too many things to check. In general writing code that is correct the first time isn't too hard - it's not breaking it later that's difficult.<\/p>\n In the context of disease modelling, automated tests are even more important than usual because the correctness of the output cannot be easily verified. The whole point of the system is to calculate an output that would be infeasible for a human to produce. Compare this scenario to web development where the desired output is usually known and easily verified. You can usually load up a web page and click a few buttons to check that the app works.<\/p>\n So where did I start? Trying to add tests to an untested codebase with thousands of lines of code is very intimidating. I couldn't simply sit down and write unit tests for every little bit of functionality because it would have taken weeks. So instead I wrote \"smoke tests\". A smoke test runs some code and checks that it doesn't crash. For example:<\/p>\n To some this may look crimininally stupid, but these tests give fantastic bang-for-buck. They don't tell you whether the model outputs are correct, but they only takes a few minutes to write. These tests catch all sorts of stupid bugs: like someone trying to add a number to a string, undefined variables, bad filepaths, etc. They doesn't help so much in reducing semantic errors, but they do help with development speed.<\/p>\n A lack of testing is the kind of problem that people don't know they have. When you tell someone \"hey we need to start writing tests!\" the typical reaction is \"hmm yeah sure I guess, sounds nice...\" and internally they're thinking \"... but I've got more important shit to do\". You can try browbeating them by telling them how irresponsible they're being etc, but that's unlikely to actually get anyone to write and run tests on their own time.<\/p>\n So how to convince people that testing is valuable? You can show<\/em> them, with the magic of \u2728continuous integration\u2728. Our code was hosted in GitHub so I set up GitHub Actions to automatically run the new smoke tests on every commit to master. I've written a short guide on how to do this here<\/a>.<\/p>\n This setup makes tests visible to everyone. There's a little tick or cross next to every commit and, importantly, next to the name of the person who broke the code.<\/p>\n With this system in place we eventually developed new norms around keeping the tests passing. People would say \"Oops! I broke the tests!\" and it became normal to run the tests locally and fix them if they were broken. It was a little harder to encourage people to invest time in writing new tests.<\/p>\n Once I become more familiar with the codebase I eventually wrote integration and unit tests for the critical modules. I've written a bit more about some testing approaches I used here<\/a>.<\/p>\n Something that stood out to me in this process was that perhaps the most valuable thing I did in that job was one of the easiest things to do. Setting up continuous integration with GitHub took me an hour to two, but it's been paying dividends for ~2 years since. How hard something is to do and how valuable it is are different things.<\/p>\nThe problem<\/h2>\n
\n
More detail<\/h2>\n
\n
Knowledge of the system<\/h2>\n
![\"email-system\"](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/email-system.png\")
<\/p>\n\n
\n
![\"web](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/webserver.png\")
<\/p>\nMy approach<\/h2>\n
\n
Did SendGrid get the email?<\/h2>\n
![\"Sendgrid](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/sendgrid-chart.png\")
<\/p>\nIs the email in the database?<\/h2>\n
![\"Django](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/django-admin.png\")
<\/p>\nDid my code explode?<\/h2>\n
![\"Sentry](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/sentry-errors.png\")
<\/p>\nDid NGINX receive the POST request?<\/h2>\n
![\"Sumologic](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/sumologic-access-search.png\")
<\/p>\n![\"Sumologic](\"https:\/\/mattsegal.dev\/img\/prod-bug-hunt\/sumologic-error-search.png\")
<\/p>\n\n
What is going wrong?<\/h2>\n
Testing the hypothesis<\/h2>\n
The fix<\/h2>\n
client_max_body_size<\/code>) to 60MB. That might be a little excessive, perhaps 30MB would have been fine, but whatever. I updated the config file in source control and deployed it to the staging and prod environments. I tested that I can send larger files by sending a 24MB email attachment to the staging server.<\/p>\nAftermath<\/h2>\n
Overview<\/h2>\n
\n
\n
The application in question<\/h2>\n
![\"notifications\"](\"https:\/\/mattsegal.dev\/img\/devops-academia\/notifications.png\")
<\/p>\n![\"calibration\"](\"https:\/\/mattsegal.dev\/img\/devops-academia\/calibration.png\")
<\/p>\nImproving our workflow with DevOps<\/h2>\n
\n
Mapping the workflow<\/h2>\n
![\"workflow\"](\"https:\/\/mattsegal.dev\/img\/devops-academia\/autumn-workflow.png\")
<\/p>\n\n
Testing the codebase<\/h2>\n
Smoke Tests<\/h2>\n
<\/span>def<\/span> test_covid_malaysia<\/span>():<\/span>\n """Ensure the Malaysia region model can run without crashing"""<\/span>\n # Load model configuration.<\/span>\n region<\/span> =<\/span> get_region<\/span>(<\/span>"malaysia"<\/span>)<\/span>\n # Build the model with default parameters.<\/span>\n model<\/span> =<\/span> region<\/span>.<\/span>build_model<\/span>()<\/span>\n # Run the model, don't check the outputs.<\/span>\n model<\/span>.<\/span>run_model<\/span>()<\/span>\n<\/code><\/pre><\/div>\n\nContinuous Integration<\/h2>\n
![\"test-failures\"](\"https:\/\/mattsegal.dev\/img\/devops-academia\/test-failures.png\")
<\/p>\n