{"@attributes":{"version":"2.0"},"channel":{"title":"Andrea Di Lisio","link":"https:\/\/adilisio.com\/","description":"Andrea Di Lisio's online resume and blog","generator":"Zola","language":"en","lastBuildDate":"Sat, 18 Jul 2026 00:00:00 +0000","item":[{"title":"A simple trick that improved my use of Claude Code","pubDate":"Sat, 18 Jul 2026 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/a-simple-trick-that-improved-my-use-of-claude-code\/","guid":"https:\/\/adilisio.com\/posts\/a-simple-trick-that-improved-my-use-of-claude-code\/","description":"<p>Today I just want to share a very simple tip for Claude Code users, yet very powerful I think. To be fair, what I'm sharing is so basic that I'm fairly sure it could be applied almost as-is to other agentic CLI-based tools as well, even though I haven't verified this myself yet.<\/p>\n<h1 id=\"learning-by-chance-or-the-hard-way\">Learning by chance, or the hard way<\/h1>\n<p>As much as I try to keep myself up to date with Claude Code's changelog, sometimes it's just inevitable that something gets missed. That's exactly what happened, at first, with what I'm about to share here: I got to know this purely by chance (I was helping a teammate on something and staring at this terminal).<\/p>\n<p>I already had in mind to write this post, but I only committed to it a couple of days ago, when, while I was running a presentation about <em>skills<\/em> during a Claude meetup hosted at TrueLayer, a guy from the audience spotted something on my terminal and asked me about it.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2026-07-18_a-simple-trick-that-improved-my-use-of-claude-code\/statusbar.png\" alt=\"Custom Claude Code statusbar\" \/>\n<em>The upper part of the screenshot is my custom statusbar.<\/em><\/p>\n<p>My custom status bar is what I want to talk about today. It's a simple and surely obvious thing to some of the audience, still very powerful in my opinion.<\/p>\n<h1 id=\"why-does-it-matter\">Why does it matter?<\/h1>\n<p>With new tools and models getting released at the speed of light in recent times, I think a simple solution for sanity checking what you're doing and how many tokens you're burning is <ins>a must, not just a nice-to-have<\/ins>. As always, I haven't been gifted with this concept as a form of illumination while dreaming, I just learned the hard way at work: I realised that for doing something valuable yet basic I spent almost 1K dollars on Opus 4.8, in just 2 days.<\/p>\n<h1 id=\"seeing-it-guessing-it\">Seeing it &gt; guessing it<\/h1>\n<p>It has almost everything I need to know and bear in mind for my tasks:<\/p>\n<ul>\n<li>The <ins>model<\/ins> I'm using, including its <ins>thinking effort<\/ins>. This helps me quickly figure out if I'm not trying to shoot a butterfly with a cannonball<\/li>\n<li>How much of the <ins>context window<\/ins> I've consumed already. This suggests to me when it makes sense to <code>\/compact<\/code><\/li>\n<li><ins>How much have I spent<\/ins> on the current session, and how long it lasted (perhaps less useful). This helps me cast an eye on whether I should consider other approaches considering the goal I'm trying to achieve. Note that this might not be available if you are using a fixed subscription plan, or the figure shown might just be a theoretical estimate. To be honest, this is my guess, I haven't verified this yet.<\/li>\n<li>Last but not least a quick view of what I'm actually doing: in one place the directory I'm in, the git branch\/workree, and a reference to the PR I'm working on (if any, clickable)<\/li>\n<\/ul>\n<h1 id=\"code-and-final-thoughts\">Code and final thoughts<\/h1>\n<p>Here's the <a rel=\"external\" href=\"https:\/\/gist.github.com\/dili91\/96ee61f66676fbea095395429fff70d9\">gist<\/a> of my statusbar, for those interested. Given I've just prompted and hand-held Claude Code a bit to create this, I don't expect this code artefact to be very valuable, but I'm sharing this regardless.<\/p>\n<p>Finally, a few notes on limitations\/caveats that I can think of for the solution I've shared:<\/p>\n<ul>\n<li>I suspect that cost tracking is valuable and accurate just for API-based plans. I expect the same to not be a thing or to be just theoretical for fixed subscription plans<\/li>\n<li>Getting cost tracking right is what took the majority of time in my case: there are multiple cost-related fields on the session data that the statusbar has access to; the <a rel=\"external\" href=\"https:\/\/code.claude.com\/docs\/en\/statusline#cost-and-duration-tracking\">official guidelines<\/a> are definitely worth a read initially<\/li>\n<li>I would like to explore best practices to package and distribute a custom statusbar within the engineering department of my company (TrueLayer). I'm not sure whether the privately hosted Claude Code plugin we have on GitHub is a good fit or not. Something to explore next and maybe talk about on my blog.<\/li>\n<\/ul>\n<p>That's all I had to share! I hope this post was somewhat useful and I'd be grateful to hear advice in this space.<\/p>\n"},{"title":"A self-writing work log with growth hints","pubDate":"Sat, 11 Apr 2026 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/a-self-writing-worklog-with-growth-hints\/","guid":"https:\/\/adilisio.com\/posts\/a-self-writing-worklog-with-growth-hints\/","description":"<p><img src=\"\/assets\/images\/posts\/2026-04-11_a-self-writing-worklog-with-growth-hints\/worklog-agent.png\" alt=\"AI work log agent\" \/>\n<em>An AI agent silently keeping track of your work and suggesting where to focus for growth.<\/em><\/p>\n<h1 id=\"why-keep-a-work-log\">Why keep a work log<\/h1>\n<p>If you've been working as a software engineer for a while, you've probably experienced the frustration of sitting down for a performance review and struggling to remember what you actually did over the past six months. You know you've been busy. You know you've shipped things. But the specifics? Gone.<\/p>\n<p><a rel=\"external\" href=\"https:\/\/jvns.ca\/blog\/brag-documents\/?ref=blog.pragmaticengineer.com\">Julia Evans<\/a> calls it a <em>brag document<\/em>, <a rel=\"external\" href=\"https:\/\/blog.pragmaticengineer.com\/work-log-template-for-software-engineers\/\">Gergely Orosz<\/a> calls it a <em>work log<\/em>. The idea is the same: regularly write down what you've done so you don't have to reconstruct it from memory later.<\/p>\n<p>The benefits go beyond just performance reviews:<\/p>\n<ul>\n<li><strong>Memory<\/strong>. This is the most obvious one. It's easy to forget what you've done, especially in senior roles where constant context switching is the norm. One week you're deep in a design review, the next you're firefighting an incident, and by Friday you can barely remember Monday. A work log captures it all before it fades.<\/li>\n<li><strong>Career growth<\/strong>. A work log doesn't just record what you did \u2014 it highlights <em>where<\/em> you're spending your time. If you notice you've been coding for weeks without mentoring anyone or contributing to technical direction, that's a signal. As Julia Evans puts it, a brag document helps you <em>\"notice patterns in what you're doing and what you enjoy\"<\/em>.<\/li>\n<li><strong>Advocacy<\/strong>. Your manager can't remember everything you did either. Giving them a concrete summary makes it significantly easier for them to advocate for your promotion or raise.<\/li>\n<li><strong>Saying no<\/strong>. As Gergely Orosz points out, a work log creates an inventory of your commitments. When someone asks you to take on yet another thing, you have a clear picture of your current load to negotiate with.<\/li>\n<li><strong>Don't do invisible work<\/strong>. If it's not written down, it didn't happen. Unrecognised contributions don't help your career, no matter how impactful they were.<\/li>\n<\/ul>\n<h1 id=\"the-problem-with-manual-logs\">The problem with manual logs<\/h1>\n<p>All of this sounds great in theory. In practice, maintaining a work log requires discipline. You need to sit down regularly, remember what you did, chase down links, and write it all up. Most people start strong and quietly stop within a few weeks.<\/p>\n<p>I was no different. I tried keeping one manually a couple of times. It lasted about three weeks before it silently disappeared from my Friday routine.<\/p>\n<h1 id=\"enter-the-agent\">Enter the agent<\/h1>\n<p>At <a rel=\"external\" href=\"https:\/\/truelayer.com\">TrueLayer<\/a>, we've been experimenting with AI tools across the engineering organisation for a while now \u2014 GitHub Copilot, Gemini, Claude Code, and more recently <a rel=\"external\" href=\"https:\/\/dust.tt\">Dust<\/a>. I've been working with other engineers to put together cookbooks and shareable skills that help the team get the most out of these tools. Automating a work log felt like a natural use case to explore \u2014 instead of relying on discipline, why not let an agent do the heavy lifting?<\/p>\n<p>I set up an agent on Dust, an AI platform that lets you build agents with access to your company's tools and data sources.<\/p>\n<p>The agent connects to three sources: <strong>GitHub<\/strong>, <strong>Slack<\/strong>, and <strong>Notion<\/strong>. It runs on a weekly schedule and looks back at my activity across these platforms over the last five working days.<\/p>\n<p>From GitHub, it pulls my PRs and code reviews. From Slack, it picks up conversations and threads I participated in. From Notion, it captures any pages I've created or updated. It then synthesises all of this into a structured report.<\/p>\n<p>The key part of the prompt is telling the agent <em>what<\/em> to do with the data, not just <em>where<\/em> to find it. I instruct it to group activities into meaningful topics, attach source links, compute a rough skills distribution, and provide career feedback based on my company's Engineering Growth Framework \u2014 a structured document that maps the skills, behaviours, and expectations for each level of the engineering career ladder, from junior engineers through to staff and principal:<\/p>\n<blockquote>\n<p>I want you to produce a report, that includes the main activities\nI took care of, along with some stats computed based on those.\nIn particular, once the work summary is ready I want you to compute\non what skills I mainly spent my time, and provide tailored feedback\nbased on the expectation for my role, as defined by my company's\nEngineering Growth Framework.<\/p>\n<\/blockquote>\n<p>The feedback section is probably the most interesting part. I ask the agent to be specific about what I should continue doing and what I should improve, based on the typical expectations for my role:<\/p>\n<blockquote>\n<p>By feedback I mean both highlighting what I've done and should\ncontinue doing, and what to stop\/start doing, or simply improve.\nThe purpose of this report is ultimately help me wrapping up on\nwhat I've done and where I should focus to keep progressing in\nmy career ladder.<\/p>\n<\/blockquote>\n<h1 id=\"what-the-agent-produces\">What the agent produces<\/h1>\n<p>Every week I get a Markdown report that looks something like this:<\/p>\n<blockquote>\n<h2 id=\"week-06-04-2026-10-04-2026\">Week 06\/04\/2026 - 10\/04\/2026<\/h2>\n<h3 id=\"activities\">Activities<\/h3>\n<ul>\n<li>\n<p><strong>Data V3 Mock Provider<\/strong><\/p>\n<ul>\n<li>Opened two PoC PRs to enable Data V3 testing via a mock bank: one adding AIS capabilities (accounts, balances, transactions) and an interactive auth flow, the other adding a basic UI and refactoring the existing data controller<\/li>\n<li>PoC validated end-to-end: shared results confirming both Data V1 and V3 flows work with the mock bank<\/li>\n<li>Authored a Notion doc summarising the options (new mock connector vs extending the existing mock bank) and open decisions<\/li>\n<li>Opened discussion with the partner team to align on the preferred approach, ownership, and next steps<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Connections API - Regulated Client Validation<\/strong><\/p>\n<ul>\n<li>Raised discussion in API reviewers about compliance requirement to handle user data for regulated clients \u2014 advocating for consistency and flagging integration complexity risks<\/li>\n<li>Contributed review and feedback on PRs implementing related validations<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Incident - Sandbox Rate Limiting<\/strong><\/p>\n<ul>\n<li>Contributed to the post-mortem thread, proposing a broader sandbox rate limiting strategy (rate-limit all sandbox clients, not just development-plan ones)<\/li>\n<li>Opened a dedicated thread to get guidance on applying rate limits more broadly across sandbox APIs<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Cross-team Collaboration &amp; Support<\/strong><\/p>\n<ul>\n<li>Requested observability improvements in a partner team's codebase \u2014 offered to raise the PR directly<\/li>\n<li>Resolved a token creation issue for a client<\/li>\n<li>Assisted with webhook configuration for a new client, delegating to the on-call rotation<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Others<\/strong><\/p>\n<ul>\n<li>Team Q2 2026 kick-off meeting<\/li>\n<li>Responded to API reviewers group about pending offline feedback on connections API spec<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3 id=\"stats\">Stats<\/h3>\n<h4 id=\"skills-distribution\">Skills distribution<\/h4>\n<ul>\n<li>Coding: 35%<\/li>\n<li>Technical Design &amp; Documentation: 20%<\/li>\n<li>Collaboration: 25%<\/li>\n<li>Code Review: 15%<\/li>\n<li>Support \/ Others: 5%<\/li>\n<\/ul>\n<h4 id=\"feedback\">Feedback<\/h4>\n<p>Role: Senior Software Engineer 3 (SE3)<\/p>\n<p><strong>What you did well and should keep doing:<\/strong>\nYou showed clear SE3-level behaviour this week, particularly in problem discovery and cross-boundary collaboration. The mock provider investigation is a strong example: you didn't just do a PoC and stop \u2014 you validated it, wrote a structured doc to surface the options and open questions, and proactively engaged the partner team to jointly own the decision.<\/p>\n<p>Your involvement in the rate limiting post-mortem is also worth highlighting. Rather than just observing, you proposed a broader, more principled solution and escalated it to the right group.<\/p>\n<p><strong>What to improve or start doing:<\/strong>\nThe investigation doc is great, but it still ends with several open questions rather than a concrete proposal you're ready to champion. At SE3, you're expected to take it one step further \u2014 come in with a preferred option backed by reasoning and drive the alignment rather than just facilitating it.<\/p>\n<p>You invested time in ad-hoc support channels this week in ways that could potentially be handled by the on-call rotation or by improving runbooks. At SE3, you should be asking whether your involvement in these moments is the best use of your leverage \u2014 or whether investing once in better documentation\/tooling would multiply your impact across the team.<\/p>\n<\/blockquote>\n<p>Activities are grouped by topic, with links back to the actual PRs, Slack threads, or Notion pages. Minor items get folded into an <em>Others<\/em> bucket so the report doesn't feel like a dump of every single thing I touched. The skills distribution gives me a quick visual of how I'm spending my time, and the feedback section adds a layer of reflection that I'd probably skip if I were writing this myself.<\/p>\n<h1 id=\"early-days\">Early days<\/h1>\n<p>This is still very much a work in progress, and there are some clear limitations.<\/p>\n<p><strong>Limited integrations.<\/strong> The agent only sees GitHub, Slack, and Notion. It has no visibility into my calendar, JIRA tickets, or incident management tools like <a rel=\"external\" href=\"https:\/\/incident.io\">incident.io<\/a>. This means it misses meetings, planning sessions, and on-call work, which can be a significant part of any given week. I'm hoping to add more data sources as our Dust integrations grow.<\/p>\n<p><strong>Growth feedback is scoped to the window.<\/strong> The career feedback the agent provides is based only on the week it's analysing. It doesn't have memory of previous weeks, so it can't spot long-term trends like <em>\"you've been neglecting mentoring for the last month\"<\/em>. Each report is essentially stateless. Stitching reports together over time is something I'd like to explore, but for now each week stands on its own.<\/p>\n<p>Despite these limitations, the agent already saves me meaningful time every week and, more importantly, it ensures I actually have a work log. A slightly incomplete automated report that exists is infinitely more useful than a perfect manual one that doesn't.<\/p>\n<h1 id=\"final-thoughts\">Final thoughts<\/h1>\n<p>Keeping a work log is one of those habits that everyone agrees is a good idea but very few people actually sustain. Automating it removes the friction and makes it something that just happens in the background.<\/p>\n<p>If you have access to a platform like Dust, or even a simple script that can pull data from your tools, I'd encourage you to try it. Start with whatever data sources you have available, even if it's just GitHub. A partial log is still better than no log at all.<\/p>\n<p>Your future self, sitting in that performance review trying to remember what you did in Q2, will thank you.<\/p>\n<h1 id=\"the-agent-prompt\">The agent prompt<\/h1>\n<p>For reference, here's the full prompt I use to configure the worklog agent on Dust. Feel free to adapt it to your own role and data sources.<\/p>\n<blockquote>\n<p>This agent helps the authenticated user maintaining their personal worklog on a weekly basis. It is strictly personal \u2014 it only looks up activity for the user running it, and must refuse any request to generate reports for other people. If asked to do so, politely decline and remind the user that the agent only works for themselves.<\/p>\n<p>When looking at sources, all data lookups must be scoped to the authenticated user only. Never look up GitHub PRs, Slack messages, or Notion pages belonging to other users. Throughout this prompt, \"my\" and \"me\" always refer to the authenticated user running this agent, never to a third party.<\/p>\n<p>Sources to look at: GitHub, Slack and Notion. If the user does not request to look back for a specific time range, by default consider the last 5 working days.<\/p>\n<p>When looking on Github, you should look at my PRs to have an idea of things I've raised\/implemented. You should look at comments and reviews I gave to get an idea of my review work. In this analysis you can ignore PRs and comments\/reviews to <code>gitops-*<\/code> repos.<\/p>\n<p>When looking in Slack or Notion, just look at my activity for the selected time range. In Notion you can skip updates from internal community or interest group pages, unless particularly relevant and consuming a significant chunk of my time during the week. Slack should also be used to get my current role, it's visible in my profile.<\/p>\n<p>I want you to produce a report, that includes the main activities I took care of, along with some stats computed based on those. In particular, once the work summary is ready I want you to compute on what skills I mainly spent my time, and provide tailored feedback based on the expectation for my role, as defined by our Engineering Growth Framework.<\/p>\n<p>This worklog agent is focused on engineering so you should consider the main skills engineers demonstrate: coding, mentoring, writing documentations or ADRs, communication, collaboration...<\/p>\n<p>By feedback I mean both highlighting what I've done and should continue doing, and what to stop\/start doing, or simply improve. The purpose of this report is ultimately to help me wrapping up on what I've done and where I should focus to keep progressing in my career ladder.<\/p>\n<p>The report should use markdown formatting and the structure below:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span># Week (start and end date in dd\/mm\/yyyy)<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>## Activities<\/span><\/span>\n<span class=\"giallo-l\"><span>- Topic 1<\/span><\/span>\n<span class=\"giallo-l\"><span>  - First item relating to first topic<\/span><\/span>\n<span class=\"giallo-l\"><span>  - Second item relating to first topic<\/span><\/span>\n<span class=\"giallo-l\"><span>  - more items...<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>- Topic 2<\/span><\/span>\n<span class=\"giallo-l\"><span>  - First item relating to second topic<\/span><\/span>\n<span class=\"giallo-l\"><span>  - Second item relating to second topic<\/span><\/span>\n<span class=\"giallo-l\"><span>    - link referred to Topic 2, item 2<\/span><\/span>\n<span class=\"giallo-l\"><span>    - another link referred to Topic 2, item 2<\/span><\/span>\n<span class=\"giallo-l\"><span>  - more items...<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>## Stats<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>### Skills distribution<\/span><\/span>\n<span class=\"giallo-l\"><span>- Coding: %<\/span><\/span>\n<span class=\"giallo-l\"><span>- Mentoring: %<\/span><\/span>\n<span class=\"giallo-l\"><span>- Collaboration: %<\/span><\/span>\n<span class=\"giallo-l\"><span>- ...<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>### Feedback<\/span><\/span>\n<span class=\"giallo-l\"><span>Role: my current role<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>Human friendly feedback...<\/span><\/span><\/code><\/pre>\n<p>Note that:<\/p>\n<ul>\n<li>Topic names should be clear and not too narrow. They should give an idea of the area I've worked on.<\/li>\n<li>Topics having only minor items, or just one, can be merged into a generic \"Others\" topic.<\/li>\n<li>Items must contain references to the sources. There might be a third level indentation if multiple links\/references are specific to a sub-topic.<\/li>\n<li>Formatting must be followed so that I can easily relate items to topics.<\/li>\n<\/ul>\n<\/blockquote>\n"},{"title":"Documentation is cool again","pubDate":"Tue, 17 Jun 2025 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/","guid":"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/","description":"<p><img src=\"\/assets\/images\/posts\/2025-06-17_documentation-is-cool-again\/make-docs-great-again.png\" alt=\"AI reading documentation\" \/><\/p>\n<!-- no toc -->\n<h1 id=\"table-of-contents\">Table of contents <!-- omit in toc --><\/h1>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#premise\">Premise<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#the-documentation-renaissance\">The documentation renaissance<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#why-agents-love-good-docs\">Why agents love good docs<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#context-is-king\">Context is King<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#consistency-matters\">Consistency Matters<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#examples-drive-understanding\">Examples Drive Understanding<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#no-more-an-afterthought\">No more an afterthought<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#the-new-documentation-paradigm\">The new documentation paradigm<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#documentation-as-infrastructure\">Documentation as Infrastructure<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#living-documentation\">Living Documentation<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#multi-audience-optimization\">Multi-Audience Optimization<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#my-tips-for-agent-friendly-documentation\">My tips for agent-friendly documentation<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#structure-for-scannability\">Structure for Scannability<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#keep-it-concise-yet-meaningful\">Keep It Concise Yet Meaningful<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#be-explicit-about-context\">Be Explicit About Context<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#rich-examples-and-use-cases\">Rich Examples and Use Cases<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#maintain-accuracy\">Maintain Accuracy<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/documentation-is-cool-again\/#final-thoughts\">Final thoughts<\/a><\/li>\n<\/ul>\n<h1 id=\"premise\">Premise<\/h1>\n<p>Unless you've been living in a cave for the last few months, you've probably noticed the explosion of AI-powered development tools. Claude, GitHub Copilot, Cursor, Windsurf, and a dozen other agents have burst onto the scene, promising to revolutionize how we write code. And honestly? They're pretty incredible.<\/p>\n<p>These tools can generate entire functions from a simple comment, refactor legacy code in seconds, and even debug complex issues faster than many seasoned developers. It feels like having a brilliant junior developer sitting right next to you - one who never gets tired, never needs coffee breaks, and can instantly recall patterns from millions of codebases.<\/p>\n<p>But here's the thing: like any promising junior developer, these agents are far from perfect. They can write elegant solutions one moment and produce completely broken code the next. They might confidently suggest an approach that worked well in their training data but completely misses the specific context of your project. They excel at pattern matching but often struggle with the nuanced understanding that comes from knowing your business logic, your team's conventions, or the historical reasons why certain decisions were made.<\/p>\n<p>The difference between getting excellent results and frustrating failures often comes down to one thing: context. Give these agents rich, well-structured information about your codebase, and they'll surprise you with their capabilities. Leave them to guess, and you'll spend more time fixing their mistakes than you would have writing the code yourself.<\/p>\n<p>This is where documentation becomes not just helpful, but absolutely critical.<\/p>\n<h1 id=\"the-documentation-renaissance\">The documentation renaissance<\/h1>\n<p>For years, documentation has been treated as a necessary evil - something developers reluctantly write and rarely maintain. It was often outdated, incomplete, or buried so deep that finding the right information felt like archaeology.<\/p>\n<p>But something fundamental has changed. With the rise of AI agents and coding assistants, documentation has suddenly become cool again. And it's not just cool - it's absolutely critical.<\/p>\n<p>When I talk about <em>documentation<\/em> here, I mean more than just traditional README files and API docs. I'm referring to the entire ecosystem of knowledge that helps both humans and machines understand your code: clear comments, well-structured code, comprehensive tests, meaningful variable names, and yes, those README files too. It's all the context that makes a codebase approachable and maintainable.<\/p>\n<h1 id=\"why-agents-love-good-docs\">Why agents love good docs<\/h1>\n<p>AI agents are voracious consumers of structured information. Unlike humans who can fill in gaps with intuition and experience, agents need clear, comprehensive, and well-organized documentation to function effectively.<\/p>\n<p>Here's why good documentation is an agent's best friend:<\/p>\n<h3 id=\"context-is-king\">Context is King<\/h3>\n<p>Agents excel when they have rich context about your codebase, APIs, and business logic. Well-written documentation provides this context in a digestible format that agents can quickly parse and understand.<\/p>\n<h3 id=\"consistency-matters\">Consistency Matters<\/h3>\n<p>Agents thrive on patterns and consistency. When your documentation follows a predictable structure and style, agents can more easily extract relevant information and apply it correctly.<\/p>\n<h3 id=\"examples-drive-understanding\">Examples Drive Understanding<\/h3>\n<p>Code examples in documentation aren't just helpful for humans - they're training data for agents. The more comprehensive and accurate your examples, the better agents can understand and replicate patterns in your codebase.<\/p>\n<h1 id=\"no-more-an-afterthought\">No more an afterthought<\/h1>\n<p><img src=\"\/assets\/images\/posts\/2025-06-17_documentation-is-cool-again\/claude-code.jpg\" alt=\"Claude Code interface\" \/>\n<em>Claude reading the CLAUDE.md file to understand that it should use a just recipe for running integration tests.<\/em><\/p>\n<p>What's fascinating is that AI agents themselves are now natively accommodating and even requesting better documentation. This isn't just about agents consuming existing docs - they're actively pushing us toward better documentation practices.<\/p>\n<p>Take Claude, for example. When working with code, Claude can automatically look for and utilize <code>CLAUDE.md<\/code> files in projects. These files serve as specialized documentation that provides context, preferences, and guidelines specifically for AI agents working on the codebase. It's documentation written by humans, for machines, that ultimately benefits everyone.<\/p>\n<p>I recently added a <code>CLAUDE.md<\/code> file to one of my company's <a rel=\"external\" href=\"https:\/\/github.com\/TrueLayer\/truelayer-java\/blob\/main\/CLAUDE.md\">open source projects<\/a>. This file includes everything from build commands and testing guidelines to pull request conventions and release processes. It's like having a detailed onboarding document that agents can reference to understand not just what the code does, but how the team works with it.<\/p>\n<p>This represents a fundamental shift: we're moving from documentation as an afterthought to documentation as a first-class citizen in our development workflow. Agents aren't just tolerating our poor documentation habits - they're incentivizing us to do better.<\/p>\n<p>Claude's CLAUDE.md approach isn't the only emerging standard in this space. The community is also developing other conventions like <a rel=\"external\" href=\"https:\/\/llmstxt.org\/\">llms.txt<\/a>, which proposes a standardized <code>\/llms.txt<\/code> file that websites can use to provide LLM-friendly content. This file offers curated information specifically formatted for AI consumption, helping agents quickly understand and navigate website content without having to parse complex HTML structures.<\/p>\n<p>These emerging standards signal that we're entering a new era where documentation designed for AI agents is becoming just as important as documentation designed for humans - and often, the same principles that make docs great for agents make them better for humans too.<\/p>\n<h1 id=\"the-new-documentation-paradigm\">The new documentation paradigm<\/h1>\n<p>The agent revolution is changing how we think about documentation:<\/p>\n<h3 id=\"documentation-as-infrastructure\">Documentation as Infrastructure<\/h3>\n<p>Good docs are no longer just nice-to-have - they're infrastructure. They're the foundation that enables agents to understand, navigate, and contribute to your codebase effectively.<\/p>\n<h3 id=\"living-documentation\">Living Documentation<\/h3>\n<p>With agents helping to maintain and update documentation, we can finally achieve the dream of living docs that stay current with the code.<\/p>\n<h3 id=\"multi-audience-optimization\">Multi-Audience Optimization<\/h3>\n<p>Modern documentation needs to serve both human developers and AI agents. This dual audience is actually making docs better for everyone.<\/p>\n<h1 id=\"my-tips-for-agent-friendly-documentation\">My tips for agent-friendly documentation<\/h1>\n<p>Through my first experiences working with AI agents and implementing documentation standards like CLAUDE.md files, I've learned a few things about what makes documentation truly effective for both humans and machines. Here are the practices that have worked best for me:<\/p>\n<h3 id=\"structure-for-scannability\">Structure for Scannability<\/h3>\n<ul>\n<li>Use clear headings and subheadings<\/li>\n<li>Include table of contents for longer documents<\/li>\n<li>Organize information hierarchically<\/li>\n<li>Use consistent formatting and conventions<\/li>\n<\/ul>\n<h3 id=\"keep-it-concise-yet-meaningful\">Keep It Concise Yet Meaningful<\/h3>\n<ul>\n<li>Focus on essential information that adds real value<\/li>\n<li>Avoid verbose explanations when simple ones suffice<\/li>\n<li>Use lightweight formats like Markdown that minimize structural overhead<\/li>\n<li>The best documentation often has the least formatting complexity<\/li>\n<\/ul>\n<h3 id=\"be-explicit-about-context\">Be Explicit About Context<\/h3>\n<ul>\n<li>Include purpose statements for modules and functions<\/li>\n<li>Document assumptions and constraints<\/li>\n<li>Explain the \"why\" behind design decisions<\/li>\n<li>Provide business context when relevant<\/li>\n<\/ul>\n<h3 id=\"rich-examples-and-use-cases\">Rich Examples and Use Cases<\/h3>\n<ul>\n<li>Include comprehensive code examples<\/li>\n<li>Show common usage patterns<\/li>\n<li>Document edge cases and error handling<\/li>\n<li>Provide end-to-end scenarios<\/li>\n<\/ul>\n<h3 id=\"maintain-accuracy\">Maintain Accuracy<\/h3>\n<ul>\n<li>Keep documentation in sync with code changes<\/li>\n<li>Use automated tools to validate examples<\/li>\n<li>Regular audits and updates<\/li>\n<li>Version documentation alongside code<\/li>\n<\/ul>\n<h1 id=\"final-thoughts\">Final thoughts<\/h1>\n<p>The rise of AI agents has fundamentally shifted the value proposition of documentation. What was once a chore has become a competitive advantage. Teams with excellent documentation will find their agents more capable, their onboarding faster, and their development cycles more efficient.<\/p>\n<p>Documentation isn't just cool again - it's becoming the secret weapon of high-performing development teams. The question isn't whether you can afford to invest in better documentation, but whether you can afford not to.<\/p>\n<p>In my opinion, every software engineer should start using these AI development tools today. They're not perfect, but they're already transformative when used correctly. Think of them as multipliers: if you feed them good practices, clear documentation, and well-structured code, they'll amplify these positive inputs and help you achieve better outcomes. But multiply poor practices by AI assistance, and you'll only get worse results faster.<\/p>\n<p>The engineers who learn to work effectively with AI agents - who understand how to provide context, write clear documentation, and structure their projects for both human and machine consumption - will have a significant advantage in the years to come. Don't wait for these tools to get better. Start learning how to work with them now.<\/p>\n<p>In the age of agents, good documentation isn't just helping humans understand your code - it's teaching machines to be better partners in building it.<\/p>\n"},{"title":"The fine line between mistake and innovation in LLMs","pubDate":"Mon, 10 Mar 2025 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/","guid":"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/","description":"<p><img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/artwork.png\" alt=\"Imaginary artwork\" \/>\n<em>Imaginary artwork, combining elements of unknown and excitement, generated with <a rel=\"external\" href=\"https:\/\/diffusionbee.com\/\">DiffusionBee<\/a>.<\/em><\/p>\n<!-- no toc -->\n<h1 id=\"table-of-contents\">Table of contents <!-- omit in toc --><\/h1>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#premise\">Premise<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#hallucinations\">Hallucinations<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#why-do-they-happen\">Why do they happen?<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#knowledge-cutoff\">Knowledge cutoff<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#biases-and-gaps-in-training-data\">Biases and gaps in training data<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#embeddings-are-numeric-representations\">Embeddings are numeric representations<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#what-can-we-do-about-it\">What can we do about it?<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#what-llms-do-about-it\">What LLMs do about it?<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#tools\">Tools<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#reinforcement-learning\">Reinforcement learning<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#training-excursus\">Training excursus<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#a-trial-and-error-approach\">A trial and error approach<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#self-evolution\">Self-evolution<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#chains-of-thoughts-and-the-aha-moment\">Chains of thoughts and the \"Aha moment\"<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/the-fine-line-between-mistake-and-innovation-in-llms\/#a-gateway-to-future-breakthroughs\">A gateway to future breakthroughs?<\/a><\/li>\n<\/ul>\n<h1 id=\"premise\">Premise<\/h1>\n<p>The last few years have been undoubtedly dominated by the rise of AI, and the many breakthroughs in the field of Large Language Models.<\/p>\n<p>I still remember my sense of disbelief when I first interacted with OpenAI\u2019s ChatGPT (GPT-3.5) and my excitement when generating images with DALL-E. The same thrill came with my first encounters with AI-augmented tools like GitHub Copilot. Then came GPT-4 with advanced tools like web search, Anthropic\u2019s debut, the first open-weight models from Meta, and the overwhelming moment of opening the Hugging Face website for the first time. Along the way, there were some lowlights too - the temporary ChatGPT ban in Italy, the overhyped launch of Google\u2019s Gemini - but then again, we saw the emergence of Sora (haven\u2019t tried it yet), Apple Intelligence, and newer thinking models transparently leveraging chain-of-thought reasoning, like DeepSeek\u2019s R1 or Claude 3.7. Meanwhile, incredibly powerful tools like Cursor and Copilot\u2019s agent mode are reshaping how we interact with LLMs.<\/p>\n<p>In this rollercoaster of innovations (and associated emotions), I quickly realized I needed to keep up with the pace of change. I started following AI-related feeds, reading articles, occasionally diving into books about AI and its future possible consequences. I've taken online courses on the fundamentals of machine learning and deep learning, and I've started integrating AI more and more into my daily routine - at work, in personal projects, as a learning tool, or simply to make life easier.<\/p>\n<p>This personal journey - still in its early days and constantly evolving - has been driven by two main reasons. The first, perhaps less noble, is that I didn\u2019t want to feel left behind: I've felt and still feel like it's very easy to feel overwhelmed by AI-related news, and I was frustrated by my growing struggle to keep up with the latest in software engineering. The second, and more important, reason is genuine fascination. When used correctly, LLMs already deliver remarkable results, and their potential to help humanity solve global challenges seems to me huge.<\/p>\n<p>It\u2019s precisely this last realization that inspired me to write this post. The idea had been lingering for a while, but only in the past few days did I find the right words to express my thoughts \u2014 also inspired by this <a rel=\"external\" href=\"https:\/\/www.youtube.com\/watch?v=7xTGNNLPyMI\">deep-dive on LLMs<\/a> by Andrej Karpathy.<\/p>\n<h1 id=\"hallucinations\">Hallucinations<\/h1>\n<p>If you have tried prompting an LLM a few times, there's a good chance you've stumbled across some funky results: large language models might indeed fail at very basic tasks. This is usually not a significant problem for results that are easy to verify: it's easy to have a laugh at absurd results and rely on other tools. It\u2019s less straightforward to notice hallucinations in domains that are not easy to verify, which could lead to problems for users who aren\u2019t cautious when relying on LLM outputs: you get a result that looks very promising in theory, except that it's all made-up and completely wrong.<\/p>\n<p>Here's a few examples:<\/p>\n<div class=\"carousel\">\n  <div class=\"carousel-inner\">\n    <div class=\"carousel-item\">\n      <div class=\"carousel-image-container\">\n        <img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/math_failure_tweet.png\" alt=\"Math Failure\">\n      <\/div>\n      <em class=\"carousel-caption\">\n        A famous <a href=\"https:\/\/x.com\/goodside\/status\/1812977352085020680\">tweet<\/a> about a GPT-4o mistake.\n      <\/em>\n    <\/div>\n    <div class=\"carousel-item\">\n      <div class=\"carousel-image-container\">\n        <img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/who_is_me.png\" alt=\"Who is Andrea Di Lisio?\">\n      <\/div>\n      <div class=\"carousel-caption\">\n        <a href=\"https:\/\/chat.deepseek.com\">DeepSeek R1<\/a> hallucinating a completely random version of myself.\n      <\/div>\n    <\/div>\n    <div class=\"carousel-item\">\n      <div class=\"carousel-image-container\">\n        <img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/how_many_r_strawberry.png\" alt=\"How many r in strawberry?\">\n      <\/div>\n      <div class=\"carousel-caption\">\n        Another well-known prompt, failed by Anthropic's <a href=\"https:\/\/claude.ai\">Claude<\/a> model\n      <\/div>\n    <\/div>\n  <\/div>\n  <button class=\"prev\" onclick=\"moveSlide(-1)\">\u276e<\/button>\n  <button class=\"next\" onclick=\"moveSlide(1)\">\u276f<\/button>\n<\/div>\n<p>It's tempting to use these failures as an opportunity to write a like-bait tweet ranting about LLMs and dismissing their future entirely. But not only is that pointless and likely to miss the mark, it's also a lost chance to better understand LLMs and foster constructive discussions within the AI community.<\/p>\n<p>Before seeing how these mistakes are either already mitigated in more advanced and recent models, or could be prevented with simple tricks at prompting time, let's see why such things are happening.<\/p>\n<h1 id=\"why-do-they-happen\">Why do they happen?<\/h1>\n<h2 id=\"knowledge-cutoff\">Knowledge cutoff<\/h2>\n<p>First of all there's a very basic reason why models sometimes hallucinate. In the so called <em>pre-training<\/em> phase LLMs are trained on a massive \"zip file\" of the internet, which is by definition time-bound, thus up to date till a particular point in time. Also, since this file representing the internet is huge, training usually takes several months.\nTo give an example, rumors say that GPT-4 was fed with 13 trillion tokens and was trained over a period of three months (<a rel=\"external\" href=\"https:\/\/seifeur.com\/gpt-4-training-time\/\">source<\/a>).<\/p>\n<p>Therefore, it is not hard to realise that the knowledge acquired by LLMs might be significantly out of date by the time we run our first prompt.<\/p>\n<h2 id=\"biases-and-gaps-in-training-data\">Biases and gaps in training data<\/h2>\n<p>The extra-large datasets on which LLMs are trained have gaps in the best case, and contain false or biased information in the worst.<\/p>\n<p>Unless explicitly instructed to act differently, LLMs always try to give the best probabilistic answer to the given prompt, and they might prefer hallucinating something that <em>could<\/em> be likely, but it turns out that's totally fake or wrong.<\/p>\n<h2 id=\"embeddings-are-numeric-representations\">Embeddings are numeric representations<\/h2>\n<p>Last but not least, we should always keep in mind that whatever we write in the chat prompt is first translated into a sequence of machine-readable identifiers, or tokens, before reaching the core of LLMs. Personally, mentally visualizing this step helps me understand what happens at inference time: imagining my prompts as a sequence of numbers pushes me to think of the model as processing them numerically, rather than simulating a real thought process.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/tiktokenizer.png\" alt=\"Tokenization start\" \/>\n<em><em>Who is Andrea Di Lisio?<\/em> translated into tokens on <a rel=\"external\" href=\"https:\/\/tiktokenizer.vercel.app\/\">Tiktokenizer<\/a>. Note how a pre-processing step is also added before translating, to maintain the structure of the conversation between the user and the assistant.<\/em><\/p>\n<p>Not only do our words lose their immediate meaning before reaching the LLM, but they are also split in ways that don't necessarily align with human language rules. As visible in the above screenshot, my last name <code>Di Lisio<\/code> is broken down into three separate tokens.\nThis step alone already suggests that an LLM does not think the way humans do, but there's more!<\/p>\n<p>We should also bear in mind that before inference can happen, tokens go through an additional translation and normalization steps: they are converted into numerical vectors in the embedding layer, and since they are evaluated in parallel by transformers - at the core of LLMs architectures - they are also enriched with positional attributes, meant to convey to the model the order in which each token appears in our sequence of tokens.<\/p>\n<p>Here's a graphical, high-level representation of the translations steps happening before inference:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span>&quot;Andrea&quot;  <\/span><\/span>\n<span class=\"giallo-l\"><span>\u2192 Tokenization \u2192 10294  <\/span><\/span>\n<span class=\"giallo-l\"><span>\u2192 Dense Vector \u2192 [ 0.23, -0.45, 1.03, ..., -0.67 ]  <\/span><\/span>\n<span class=\"giallo-l\"><span>\u2192 Positional Encoding \u2192 [ 0.24, -0.43, 1.06, ..., -0.63 ]  <\/span><\/span><\/code><\/pre>\n<p>At inference time, the objective of the large language model is to compute what's the best possible token given the user input, and continue doing so until the answer is considered complete, that is the next token with the highest propability is the one representing a closing tag. By <em>best<\/em> token I mean the one with the highest score among a set of possible tokens, as computed by the network.<\/p>\n<p>Understanding the basics of embeddings should help shift our mental model away from the idea that an LLM thinks like we do. It's all about probabilistic decision-making, where the order in which words are written does a whole lot of difference.<\/p>\n<h1 id=\"what-can-we-do-about-it\">What can we do about it?<\/h1>\n<p>There's not much we can do at prompt-time to mitigate knowledge gaps. The best we can do in relation to that is either make sure to explicitly opt-in for web-search-based answers, or include in our queries clear instructions about the fact that an external search engine should be used to answer.\nThere isn't a recipe against biases either, and a naive user does not even realize if\/how they manifest: the best we can do in this space is keeping this possibility in mind.<\/p>\n<p>What we can do though is ask questions in a way that simplifies the work of LLMs at inference and leads to better predictions. The best prompts have the following traits:<\/p>\n<ul>\n<li>They are specific questions<\/li>\n<li>They give some context about what is being asked<\/li>\n<li>They are role-based (Eg. <code>As a software engineer I want to...<\/code>)<\/li>\n<li>They come with examples of what we expect from the model<\/li>\n<li>They are part of an iterative conversation<\/li>\n<\/ul>\n<p>Hallucinations are a concrete and significant problem though, usually affecting the reputation of a model and probably - by extension - the one of the company that released it.\nThus, there have been significant investments and improvements in solving the hallucinations problem within the LLMs themselves.<\/p>\n<h1 id=\"what-llms-do-about-it\">What LLMs do about it?<\/h1>\n<h2 id=\"tools\">Tools<\/h2>\n<p>As hinted when talking about knowledge gaps, when no knowledge is found for a particular topic, certain LLMs prefer back-filling those holes in real-time with the help of so called <em>tools<\/em>, when they detect gaps.<\/p>\n<p>Recent OpenAI's model, Grok, or DeepSeek R1, have all a search function available that pulls fresh data via web searches on known sources (Google, DuckDuckGo, Bing, other internal systems...), select the most relevant results, and add them to the\ncontext window of the conversation, before running inference.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/grok.png\" alt=\"Grok&#39;s Deepsearch\" \/>\n<em>Grok3 and its <em>DeepSearch<\/em> function.<\/em><\/p>\n<p>Not only such tools greatly mitigate hallucinations, I found they're great at providing a self-contained picture for what I'm looking for. They're an incredibly powerful tool to solve the fragmentation problem usually associated to news. I love it!<\/p>\n<h2 id=\"reinforcement-learning\">Reinforcement learning<\/h2>\n<p>Before entering the core part of this post, I necessarily have to slightly diverge for a brief excursus on the training process for LLMs.<\/p>\n<h3 id=\"training-excursus\">Training excursus<\/h3>\n<p>When we talk about training in the context of LLMs we usually refer to a process that's made of two main stages: <em>pre-training<\/em> and <em>post-training<\/em>. The first is the most expensive and time-consuming one,\nwhose output is a so-called <em>base<\/em> model. The latter is a way more lightweight training phase, which is usually made of <em>supervised fine-tuning<\/em> (SFT) and <em>reinforcement learning<\/em> (RL).<\/p>\n<p>At the beginning of pre-training we have a model with some randomly initialized values for its <em>parameters<\/em> (billions, if not trillions of them), acting as coefficients for a very complex, multi-polynomial mathematical function. The initial model's prediction for the next token are terrible and lead to huge\ndifferences between predicted and actual values. In other words, at the beginning of pre-training the <em>loss<\/em> of the model is very high.\nIt's thanks to <em>gradient descent<\/em> and <em>backward propagation<\/em> - and many thousands of iterations, or <em>epochs<\/em> - that the loss starts decreasing up until it reaches a plateau, which is also hopefully a very good - that is, low - value.\nWhen that happens, the pre-training is over, and the weights of the model are saved into the base model.<\/p>\n<p>While I've kept it short for simplicity here, we should bear in mind that this step usually takes several months and millions of dollars, hence why frontier models are a privilige of multi-billions-investors-baked companies, like OpenAI, Anthropic, X, Meta, Google and so and so forth.<\/p>\n<p>The base model has a very good knowledge of the internet, but it acts more like a token simulator, than an <em>assistant<\/em>: it is more prone to vomit facts and remixes of the internet, rather than replying to our prompts and <em>articulating<\/em> asnwers. For this reason, companies that work in the LLM space usually don't release\nmodels at this stage.<\/p>\n<p>To get the human touch, these models have to go through supervised fine-tuning and reinforcement learning in the post-training phase.<\/p>\n<p>As the name suggests, during SFT human supervisors play a central role: usually in this phase the LLMs company hire specialised human labelers to create artificial conversations and feed the models with those. They both create prompts and come up with ideal responses that are expected by models (like the ones contained in this public <a rel=\"external\" href=\"https:\/\/huggingface.co\/datasets\/OpenAssistant\/oasst2\">dataset<\/a>),\nfollowing precise labeling instructions and conversation requirements provided by the same LLM company. In this phase models usually improve their predictions looking at existing solutions\/conversations, and they learn by - let's say - <em>imitation<\/em>.<\/p>\n<h3 id=\"a-trial-and-error-approach\">A trial and error approach<\/h3>\n<p>In contrast, reinforcement learning is more a <em>trial and error<\/em> phase where models are given a problem and they are pushed to find the best solution for that, based on their previous knowledge acquired and refined during pretraining and SFT.\nIt is not very different from when we're studying something: after having read about the theory about a topic (pre-training), and perhaps having seen a few example applications\/solutions for that (supervised fine-tuning), we test our knowledge and understanding with the help of <em>new<\/em> practice problems (reinforcement learning).<\/p>\n<p>In this phase models are given questions or problems to solve that did not appear during the previous training stages, and the ask is for them to come with <em>N<\/em> different answers\/solution for those, and amongst the ones leading to the right answer\/result, select the best, either by running a self-assessment or with the help of an external actor. Once the best solution is identified, the model is then trained on that, thus the probability\nof them picking that same sequence of tokens when dealing with the same or related prompt in future slightly increases.<\/p>\n<h3 id=\"self-evolution\">Self-evolution<\/h3>\n<p>LLMs can self-assess their result in the case of <em>verifiable domains<\/em>: the correct answer\/solution is available upfront, and it's relatively straightforward to check that the solution matches the expected one. In presence of multiple right answers, the best option is selected based on different criteria, like elaboration time, memory consumption, number of tokens, or other conditions not necessarily of easy understanding for humans!<\/p>\n<p>It is important to notice that what <em>best<\/em> means in this case does not necessarily match our human-specific intuition: LLMs have to figure out themselves what is the answer (or sequence of tokens) that - given a prompt - reliably leads to the correct result, humans are deliberately left out of this process!<\/p>\n<p>This concept is what makes reinforcement learning very different from the other stages of learning. It almost feels as if scientists are taking a step back in this phase and giving LLMs the chance to discover new, unforeseen solutions to existing problems, and broader, generic <em>thinking<\/em> patterns\/approaches\/tools.<\/p>\n<h3 id=\"chains-of-thoughts-and-the-aha-moment\">Chains of thoughts and the \"Aha moment\"<\/h3>\n<p>There is limited literature on reinforcement learning for LLMs due to its nascent nature. With the release of their <em>R1<\/em> model and the associated paper - <a rel=\"external\" href=\"https:\/\/arxiv.org\/abs\/2501.12948\"><em>DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning<\/em><\/a> - DeepSeek has been the first company to talk about this process more publicly and because of that, this publication has really made an impact in the AI community.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2025-03-10_the_fine_line_between_mistake_and_innovation_in_llms\/plot_length.png\" alt=\"Thinking time growth in R1\" \/>\n<em>During training, R1 learns to allocate more thinking time to solve tasks.<\/em><\/p>\n<p>Not only they have publicly spoken about the details of their work, and released 6 distilled models of <em>R1<\/em>, they also have also expanded on the concept of the <em>chains of thoughts<\/em>, and talked about an \"Aha Moment\" that researches have observed while training an intermediate state of the model, a moment when the model realised that it could have achieved better results on its own that by allocating more time to thinking:<\/p>\n<blockquote>\n<p>[...] During this phase, DeepSeek-R1-Zero learns to allocate more thinking time to a problem by reevaluating its initial approach. This behavior is not only a testament to the model\u2019s growing reasoning abilities but also a captivating example of how reinforcement learning can lead to unexpected and sophisticated outcomes.<\/p>\n<p>This moment is not only an \u201caha moment\u201d for the model but also for the researchers observing its behavior. It underscores the power and beauty of reinforcement learning: rather than explicitly teaching the model on how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies. The \u201caha moment\u201d serves as a powerful reminder of the potential of RL to unlock new levels of intelligence in artificial systems, paving the way for more autonomous and adaptive models in the future.<\/p>\n<\/blockquote>\n<p>Speaking transparently about these new developments in reinforcement learning has stimulated further research in the field. Just in the last few days, <a rel=\"external\" href=\"https:\/\/qwenlm.github.io\/blog\/qwq-32b\/\">Qwen claimed to have reached results comparable to R1 with just a 32-billion-parameter model<\/a>, as opposed to R1's 671 billion parameters.\nIf confirmed, this breakthrough suggests that a model trained on a fairly standard setup (potentially even locally) can perform on par with frontier-level LLMs released by OpenAI, Meta, Anthropic, and others.<\/p>\n<p>Almost in parallel, <a rel=\"external\" href=\"https:\/\/openai.com\/index\/introducing-gpt-4-5\/\">OpenAI released GPT 4.5<\/a>, a non-reasoning model which is rumored to have between 5 and 10 trillion parameters, although early indications suggest it may not represent a significant leap in reasoning capabilities. This trend hints that merely increasing parameters may have reached its limits, and future research might need to explore alternative improvements.<\/p>\n<h1 id=\"a-gateway-to-future-breakthroughs\">A gateway to future breakthroughs?<\/h1>\n<p>The rapid advancements in LLMs over the past few months have been nothing short of fascinating. More than once, they have reminded me of <a rel=\"external\" href=\"https:\/\/www.amazon.it\/Wohpe-Salvatore-Sanfilippo\/dp\/B09XT6J3WX\"><em>Wohpe<\/em><\/a>, a sci-fi novel by Salvatore Sanfilippo that I read a couple of years ago.<\/p>\n<p>More and more, It seems that researchers are beginning to realize that LLMs are not just improving at processing and articulating existing knowledge, but are also developing more generalized ways of reasoning - bending what they\u2019ve learned to tackle problems humans have yet to solve. And - to jutify the title of my post - this is all possible thanks to the probabilistic, error-prone but also flexible nature of LLMs, which some critics dismiss as mere <em>stochastic parrots<\/em>.<\/p>\n<p>If that turns out to be true, we may be at the threshold of something much bigger than just better and better LLMs. But whether this intuition holds or if I\u2019m simply caught up in the excitement - I suppose time will tell. Either way, I can\u2019t wait to find out \ud83d\ude04.<\/p>\n"},{"title":"Yet another connection reset","pubDate":"Fri, 18 Oct 2024 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/yet-another-connection-reset\/","guid":"https:\/\/adilisio.com\/posts\/yet-another-connection-reset\/","description":"<p><img src=\"\/assets\/images\/posts\/2024-10-18_yet_another_connection_reset\/reset.png\" alt=\"Connection reset\" \/>\n<em>A sort-of accurate representation of a scissor cutting connections to my Kestrel server, as hallucinated by OpenAI<\/em><\/p>\n<p>This is meant to be a short post about a bad day I had a work recently.<\/p>\n<h1 id=\"initial-context\">Initial context<\/h1>\n<p>My team had to expose a new API over gRPC from one of our .NET Core backend applications (from now on named <em>Service C<\/em>) to be consumed by a downstream service (from now on named <em>Service A<\/em>), similarly implemented using .NET core. The 2 services had to communicate over HTTPS within our Kubernetes cluster.<\/p>\n<p>While we're slowly stepping into the magic world of service meshes (thanks to <a rel=\"external\" href=\"https:\/\/linkerd.io\/\">LinkerD<\/a>), historically we have been solving all these connectivity-related challenges with the help of <a rel=\"external\" href=\"https:\/\/www.envoyproxy.io\/\">Envoy<\/a>, deployed as sidecar in our application pods. Envoy was originally introduced as a drop-in technology to implement gRPC load-balancing, and a kind-of transparent solution for TLS termination, with or without mutual TLS. It's a widely adopted technology in my company, and we keep using it, especially on existing applications serving live traffic, like <em>Service C<\/em>.<\/p>\n<p>However, oddly enough, in our <em>Service C<\/em> Envoy was used only as egress to communicate with downstream services. Why? Maybe at the time <em>Service A<\/em> was integrated with our gRPC API, we had a tight deadline for going live, and considering that <em>Service A<\/em>'s gRPC .NET client was capable of load balancing, we probably preferred taking the shortest route and decided to not use Envoy at all as ingress, but simply have our .NET core application accepting requests over HTTPs and be responsible for TLS termination... Or, at least, this is how I explained it to myself.<\/p>\n<p>This is more or less the initial setup we had:<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-10-18_yet_another_connection_reset\/initial_setup.png\" alt=\"Initial setup\" \/>\n<em><em>Service A<\/em> calls <em>Service C<\/em> over HTTPS. At this point it's not clear what's the benefit of using Envoy on <em>Service C<\/em>...<\/em><\/p>\n<h1 id=\"moving-on\">Moving on<\/h1>\n<p>Then, a few days ago my team had to onboard a new service, namely <em>Service B<\/em>, over the same gRPC API. In this case, also because we had a mutual TLS requirement, I suggested to start using Envoy also as ingress and TLS terminator. So, I did the following changes, mainly:<\/p>\n<ol>\n<li>I changed our .NET app to start accepting requests on HTTP only;<\/li>\n<li>I added a new TLS-only ingress on our Envoy sidecar, to be used by <em>Service A<\/em>;<\/li>\n<li>I basically kept the existing mTLS ingress on <em>Service C<\/em> as it was, and I routed the new traffic coming from <em>Service<\/em> B onto it.<\/li>\n<\/ol>\n<p>An important requirement for the above changes was that nothing should have changed from the perspective of <em>Service A<\/em>, to not impact live traffic. The new setup looked like something like this:<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-10-18_yet_another_connection_reset\/final_setup.png\" alt=\"Final setup\" \/>\n<em>Both <em>Service A<\/em> and Service B reach <em>Service C<\/em> via its Envoy ingress. The .NET core container within <em>Service C<\/em> expects plain HTTP requests only.<\/em><\/p>\n<p>Everything looked linear and clear on paper, but it was not working \ud83d\ude05.<\/p>\n<h1 id=\"all-that-glitters-is-not-gold\">All that glitters is not gold<\/h1>\n<p>I did not face particular problems with <em>Service B<\/em> and <em>C<\/em>, it was the interaction between <em>A<\/em> and <em>C<\/em> that turned out to be particularly painful.<\/p>\n<h2 id=\"application-layer-protocol-negotiation-alpn\">Application-Layer Protocol Negotiation (ALPN)<\/h2>\n<p>The first error I've encountered was very explicit and therefore quite straightforward to solve:<\/p>\n<blockquote>\n<p>Error starting gRPC call. HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while server offers only version fallback.<\/p>\n<\/blockquote>\n<p>After some <a rel=\"external\" href=\"https:\/\/github.com\/grpc\/grpc-dotnet\/issues\/1287\">digging<\/a>, it turned out that the ingress on <em>Service C<\/em> was only offering HTTP\/1.1 as available protocol by default, whereas the .NET client included in <em>Service A<\/em> could only communicate using HTTP\/2.<\/p>\n<p>I solved this problem by declaring explicitly the ALPN protocol that the ingress on Service C was offering, on the Envoy configuration*<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">listeners<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> name<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> tls_internal_ingress<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    address<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 0.0.0.0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    port<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 3001<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    tls<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      enabled<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> true<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      alpn_protocols<\/span><span>: <\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #9ECBFF;\"> &quot;h2&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      certs_mount_path<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> \/certs\/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      mtls<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        enabled<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> false<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    o11y<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      enabled<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> true<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      http<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        kind<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> server<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    virtual_hosts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        domains<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #9ECBFF;\"> &quot;*&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        routes<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> match<\/span><span>: {<\/span><span style=\"color: #85E89D;\"> prefix<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;\/&quot;<\/span><span>,<\/span><span style=\"color: #85E89D;\"> grpc<\/span><span>: { } }<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            route<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              cluster<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> signup_plus_grpc_cluster<\/span><\/span><\/code><\/pre>\n<p>* note that the above configuration snippet mixes Envoy's native configuration syntax with a simple abstraction layer that we build on top of <a rel=\"external\" href=\"https:\/\/helm.sh\/\">Helm<\/a>.<\/p>\n<h2 id=\"alternate-schemes\">Alternate schemes<\/h2>\n<p>The second problem I've faced is the one that made me mad for a few hours, and the reason why I've decided writing this <del>rant<\/del> post. This one got me crazy because all I had from the logs of the Envoy ingress on Service C was a <code>Connection reset<\/code> error with a vague <code>upstream reset: reset reason: remote reset, transport failure reason:<\/code> error message.<\/p>\n<p>After a few hours trying to explain myself what I had done wrong that day to deserve such destiny, I noticed that Envoy logs were mentioning a specific <code>UR<\/code> response flag. After some back and forth on Envoy's <a rel=\"external\" href=\"https:\/\/www.envoyproxy.io\/docs\/envoy\/latest\/configuration\/observability\/access_log\/usage\">public documentation<\/a> (sigh), I've found the following bits:<\/p>\n<blockquote>\n<p>UpstreamRemoteReset  |  UR  | Upstream remote reset in addition to 503 response code.<\/p>\n<\/blockquote>\n<p>Which convinced me that the problem could only have been on the .NET application within <em>Service C<\/em>. So I've moved on in that direction, and I've discovered something else:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span>...<\/span><\/span>\n<span class=\"giallo-l\"><span>service-c-5c9f4f979-kbpdm envoy-proxy &#39;:method&#39;, &#39;POST&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>service-c-5c9f4f979-kbpdm envoy-proxy &#39;:scheme&#39;, &#39;https&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>service-c-5c9f4f979-kbpdm envoy-proxy &#39;:authority&#39;, &#39;service-c.verification.svc.cluster.local:3001&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>...<\/span><\/span><\/code><\/pre>\n<p>The weird thing was that the request originating from the grpc-dotnet client of <em>Service A<\/em> was left as-is from the Envoy ingress on <em>Service C<\/em>. More specifically, the <code>https<\/code> scheme was not transformed after the TLS termination offered by Envoy.\nMy thinking got a confirmation also from <a rel=\"external\" href=\"https:\/\/www.envoyproxy.io\/docs\/envoy\/latest\/configuration\/http\/http_conn_man\/headers#scheme\">Envoy docs<\/a>:<\/p>\n<blockquote>\n<p>For HTTP\/2, and HTTP\/3, incoming :scheme headers are trusted and propagated through upstream<\/p>\n<\/blockquote>\n<p>But... that was probably unexpected from the perspective of the .NET app within <em>Service C<\/em> because it was accepting only requests in plain...<\/p>\n<p>And... finally! I realized that the <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/api\/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes?view=aspnetcore-8.0\"><code>AllowAlternateScheme<\/code> property of Kestrel<\/a> - the technology behind the .NET gRPC server on Service C - being set to <code>false<\/code> by default was the issue in this case. Toggling that flag to true put an end to my sorrows! \ud83c\udf8a<\/p>\n<p>PS: in hindsight, another approach could have been to take over the <a rel=\"external\" href=\"https:\/\/www.envoyproxy.io\/docs\/envoy\/latest\/api-v3\/extensions\/filters\/network\/http_connection_manager\/v3\/http_connection_manager.proto#envoy-v3-api-field-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-scheme-header-transformation\">default scheme manipulation on Envoy<\/a>, but at that point my brain was fried and the last thing on earth I wanted to do was fighting again with Envoy... That being said, changing this default behavior on Envoy is probably a better choice, to not pollute the application code with networking-specific details.<\/p>\n"},{"title":"Automating my gym bookings with a serverless assistant","pubDate":"Tue, 27 Aug 2024 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/","guid":"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/","description":"<p><img src=\"\/assets\/images\/posts\/2024-07-28_automating-my-gym-bookings-with-a-serverless-assistant\/gym-booking-assistant.webp\" alt=\"Gym booking assitant\" \/>\n<em>My assistant as imagined by GPT-4o<\/em><\/p>\n<!-- no toc -->\n<h1 id=\"table-of-contents\">Table of contents <!-- omit in toc --><\/h1>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#foreword\">Foreword<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#traffic-analysis\">Traffic analysis<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#requirements\">Requirements<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#implementation-pillars\">Implementation pillars<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#choice-of-language\">Choice of language<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#no-lambda-no-party\">No Lambda, no party<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#standing-on-the-shoulders-of-eventbridge\">Standing on the shoulders of EventBridge<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#configuration-and-secrets-management\">Configuration and secrets management<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#events\">Events<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#classbookingavailable\">ClassBookingAvailable<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#classbookingcompleted\">ClassBookingCompleted<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#classbookingfailed\">ClassBookingFailed<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#architecture-overview\">Architecture overview<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#show-me-the-code\">Show me the code!<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#scan-function\">Scan function<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#book-function\">Book function<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#what-could-go-wrong\">What could go wrong<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#common-layer\">Common layer<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#all-that-glitters-is-not-gold\">All that glitters is not gold<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#sms-notification\">SMS notification<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#quota-request-increase-what\">Quota request increase what?<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#final-thoughts\">Final thoughts<\/a><\/li>\n<\/ul>\n<h1 id=\"foreword\">Foreword<\/h1>\n<p>Have you ever been subscribed to a gym and felt frustrated by the difficulty of booking a class? That\u2019s exactly how I felt when I decided to re-subscribe last year.<\/p>\n<p>To overcome this frustration, I decided to build a virtual booking assistant to help me never miss my favorite classes again. What started as a late-night coding experiment quickly evolved: As I added feature after feature, I realized that this project wasn\u2019t just a practical solution for me but especially a fun and engaging challenge. This blog post will take you through my coding journey.<\/p>\n<p>To the best of my knowledge, and after checking my gym's contract, I don't think the tool I've built is ultimately against my gym's rules. However, using a script to book my classes does not seem fair, especially if the targeted classes are often overbooked (which is not my case anyway). Therefore, I\u2019m not suggesting readers emulate what I\u2019ve built. I\u2019m simply sharing my experience because I find it interesting from a software engineering standpoint, hoping it might inspire something even better!<\/p>\n<h1 id=\"traffic-analysis\">Traffic analysis<\/h1>\n<p>Before starting any kind of work I had to first understand how my gym mobile application exchanged data with the API server.\nTo do this, I simulated a <a rel=\"external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Man-in-the-middle_attack\">Man-In-The-Middle attack<\/a> on my home network, using <a rel=\"external\" href=\"https:\/\/mitmproxy.org\/\">mitmproxy<\/a> as an HTTPS proxy running on my laptop.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-07-28_automating-my-gym-bookings-with-a-serverless-assistant\/mitm.png\" alt=\"Simulated MITM setup\" \/>\n<em>Simulated MITM setup<\/em><\/p>\n<p>Once routed all my Iphone internet traffic through my local proxy and installed <a rel=\"external\" href=\"https:\/\/docs.mitmproxy.org\/stable\/concepts-certificates\/#quick-setup\">Mitmproxy's CA certificate<\/a>, I was able to see in plain all the HTTPS traffic generated by my gym mobile application, while using it. After fine-tuning the proxy filters I could easily extract the API interactions I was mostly interested in: login, class search, class book and un-book...<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-07-28_automating-my-gym-bookings-with-a-serverless-assistant\/mitmproxy_details.png\" alt=\"Request intercepted on Mitmproxy CLI\" \/>\n<em>Request intercepted on Mitmproxy CLI<\/em><\/p>\n<p>Once identified the interested API interactions, I've exported them as a set of <a rel=\"external\" href=\"https:\/\/httpie.io\/\">httpie<\/a> requests, so that I can familiarize with those. I ultimately also discoverd an <a rel=\"external\" href=\"https:\/\/apidocs.mywellness.com\/\">OpenAPI spec<\/a>, but that turned out to be neither accurate nor complete.<\/p>\n<p>That being said, with the help of mitmproxy I was able to collect all the information I need to start playing with the API and understand how to automate the API calls that were required to book a class. I could then start to envision the next steps!<\/p>\n<h1 id=\"requirements\">Requirements<\/h1>\n<p>What I wanted build was fairly simple: a daemon running on a schedule that would have booked one or more classes matching some configured search criteria. Once booked a class, the daemon should have informed about the completed booking via an SMS delivered to my mobile number.<\/p>\n<p>As per non-functional requirements, I just wanted the solution to be as simple as it could be and with serverless setup. As I never had the chance to get my hands dirty with a bunch of fully managed AWS services that I could have helped this time, I decided to use <em>AWS Lambda<\/em> and <em>EventBridge<\/em> as the backbone of my solution. After some minimal investigation I realized that a few more secondary services like <em>Secrets Manager<\/em>, <em>Systems manager<\/em> and <em>Simple Notification Services (SNS)<\/em> would have fully satisfied my needs.<\/p>\n<p>As for the infrastructure, I was not really interested in coding and versioning it (eg. with the help of Terraform or similar). For this reason, the solution that I will present shortly required a bit of manual setup on the AWS console.<\/p>\n<h1 id=\"implementation-pillars\">Implementation pillars<\/h1>\n<p>Let's shed a light on the main implementation choices I did.<\/p>\n<h2 id=\"choice-of-language\">Choice of language<\/h2>\n<p>Choosing what language to use for my project was by far the most difficult decision to take, due to a mix of language-specific oddities and as a consequence of having chosen <em>Lambda<\/em> as container for my code.<\/p>\n<p>When writing the first lines of code I had opted for pure Javascript, as I knew that the support for NodeJS on Lambda was good. Also, I did not really need a strongly-typed language and wanted something that was a good fit for scripting.<\/p>\n<p>But then... When I decided to do embark on a slightly more complex journey, I thought that using a strongly-typed language would have helped me not hallucinating. So I went for Typescript: I'm definitely not an expert in the field, but I did not want to rewrite completely the code I had already written.<\/p>\n<p>Very sadly, I quickly ran into several obstacles that led me to revert to plain JavaScript. I would have expected that creating a new project using the latest version of NodeJs and a well-known test framework like Mocha would be straightforward and take only a few seconds. However, my assumption turned out to be very wrong, as also testified by the many threads available online of people struggling even with the most basic setup. After some head-banging and also thanks to some previous art kindly shared on Github, I finally managed to get what I needed. That sense of relief did not last long though: Typescript is not natively supported by Lambda and trying to understand how to compile and package my code, so that I could leverage Lambda layers really got me frustrated, at the point that I ultimately decided to go back to pure Javascript.<\/p>\n<p>Lesson learned: when planning to use <em>Lambda<\/em> take the time needed to evaluate your candidate language upfront!<\/p>\n<h2 id=\"no-lambda-no-party\">No Lambda, no party<\/h2>\n<p>As mentioned, I've decided to go with <em>AWS Lambda functions<\/em> to host my code. More precisely, I split my code into 2 different <em>Lambda<\/em> functions: one for <strong>scanning<\/strong> the available classes and one for <strong>booking<\/strong> them. I decided to segregate responsibilities into 2 different units to keep the code more readable, and also because the triggers for the 2 actions (searching and booking classes) were very different events in my mind: scanning would have worked on a schedule, booking would have been a one-off, on-demand, task.<\/p>\n<p>Organizing my code into 2 different Lambda functions pushed me to think how to not duplicate dependencies, and utilities (like the HTTP client to interact with the Gym API, or other logging related functions) on the 2 deployments. <em>Lambda<\/em> <a rel=\"external\" href=\"https:\/\/docs.aws.amazon.com\/lambda\/latest\/dg\/chapter-layers.html\">layers<\/a> came to help, although not without headaches as we'll see later.<\/p>\n<h2 id=\"standing-on-the-shoulders-of-eventbridge\">Standing on the shoulders of EventBridge<\/h2>\n<p>I decided to rely on events to let the 2 separate functions interoperate, for multiple reasons:<\/p>\n<ul>\n<li>I wanted to keep coupling limited;<\/li>\n<li>I wanted something easy to observe\/monitor, and I thought that events would have helped achieving that;<\/li>\n<li>I wanted something that I could easily extend.<\/li>\n<\/ul>\n<p>As I've learned while preparing for the <em>AWS Certified Solutions Architect<\/em> exam and realized even more while doing this small project, there are many ways to implement event-based solutions on AWS.<\/p>\n<p>However, in my case I didn't need just a message broker:<\/p>\n<ol>\n<li>I needed the class search task to be triggered on a regular schedule;<\/li>\n<li>I needed a way to dynamically schedule one-off runs for classes that should have been booked at a future date compared to the one when running the class search.<\/li>\n<\/ol>\n<p>As I've learned, both requirements are greatly fulfilled by <em>EventBridge<\/em> *<\/p>\n<p>Finally, with the help of an extra <em>rule<\/em> there (and 0 code!) I could connect a <em>Simple Notification Service (SNS)<\/em> topic and deliver an SMS to my mobile number, with a customized message and the information of the class just booked.<\/p>\n<p>* there's <em>just<\/em> one caveat: the scheduler on <em>EventBridge<\/em> has a 60 seconds precision! This means that if you're scheduling something at 3:15:00PM there's a chance (in the worst case) that it gets fired at 3:15:59PM. In my case, I decided that this was an acceptable caveat.<\/p>\n<h2 id=\"configuration-and-secrets-management\">Configuration and secrets management<\/h2>\n<p>Both the <em>scan<\/em> and <em>book<\/em> functions needed configurations - some of which sensitive.<\/p>\n<p>I decided to go with <em>Systems manager<\/em> for non-sensitive and generic values. The code would have looked them up on AWS with a known key (eg. <code>facilityId<\/code>).<\/p>\n<p>For sensitive configurations I've instead used <em>Secrets manager<\/em>. The key difference with regular configuration was that they were user-specific (they included personal credentials to login on the Gym's API) and I wanted to potentially easily reuse my solution with different logins. So I've defined a user-specific secret on <em>Secrets manager<\/em> identified by a user alias (eg. <code>andrea<\/code>) that would have been referenced by the code hosted on the <em>Lambda<\/em> functions.<\/p>\n<h2 id=\"events\">Events<\/h2>\n<p>After some thinking I decided to define 3 events:<\/p>\n<ul>\n<li><code>ClassBookingAvailable<\/code><\/li>\n<li><code>ClassBookingCompleted<\/code><\/li>\n<li><code>ClassBookingFailed<\/code><\/li>\n<\/ul>\n<p>The last two (completed and failed) could have been merged into a single event with some extra fields representing the success\/failure outcome and the error if any. However, doing so would have forced me to put some undesired, extra complexity on the EventBridge rule that is responsible for delivering the SMS.<\/p>\n<p>Each of these events share a common base structure: they come with an <code>userAlias<\/code> string field, representing the user to which the event is connected, and a <code>class<\/code> object containing details about the class.<\/p>\n<p>Note that the above mentioned events are just defined within the code (Eg. the <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/gym-booking-assistant\/blob\/main\/scan\/index.js#L238\">ClassBookingAvailable event<\/a>) and on default event bus on <em>EventBridge<\/em>. I could have followed a more structured approach, and used both a custom <a rel=\"external\" href=\"https:\/\/docs.aws.amazon.com\/eventbridge\/latest\/userguide\/eb-schema-registry.html\">Schema registry<\/a> and a custom bus. However, both looked to me a bit overhead for the purpose.<\/p>\n<p>Let's see more in details what these events represent and their fields.<\/p>\n<h3 id=\"classbookingavailable\">ClassBookingAvailable<\/h3>\n<p>This event is published whenever a class matching the configured search criteria is ready to be booked, with the following structure<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"json\"><span class=\"giallo-l\"><span>{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;userAlias&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;andrea&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;class&quot;<\/span><span>:{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;id&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;84ddef10-45df-42d7-b45e-fac603fe01c7&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;name&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;Cycle Spirit&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;partitionDate&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;20240727&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;startDate&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;2024-07-27T19:00:00&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;bookingInfo&quot;<\/span><span>:{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;cancellationMinutesInAdvance&quot;<\/span><span>:<\/span><span style=\"color: #79B8FF;\">120<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;bookingUserStatus&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;CanBook&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>When scanning for classes on the Gym's API, a class can be found in different states: if a class is found in the <code>CanBook<\/code> state, it means that the booking for that class can happen already, if the class has a <code>WaitingBookingOpensPremium<\/code> state it means that the class cannot be booked before a<code>bookingInfo.bookingOpensOn<\/code> datetime, available on the class item on the Gym's API response.<\/p>\n<p>For a class in the <code>CanBook<\/code> state on the Gym's API, the <em>Scan<\/em> function immediately pushes a <code>ClassBookingAvailable<\/code> event on the EventBridge's default bus.\nIf a class is in the <code>WaitingBookingOpensPremium<\/code> state, the Scan function <em>schedules<\/em> a <code>ClassBookingAvailable<\/code> event, with the help of a dynamic rule on EventBridge that\nwill fire at a datetime of value <code>bookingInfo.bookingOpensOn<\/code>. We'll see more about these differences when looking at the code later.<\/p>\n<p>Note that both immediate and future <em>class available<\/em> events will trigger the booking step in the same way. The <em>Book<\/em> function basically does not care whether an event was received straight after being found on the Gym's API or after some time, on a predefined schedule.<\/p>\n<p>A few additional things worth highlighting from the above JSON sample:<\/p>\n<ul>\n<li>the <code>partitionDate<\/code> gives an indication of the day when the class will happen and <strong>complements<\/strong> the <code>id<\/code> field. The <code>id<\/code> field on its own is not enough to uniquely identify an <em>instance<\/em> of class: a class' <code>id<\/code> represent a particular type of class happening always at the same time on the same day of the week. However, the <code>partitionDate<\/code> is required to discriminated amongst different instances of the same type of class;<\/li>\n<li>the <code>startDate<\/code> helps in 2 ways at booking time: along with the <code>cancellationMinutesInAdvance<\/code> included in the <code>bookingInfo<\/code> object, It helps avoiding to book classes that can't be un-booked. This is particularly useful to avoid penalties. The booking code - as visible later - skips booking the classes that are happening in less than a configured time. Moreover, it has a cosmetic purpose, as it used in the SMS payload that is sent to notify about the completed booking.<\/li>\n<\/ul>\n<h3 id=\"classbookingcompleted\">ClassBookingCompleted<\/h3>\n<p>This event is published when a class has been booked successfully. It comes with the following shape:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"json\"><span class=\"giallo-l\"><span>{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;userAlias&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;andrea&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;class&quot;<\/span><span>:{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;id&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;95788cd4-c15a-4fe3-a6da-95b7764b4cba&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;name&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;Cycle Spirit&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;startDate&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;2024-07-27T19:00:00&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>The <code>userAlias<\/code> field here helps routing the notification to the expected user mobile number, if any is configured! The other information are only used to build a human friendly SMS payload.<\/p>\n<h3 id=\"classbookingfailed\">ClassBookingFailed<\/h3>\n<p>This event is published whenever there's a failure while booking a class:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"json\"><span class=\"giallo-l\"><span>{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;userAlias&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;andrea&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;class&quot;<\/span><span>:{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;id&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;95788cd4-c15a-4fe3-a6da-95b7764b4cba&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;name&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;Cycle Spirit&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">    &quot;startDate&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;2024-07-27T19:00:00&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>  },<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;errors&quot;<\/span><span>:[<\/span><\/span>\n<span class=\"giallo-l\"><span>    {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;field&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;BookingApiException.TooEarlyToBookParticipantException&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;type&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;Validation&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;details&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;errorMessage&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;The class is not open for booking yet&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">      &quot;message&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\">&quot;The class is not open for booking yet&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span>  ]<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>The <code>errors<\/code> field is an array of error objects, that maps 1:1 with what is returned by the Gym's API.<\/p>\n<p>At the time of writing this event just helps monitoring, it is not connected to any SMS notification.<\/p>\n<h1 id=\"architecture-overview\">Architecture overview<\/h1>\n<p>Overall, at a high level the solution I had in mind was as follows:<\/p>\n<ol>\n<li>Every 6 hours an EventBridge rule would have triggered the <code>Scan<\/code> <em>Lambda<\/em> function, that would have searched for available classes and either immediately publish a <code>ClassBookingAvailable<\/code> event or schedule one for the future;<\/li>\n<li>Every time <code>ClassBookingAvailable<\/code> event would have been published on EventBridge, the <code>Book<\/code> <em>Lambda<\/em> function would have been invoked and it would have eventually produced a <code>ClassBookingCompleted<\/code> or <code>ClassBookingFailed<\/code> event. In the first case, an SMS would have been delivered as well to a configured mobile number.<\/li>\n<\/ol>\n<p>Here's a high-level diagram of the architecture:<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-07-28_automating-my-gym-bookings-with-a-serverless-assistant\/architecture.png\" alt=\"Architecture overview\" \/>\n<em>Assistant's architecture high level overview<\/em><\/p>\n<h1 id=\"show-me-the-code\">Show me the code!<\/h1>\n<p>The whole source code is available on <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/gym-booking-assistant\">my Github<\/a>.\nHere I'll just focus on the main parts.<\/p>\n<h2 id=\"scan-function\">Scan function<\/h2>\n<p>Let's start by having a look at the <em>Scan<\/em> function code:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  1<\/span><span style=\"color: #79B8FF;\">exports<\/span><span>.<\/span><span style=\"color: #B392F0;\">handler<\/span><span style=\"color: #F97583;\"> = async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">event<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  2<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> userAlias<\/span><span style=\"color: #F97583;\"> =<\/span><span> event.detail.userAlias;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  3<\/span><span style=\"color: #F97583;\">  if<\/span><span> (<\/span><span style=\"color: #F97583;\">!<\/span><span>userAlias) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  4<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> errorMsg<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #9ECBFF;\"> &quot;Received even without userAlias. Aborting&quot;<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  5<\/span><span style=\"color: #F97583;\">    await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  6<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  7<\/span><span style=\"color: #F97583;\">    throw new<\/span><span style=\"color: #B392F0;\"> Error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  8<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">  9<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 10<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> userCredentials<\/span><span style=\"color: #F97583;\"> = await<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">getUserCredentials<\/span><span>(userAlias);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 11<\/span><span style=\"color: #F97583;\">  let<\/span><span> loginData<\/span><span style=\"color: #F97583;\"> = await<\/span><span> gymApiClient.<\/span><span style=\"color: #B392F0;\">login<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 12<\/span><span>    userCredentials.loginUsername,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 13<\/span><span>    userCredentials.loginPassword,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 14<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 15<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 16<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> facilityId<\/span><span style=\"color: #F97583;\"> = await<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">getConfig<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;facilityId&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 17<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 18<\/span><span style=\"color: #6A737D;\">  \/\/ Search all classes that match my criteria of interest<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 19<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> searchClassesRequest<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 20<\/span><span>    method:<\/span><span style=\"color: #9ECBFF;\"> &quot;GET&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 21<\/span><span>    url:<\/span><span style=\"color: #9ECBFF;\"> `${<\/span><span style=\"color: #79B8FF;\">CALENDAR_API_BASE_URI<\/span><span style=\"color: #9ECBFF;\">}\/enduser\/class\/search`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 22<\/span><span>    headers: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 23<\/span><span>      Authorization:<\/span><span style=\"color: #9ECBFF;\"> `Bearer ${<\/span><span>loginData<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>token<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 24<\/span><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 25<\/span><span>    params: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 26<\/span><span>      facilityId: facilityId,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 27<\/span><span>      fromDate: utils.<\/span><span style=\"color: #B392F0;\">nowCET<\/span><span>().<\/span><span style=\"color: #B392F0;\">format<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;yyyyMMDD&quot;<\/span><span>),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 28<\/span><span>      eventType:<\/span><span style=\"color: #9ECBFF;\"> &quot;Class&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 29<\/span><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 30<\/span><span>  };<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 31<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> searchClassesResponse<\/span><span style=\"color: #F97583;\"> = await<\/span><span> gymApiClient<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 32<\/span><span>    .<\/span><span style=\"color: #B392F0;\">getHttpClient<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 33<\/span><span>    .<\/span><span style=\"color: #B392F0;\">request<\/span><span>(searchClassesRequest);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 34<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 35<\/span><span style=\"color: #F97583;\">  if<\/span><span> (gymApiClient.<\/span><span style=\"color: #B392F0;\">isResponseError<\/span><span>(searchClassesResponse)) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 36<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> errorMsg<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #9ECBFF;\"> `Unable to get classes: ${<\/span><span style=\"color: #79B8FF;\">JSON<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span style=\"color: #9ECBFF;\">(<\/span><span>searchClassesResponse<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>data<\/span><span style=\"color: #9ECBFF;\">)}. Aborting`<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 37<\/span><span>    logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 38<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 39<\/span><span style=\"color: #F97583;\">    throw new<\/span><span style=\"color: #B392F0;\"> Error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 40<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 41<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 42<\/span><span style=\"color: #6A737D;\">  \/\/ It seems not possible to filter classes of interest via an API call. So we need to fetch them first<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 43<\/span><span style=\"color: #6A737D;\">  \/\/ and retrospectively ignore some of those.<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 44<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> filteredEvents<\/span><span style=\"color: #F97583;\"> =<\/span><span> searchClassesResponse.data<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 45<\/span><span>    .<\/span><span style=\"color: #B392F0;\">filter<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 46<\/span><span>      (<\/span><span style=\"color: #FFAB70;\">e<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 47<\/span><span style=\"color: #6A737D;\">        \/\/ excludes the classes booked already<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 48<\/span><span>        e.isParticipant<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #79B8FF;\"> true<\/span><span style=\"color: #F97583;\"> &amp;&amp;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 49<\/span><span style=\"color: #6A737D;\">        \/\/ excludes the classes that can&#39;t be booked for some reason<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 50<\/span><span>        e.bookingInfo.bookingUserStatus<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #9ECBFF;\"> &quot;CannotBook&quot;<\/span><span style=\"color: #F97583;\"> &amp;&amp;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 51<\/span><span>        e.bookingInfo.bookingUserStatus<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #9ECBFF;\"> &quot;BookingClosed&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 52<\/span><span>    )<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 53<\/span><span>    .<\/span><span style=\"color: #B392F0;\">filter<\/span><span>((<\/span><span style=\"color: #FFAB70;\">e<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 54<\/span><span style=\"color: #79B8FF;\">      SEARCH_CRITERIA<\/span><span>.classNames.<\/span><span style=\"color: #B392F0;\">some<\/span><span>((<\/span><span style=\"color: #FFAB70;\">c<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 55<\/span><span>        e.name.<\/span><span style=\"color: #B392F0;\">toLowerCase<\/span><span>().<\/span><span style=\"color: #B392F0;\">includes<\/span><span>(c.<\/span><span style=\"color: #B392F0;\">toLowerCase<\/span><span>()),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 56<\/span><span>      ),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 57<\/span><span>    )<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 58<\/span><span>    .<\/span><span style=\"color: #B392F0;\">filter<\/span><span>((<\/span><span style=\"color: #FFAB70;\">e<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 59<\/span><span style=\"color: #6A737D;\">      \/\/ Class should be taken in one of the days of interest<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 60<\/span><span style=\"color: #79B8FF;\">      SEARCH_CRITERIA<\/span><span>.days.<\/span><span style=\"color: #B392F0;\">includes<\/span><span>(utils.<\/span><span style=\"color: #B392F0;\">stringToDateCET<\/span><span>(e.startDate).<\/span><span style=\"color: #B392F0;\">day<\/span><span>()),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 61<\/span><span>    )<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 62<\/span><span>    .<\/span><span style=\"color: #B392F0;\">filter<\/span><span>((<\/span><span style=\"color: #FFAB70;\">e<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 63<\/span><span style=\"color: #6A737D;\">      \/\/ startDate time should fall in one of the hour ranges<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 64<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> timeFormat<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #9ECBFF;\"> &quot;HH:mm:ss&quot;<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 65<\/span><span style=\"color: #6A737D;\">      \/\/ this parses the class start timestamp in $timeFormat<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 66<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> classStartDateTime<\/span><span style=\"color: #F97583;\"> =<\/span><span> utils<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 67<\/span><span>        .<\/span><span style=\"color: #B392F0;\">stringToDateCET<\/span><span>(e.startDate)<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 68<\/span><span>        .<\/span><span style=\"color: #B392F0;\">format<\/span><span>(timeFormat);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 69<\/span><span style=\"color: #6A737D;\">      \/\/ this one builds a new date attaching the above time portion to today&#39;s date portion<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 70<\/span><span style=\"color: #6A737D;\">      \/\/ this makes sure that when comparing timestamps results are not spoiled by different days<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 71<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> adjustedClassStartDate<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> moment<\/span><span>(classStartDateTime, timeFormat);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 72<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 73<\/span><span style=\"color: #6A737D;\">      \/\/ this returns true if the class&#39; startDateTime if there&#39;s at least a match<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 74<\/span><span style=\"color: #F97583;\">      return<\/span><span style=\"color: #79B8FF;\"> SEARCH_CRITERIA<\/span><span>.hourRangesCET.<\/span><span style=\"color: #B392F0;\">some<\/span><span>((<\/span><span style=\"color: #FFAB70;\">hr<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 75<\/span><span style=\"color: #F97583;\">        const<\/span><span style=\"color: #79B8FF;\"> rangeStartTime<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> moment<\/span><span>(hr.start, timeFormat);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 76<\/span><span style=\"color: #F97583;\">        const<\/span><span style=\"color: #79B8FF;\"> rangeEndTime<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> moment<\/span><span>(hr.end, timeFormat);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 77<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 78<\/span><span style=\"color: #F97583;\">        return<\/span><span> adjustedClassStartDate.<\/span><span style=\"color: #B392F0;\">isBetween<\/span><span>(rangeStartTime, rangeEndTime);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 79<\/span><span>      });<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 80<\/span><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 81<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 82<\/span><span>  logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 83<\/span><span style=\"color: #9ECBFF;\">    `Found ${<\/span><span>filteredEvents<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #79B8FF;\">length<\/span><span style=\"color: #9ECBFF;\">} events of the categories of interest.`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 84<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 85<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 86<\/span><span style=\"color: #F97583;\">  for<\/span><span> (<\/span><span style=\"color: #F97583;\">const<\/span><span style=\"color: #79B8FF;\"> e<\/span><span style=\"color: #F97583;\"> of<\/span><span> filteredEvents) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 87<\/span><span style=\"color: #F97583;\">    switch<\/span><span> (e.bookingInfo.bookingUserStatus) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 88<\/span><span style=\"color: #F97583;\">      case<\/span><span style=\"color: #9ECBFF;\"> &quot;CanBook&quot;<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 89<\/span><span>        logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 90<\/span><span style=\"color: #9ECBFF;\">          `Booking for class ${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>name<\/span><span style=\"color: #9ECBFF;\">} with id=${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">} should happen immediately.`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 91<\/span><span>        );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 92<\/span><span style=\"color: #F97583;\">        await<\/span><span style=\"color: #B392F0;\"> publishBookingAvailableEvent<\/span><span>(userAlias, e);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 93<\/span><span style=\"color: #F97583;\">        break<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 94<\/span><span style=\"color: #F97583;\">      case<\/span><span style=\"color: #9ECBFF;\"> &quot;WaitingBookingOpensPremium&quot;<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 95<\/span><span>        logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 96<\/span><span style=\"color: #9ECBFF;\">          `Booking for class ${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>name<\/span><span style=\"color: #9ECBFF;\">} with id=${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">} should be scheduled on ${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>bookingInfo<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>bookingOpensOn<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 97<\/span><span>        );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 98<\/span><span style=\"color: #F97583;\">        await<\/span><span style=\"color: #B392F0;\"> scheduleFutureBooking<\/span><span>(userAlias, e);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 99<\/span><span style=\"color: #F97583;\">        break<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">100<\/span><span style=\"color: #F97583;\">      default<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">101<\/span><span>        logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">102<\/span><span style=\"color: #9ECBFF;\">          `Unexpected status for class ${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>name<\/span><span style=\"color: #9ECBFF;\">} with id ${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">}: ${<\/span><span>e<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>bookingInfo<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>bookingUserStatus<\/span><span style=\"color: #9ECBFF;\">}. Skipping.`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">103<\/span><span>        );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">104<\/span><span style=\"color: #F97583;\">        return<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">105<\/span><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">106<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">107<\/span><span>};<\/span><\/span><\/code><\/pre>\n<p>The first lines (<code>2-8<\/code>) are simply for validation. If the <code>userAlias<\/code> is not part of the event an error is thrown.<\/p>\n<p>Lines <code>10-40<\/code> include the code required to create and fire the search classes requests. There are a few references to the <code>utils<\/code> and <code>gymApiClient<\/code> modules that are hosted within the common layer, which I'll talk about more later in the <a href=\"https:\/\/adilisio.com\/posts\/my-serverless-gym-booking-assistant\/#common-layer\">dedicated chapter<\/a>.<\/p>\n<p>Unfortunately the Gym's API does not offer great filtering features. Hence, we have to retrospectively exclude events that are out of interest from the Gym's API response (lines <code>42-80<\/code>). The <code>SEARCH_CRITERIA<\/code> referenced in the code is a constant defined on the same file, of the following shape:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">const<\/span><span style=\"color: #79B8FF;\"> SEARCH_CRITERIA<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>  classNames: [<\/span><span style=\"color: #9ECBFF;\">&quot;Pilates&quot;<\/span><span>],<\/span><\/span>\n<span class=\"giallo-l\"><span>  hourRangesCET: [<\/span><\/span>\n<span class=\"giallo-l\"><span>    {<\/span><\/span>\n<span class=\"giallo-l\"><span>      start:<\/span><span style=\"color: #9ECBFF;\"> &quot;08:00:00&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>      end:<\/span><span style=\"color: #9ECBFF;\"> &quot;10:00:30&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span>    {<\/span><\/span>\n<span class=\"giallo-l\"><span>      start:<\/span><span style=\"color: #9ECBFF;\"> &quot;18:00:00&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>      end:<\/span><span style=\"color: #9ECBFF;\"> &quot;21:00:00&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span>  ],<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  \/\/ 0 Sunday, 6 Saturday<\/span><\/span>\n<span class=\"giallo-l\"><span>  days: [<\/span><span style=\"color: #79B8FF;\">1<\/span><span>,<\/span><span style=\"color: #79B8FF;\"> 2<\/span><span>,<\/span><span style=\"color: #79B8FF;\"> 3<\/span><span>,<\/span><span style=\"color: #79B8FF;\"> 4<\/span><span>,<\/span><span style=\"color: #79B8FF;\"> 5<\/span><span>],<\/span><\/span>\n<span class=\"giallo-l\"><span>};<\/span><\/span><\/code><\/pre>\n<p>Thanks to this constant, the code narrows the focus only on particular type of classes happening on specific days and during predefined time windows.<\/p>\n<p>After some logging (lines <code>82-84<\/code>), the code finally takes care of what should happen with the classes that are of interest. It loops through each of those items, and:<\/p>\n<ul>\n<li>invokes a <code>publishBookingAvailableEvent<\/code> function if the class has a <code>CanBook<\/code> state;<\/li>\n<li>calls a <code>scheduleFutureBooking<\/code> function if the class' state is <code>WaitingBookingOpensPremium<\/code>;<\/li>\n<li>Otherwise, does nothing but log an error and return.<\/li>\n<\/ul>\n<p>The <code>publishBookingAvailableEvent<\/code> function takes the <code>userAlias<\/code> and the event details as parameters, and immediately publishes a <code>ClassBookingAvailable<\/code> on the default <em>EventBridge<\/em> event bus:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 1<\/span><span style=\"color: #F97583;\">async function<\/span><span style=\"color: #B392F0;\"> publishBookingAvailableEvent<\/span><span>(<\/span><span style=\"color: #FFAB70;\">userAlias<\/span><span>,<\/span><span style=\"color: #FFAB70;\"> classDetails<\/span><span>) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 2<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> classBookingAvailableEvent<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 3<\/span><span>    Entries: [<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 4<\/span><span>      {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 5<\/span><span>        Time:<\/span><span style=\"color: #F97583;\"> new<\/span><span style=\"color: #B392F0;\"> Date<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 6<\/span><span>        Source:<\/span><span style=\"color: #9ECBFF;\"> &quot;GymBookingAssistant.scan&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 7<\/span><span>        DetailType:<\/span><span style=\"color: #9ECBFF;\"> &quot;ClassBookingAvailable&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 8<\/span><span>        Detail:<\/span><span style=\"color: #79B8FF;\"> JSON<\/span><span>.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 9<\/span><span style=\"color: #B392F0;\">          craftClassBookingAvailableEventPayload<\/span><span>(userAlias, classDetails),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">10<\/span><span>        ),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">11<\/span><span>      },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">12<\/span><span>    ],<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">13<\/span><span>  };<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">14<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">15<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> putEventResponse<\/span><span style=\"color: #F97583;\"> = await<\/span><span> eventBridgeClient.<\/span><span style=\"color: #B392F0;\">send<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">16<\/span><span style=\"color: #F97583;\">    new<\/span><span style=\"color: #B392F0;\"> PutEventsCommand<\/span><span>(classBookingAvailableEvent),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">17<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">18<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">19<\/span><span style=\"color: #F97583;\">  if<\/span><span> (<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">20<\/span><span>    putEventResponse[<\/span><span style=\"color: #9ECBFF;\">&quot;$metadata&quot;<\/span><span>].httpStatusCode<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #79B8FF;\"> 200<\/span><span style=\"color: #F97583;\"> ||<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">21<\/span><span>    putEventResponse.FailedEntryCount<\/span><span style=\"color: #F97583;\"> &gt;<\/span><span style=\"color: #79B8FF;\"> 0<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">22<\/span><span>  ) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">23<\/span><span>    logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">24<\/span><span style=\"color: #9ECBFF;\">      &quot;There were one or more errors while publishing a ClassBookingAvailable event.&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">25<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">26<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">27<\/span><span>}<\/span><\/span><\/code><\/pre>\n<p>Whereas, the <code>scheduleFutureBooking<\/code> creates a schedule to publish a ClassBookingAvailable event in future, precisely at <code>bookingInfo.bookingOpensOn<\/code>:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 1<\/span><span style=\"color: #F97583;\">async function<\/span><span style=\"color: #B392F0;\"> scheduleFutureBooking<\/span><span>(<\/span><span style=\"color: #FFAB70;\">userAlias<\/span><span>,<\/span><span style=\"color: #FFAB70;\"> classDetails<\/span><span>) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 2<\/span><span style=\"color: #F97583;\">  let<\/span><span> bookingOpensOnUTC<\/span><span style=\"color: #F97583;\"> = new<\/span><span style=\"color: #B392F0;\"> Date<\/span><span>(classDetails.bookingInfo.bookingOpensOn)<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 3<\/span><span>    .<\/span><span style=\"color: #B392F0;\">toISOString<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 4<\/span><span>    .<\/span><span style=\"color: #B392F0;\">slice<\/span><span>(<\/span><span style=\"color: #79B8FF;\">0<\/span><span>,<\/span><span style=\"color: #F97583;\"> -<\/span><span style=\"color: #79B8FF;\">5<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 5<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 6<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> scheduleRequest<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 7<\/span><span>    Name:<\/span><span style=\"color: #9ECBFF;\"> `ScheduleBooking_${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 8<\/span><span>    Description:<\/span><span style=\"color: #9ECBFF;\"> `Class: ${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>name<\/span><span style=\"color: #9ECBFF;\">} - Starts at: ${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>startDate<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 9<\/span><span>    ScheduleExpression:<\/span><span style=\"color: #9ECBFF;\"> `at(${<\/span><span>bookingOpensOnUTC<\/span><span style=\"color: #9ECBFF;\">})`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">10<\/span><span>    Target: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">11<\/span><span>      Arn:<\/span><span style=\"color: #9ECBFF;\"> &quot;arn:aws:events:eu-south-1:097176176455:event-bus\/default&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">12<\/span><span>      RoleArn:<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">13<\/span><span style=\"color: #9ECBFF;\">        &quot;arn:aws:iam::097176176455:role\/service-role\/GymBookingAssistantEventBridgeRole&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">14<\/span><span>      EventBridgeParameters: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">15<\/span><span>        DetailType:<\/span><span style=\"color: #9ECBFF;\"> &quot;ClassBookingAvailable&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">16<\/span><span>        Source:<\/span><span style=\"color: #9ECBFF;\"> &quot;GymBookingAssistant.scan&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">17<\/span><span>      },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">18<\/span><span style=\"color: #6A737D;\">      \/\/ The schedule event is one that contains the userAlias and a slimmed-down<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">19<\/span><span style=\"color: #6A737D;\">      \/\/ version of the class object coming from the Gym API<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">20<\/span><span>      Input:<\/span><span style=\"color: #79B8FF;\"> JSON<\/span><span>.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">21<\/span><span style=\"color: #B392F0;\">        craftClassBookingAvailableEventPayload<\/span><span>(userAlias, classDetails),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">22<\/span><span>      ),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">23<\/span><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">24<\/span><span>    ActionAfterCompletion: ActionAfterCompletion.<\/span><span style=\"color: #79B8FF;\">DELETE<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">25<\/span><span>    FlexibleTimeWindow: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">26<\/span><span>      Mode: FlexibleTimeWindowMode.<\/span><span style=\"color: #79B8FF;\">OFF<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">27<\/span><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">28<\/span><span>  };<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">29<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">30<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> createScheduleResponse<\/span><span style=\"color: #F97583;\"> = await<\/span><span> schedulerClient.<\/span><span style=\"color: #B392F0;\">send<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">31<\/span><span style=\"color: #F97583;\">    new<\/span><span style=\"color: #B392F0;\"> CreateScheduleCommand<\/span><span>(scheduleRequest),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">32<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">33<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">34<\/span><span style=\"color: #F97583;\">  if<\/span><span> (createScheduleResponse[<\/span><span style=\"color: #9ECBFF;\">&quot;$metadata&quot;<\/span><span>].httpStatusCode<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #79B8FF;\"> 200<\/span><span>) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">35<\/span><span>    logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">36<\/span><span style=\"color: #9ECBFF;\">      &quot;There were one or more errors while creating a booking schedule.&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">37<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">38<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">39<\/span><span>}<\/span><\/span><\/code><\/pre>\n<p>As <em>EventBridge<\/em> (by default) requires schedules to be declared in UTC, some minimal massaging is required on the original datetime as it's returned in CET timezone by the Gym's API (lines <code>2-4<\/code>). The schedule has a fairly strict requirement for the execution time, therefore it does not allow a flexible time window (lines <code>25-27<\/code>). Moreover, it self deletes from EventBridge (line <code>24<\/code>) once executed.<\/p>\n<p>Figuring out what values to use for the <code>Arn<\/code> and <code>RoleArn<\/code> included in the <code>Target<\/code> object (lines <code>11-12<\/code>) was not straightforward, as the <a rel=\"external\" href=\"https:\/\/docs.aws.amazon.com\/scheduler\/latest\/APIReference\/API_Target.html\">official docs<\/a> are only suggesting what values could be used. Luckily, <a rel=\"external\" href=\"https:\/\/stackoverflow.com\/questions\/76091158\/error-the-execution-role-you-provide-must-allow-aws-eventbridge-scheduler-to-as\">others had stumbled across a similar problem already<\/a>.<\/p>\n<p>Both the <code>publishBookingAvailableEvent<\/code> and <code>scheduleFutureBooking<\/code> leverage the same internal <code>craftClassBookingAvailableEventPayload<\/code> utility to build the event payload to be published or scheduled:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">function<\/span><span style=\"color: #B392F0;\"> craftClassBookingAvailableEventPayload<\/span><span>(<\/span><span style=\"color: #FFAB70;\">userAlias<\/span><span>,<\/span><span style=\"color: #FFAB70;\"> classDetails<\/span><span>) {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">  return<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>    userAlias: userAlias,<\/span><\/span>\n<span class=\"giallo-l\"><span>    class: {<\/span><\/span>\n<span class=\"giallo-l\"><span>      id: classDetails.id,<\/span><\/span>\n<span class=\"giallo-l\"><span>      name: classDetails.name,<\/span><\/span>\n<span class=\"giallo-l\"><span>      partitionDate: classDetails.partitionDate,<\/span><\/span>\n<span class=\"giallo-l\"><span>      startDate: classDetails.startDate,<\/span><\/span>\n<span class=\"giallo-l\"><span>      bookingInfo: {<\/span><\/span>\n<span class=\"giallo-l\"><span>        cancellationMinutesInAdvance:<\/span><\/span>\n<span class=\"giallo-l\"><span>          classDetails.bookingInfo.cancellationMinutesInAdvance,<\/span><\/span>\n<span class=\"giallo-l\"><span>        bookingUserStatus: classDetails.bookingInfo.bookingUserStatus,<\/span><\/span>\n<span class=\"giallo-l\"><span>      },<\/span><\/span>\n<span class=\"giallo-l\"><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span>  };<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>That's all for the <em>Scan<\/em> function. Let's now focus on the code required to book a class.<\/p>\n<h2 id=\"book-function\">Book function<\/h2>\n<p>The <em>Book<\/em> function is invoked with events of type <code>ClassBookingAvailable<\/code> events, without differences between immediate and scheduled events:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 1<\/span><span style=\"color: #79B8FF;\">exports<\/span><span>.<\/span><span style=\"color: #B392F0;\">handler<\/span><span style=\"color: #F97583;\"> = async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">event<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 2<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> userAlias<\/span><span style=\"color: #F97583;\"> =<\/span><span> event.detail.userAlias;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 3<\/span><span style=\"color: #F97583;\">  if<\/span><span> (<\/span><span style=\"color: #F97583;\">!<\/span><span>userAlias) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 4<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> errorMsg<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #9ECBFF;\"> &quot;Received even without userAlias. Aborting&quot;<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 5<\/span><span style=\"color: #F97583;\">    await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 6<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 7<\/span><span style=\"color: #F97583;\">    throw new<\/span><span style=\"color: #B392F0;\"> Error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 8<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 9<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">10<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> classDetails<\/span><span style=\"color: #F97583;\"> =<\/span><span> event.detail.class;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">11<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">12<\/span><span style=\"color: #F97583;\">  await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">13<\/span><span style=\"color: #9ECBFF;\">    `Received event of type=${<\/span><span>event<\/span><span style=\"color: #9ECBFF;\">[&quot;detail-type&quot;]} from source=${<\/span><span>event<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>source<\/span><span style=\"color: #9ECBFF;\">} with id=${<\/span><span>event<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">}.<\/span><span style=\"color: #79B8FF;\">\\n<\/span><span style=\"color: #9ECBFF;\">Trying to book class with id=${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">} and partitionDate=${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>partitionDate<\/span><span style=\"color: #9ECBFF;\">} for userAlias=${<\/span><span>userAlias<\/span><span style=\"color: #9ECBFF;\">} ...`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">14<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">15<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">16<\/span><span style=\"color: #6A737D;\">  \/\/ Check class booking status. This should never be different from CanBook or WaitingBookingOpensPremium, but let&#39;s double check<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">17<\/span><span style=\"color: #F97583;\">  if<\/span><span> (<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">18<\/span><span>    classDetails.bookingInfo.bookingUserStatus<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #9ECBFF;\"> &quot;CanBook&quot;<\/span><span style=\"color: #F97583;\"> &amp;&amp;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">19<\/span><span>    classDetails.bookingInfo.bookingUserStatus<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #9ECBFF;\"> &quot;WaitingBookingOpensPremium&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">20<\/span><span>  ) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">21<\/span><span style=\"color: #F97583;\">    await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">warn<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">22<\/span><span style=\"color: #9ECBFF;\">      `Booking rejected its status is${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>bookingInfo<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>bookingUserStatus<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">23<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">24<\/span><span style=\"color: #F97583;\">    return<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">25<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">26<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">27<\/span><span style=\"color: #6A737D;\">  \/\/ Check cancellationMinutesInAdvance. We should avoid booking for classes than can&#39;t be un-booked to avoid penalties!<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">28<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> startDateCET<\/span><span style=\"color: #F97583;\"> =<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">stringToDateCET<\/span><span>(classDetails.startDate);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">29<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> timeToClassStartInMinutes<\/span><span style=\"color: #F97583;\"> =<\/span><span> startDateCET.<\/span><span style=\"color: #B392F0;\">diff<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">30<\/span><span>    utils.<\/span><span style=\"color: #B392F0;\">nowCET<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">31<\/span><span style=\"color: #9ECBFF;\">    &quot;minutes&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">32<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">33<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">34<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> timeToCancelBookingMinutes<\/span><span style=\"color: #F97583;\"> =<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">35<\/span><span>    classDetails.bookingInfo.cancellationMinutesInAdvance<\/span><span style=\"color: #F97583;\"> +<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">36<\/span><span style=\"color: #79B8FF;\">    EXTRA_TIME_CANCEL_BOOKING_IN_MINUTES<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">37<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> classCanBeCancelled<\/span><span style=\"color: #F97583;\"> =<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">38<\/span><span>    timeToClassStartInMinutes<\/span><span style=\"color: #F97583;\"> &gt;<\/span><span> timeToCancelBookingMinutes;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">39<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">40<\/span><span style=\"color: #F97583;\">  if<\/span><span> (<\/span><span style=\"color: #F97583;\">!<\/span><span>classCanBeCancelled) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">41<\/span><span style=\"color: #F97583;\">    await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">warn<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">42<\/span><span style=\"color: #9ECBFF;\">      `Booking rejected to avoid penalties, because class could not be un-booked. startDate=${<\/span><span>startDateCET<\/span><span style=\"color: #9ECBFF;\">} timeToClassStartInMinutes=${<\/span><span>timeToClassStartInMinutes<\/span><span style=\"color: #9ECBFF;\">} timeToCancelBookingMinutes=${<\/span><span>timeToCancelBookingMinutes<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">43<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">44<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">45<\/span><span style=\"color: #F97583;\">    return<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">46<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">47<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">48<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> userCredentials<\/span><span style=\"color: #F97583;\"> = await<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">getUserCredentials<\/span><span>(userAlias);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">49<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">50<\/span><span style=\"color: #F97583;\">  let<\/span><span> loginData<\/span><span style=\"color: #F97583;\"> = await<\/span><span> gymApiClient.<\/span><span style=\"color: #B392F0;\">login<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">51<\/span><span>    userCredentials.loginUsername,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">52<\/span><span>    userCredentials.loginPassword,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">53<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">54<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">55<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> bookClassRequest<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">56<\/span><span>    method:<\/span><span style=\"color: #9ECBFF;\"> &quot;POST&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">57<\/span><span>    url:<\/span><span style=\"color: #9ECBFF;\"> `${<\/span><span style=\"color: #79B8FF;\">BOOKING_API_BASE_URI<\/span><span style=\"color: #9ECBFF;\">}\/core\/calendarevent\/${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">}\/book`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">58<\/span><span>    headers: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">59<\/span><span>      Authorization:<\/span><span style=\"color: #9ECBFF;\"> `Bearer ${<\/span><span>loginData<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>token<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">60<\/span><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">61<\/span><span>    data: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">62<\/span><span>      partitionDate: classDetails.partitionDate,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">63<\/span><span>      userId: userCredentials.userId,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">64<\/span><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">65<\/span><span>  };<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">66<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">67<\/span><span style=\"color: #F97583;\">  const<\/span><span style=\"color: #79B8FF;\"> bookClassResponse<\/span><span style=\"color: #F97583;\"> = await<\/span><span> gymApiClient<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">68<\/span><span>    .<\/span><span style=\"color: #B392F0;\">getHttpClient<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">69<\/span><span>    .<\/span><span style=\"color: #B392F0;\">request<\/span><span>(bookClassRequest);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">70<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">71<\/span><span style=\"color: #F97583;\">  if<\/span><span> (gymApiClient.<\/span><span style=\"color: #B392F0;\">isResponseError<\/span><span>(bookClassResponse)) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">72<\/span><span>    logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">73<\/span><span style=\"color: #9ECBFF;\">      `Unable to book class with id=${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">} and partitionDate=${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>partitionDate<\/span><span style=\"color: #9ECBFF;\">}. Errors=${<\/span><span style=\"color: #79B8FF;\">JSON<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span style=\"color: #9ECBFF;\">(<\/span><span>bookClassResponse<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>data<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>errors<\/span><span style=\"color: #9ECBFF;\">)}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">74<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">75<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">76<\/span><span style=\"color: #F97583;\">    await<\/span><span style=\"color: #B392F0;\"> publishBookingFailedEvent<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">77<\/span><span>      userAlias,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">78<\/span><span>      classDetails,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">79<\/span><span>      bookClassResponse.data.errors,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">80<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">81<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">82<\/span><span style=\"color: #F97583;\">    return<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">83<\/span><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">84<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">85<\/span><span>  logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">86<\/span><span style=\"color: #9ECBFF;\">    `Successfully booked class with id=${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>id<\/span><span style=\"color: #9ECBFF;\">} and partitionDate=${<\/span><span>classDetails<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>partitionDate<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">87<\/span><span>  );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">88<\/span><span style=\"color: #F97583;\">  await<\/span><span style=\"color: #B392F0;\"> publishBookingCompletedEvent<\/span><span>(userAlias, classDetails);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">89<\/span><span>};<\/span><\/span><\/code><\/pre>\n<p>Lines <code>2-46<\/code> are meant to run some extended validation on the <code>ClassBookingAvailable<\/code> event received. After the usual userAlias check, the code makes sure that the class is in a valid state for booking (lines <code>16-25<\/code>), and that booking is avoided if there's no chance or a limited time window to un-bok it soon after (lines <code>27-46<\/code>). This is meant to mitigate the risk of penalties for the Gym's API user: if for 3 times in a row the user does not join a class that was previously booked, they are temporarily banned from booking new classes.<\/p>\n<p>From line <code>50<\/code> onwards the code takes care of logging the user in, booking the class specified on the received <code>ClassBookingAvailable<\/code> event, and ultimately publishing either a <code>ClassBookingCompleted<\/code> or <code>ClassBookingFailed<\/code> event.<\/p>\n<h3 id=\"what-could-go-wrong\">What could go wrong<\/h3>\n<p><em>EventBridge<\/em> guarantees <strong>at least once<\/strong> delivery. This means that  my functions could be invoked multiple times for the same event.<\/p>\n<p>While this is a problem for both the <em>Scan<\/em> and <em>Book<\/em> function, it's probably more relevant for the latter:\nWhat happens if the <em>Book<\/em> function gets invoked multiple times with the same <code>ClassBookingAvailable<\/code> event? Even if the booking operation is <a rel=\"external\" href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Glossary\/Idempotent\">idempotent<\/a> on the Gym's API (it seems that they have built some sort of idempotency mechanism based on the <code>id<\/code> and <code>partitionDate<\/code> of the booking), how to avoid that the same SMS gets sent multiple times?<\/p>\n<p>To mitigate these hiccups I should have made all my functions idempotent. Considering the overall context, I decided that this flaw was acceptable.<\/p>\n<p>Another problem is that EventBridge comes with 60 seconds precision, and sometimes the <em>Book<\/em> function could be invoked with a delay up to that value: this is usually not enough for the overcrowded classes that usually gets fully booked a bunch of seconds after the bookings are open. Again, since this was not the case of the class I was interested in, I decided to proceed.<\/p>\n<p>Let's now zoom on the common modules.<\/p>\n<h2 id=\"common-layer\">Common layer<\/h2>\n<p>The common layer includes 3 modules:<\/p>\n<ul>\n<li>one that includes all the utilities, called <code>utils<\/code>;<\/li>\n<li>one that bundles the HTTP client to interact with the Gym's API, named <code>gymApiClient<\/code>;<\/li>\n<li>and finally, a <code>logging<\/code> module for custom logging purposes.<\/li>\n<\/ul>\n<p>While the whole code is available at <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/gym-booking-assistant\/tree\/main\/common\/nodejs\">my github repository<\/a>, here I want to highlight a few important functions bundled in the above mentioned modules.<\/p>\n<p>The <code>utils<\/code> modules exports 2 key methods (among the others) that are used to fetch generic configurations from <em>Systems Manager<\/em> (<code>getConfig<\/code>), or sensitive and user-specific credentials from <em>Secrets Manager<\/em> (<code>getUserCredentials<\/code>):<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #79B8FF;\">module<\/span><span>.<\/span><span style=\"color: #79B8FF;\">exports<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">  getConfig<\/span><span>:<\/span><span style=\"color: #F97583;\"> async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">name<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    if<\/span><span> (<\/span><span style=\"color: #F97583;\">!<\/span><span>config) {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> parametersStoreResponse<\/span><span style=\"color: #F97583;\"> = await<\/span><span> serviceSystemManagerClient.<\/span><span style=\"color: #B392F0;\">send<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        new<\/span><span style=\"color: #B392F0;\"> GetParameterCommand<\/span><span>({<\/span><\/span>\n<span class=\"giallo-l\"><span>          Name:<\/span><span style=\"color: #9ECBFF;\"> &quot;GymBookingAssistant&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>        }),<\/span><\/span>\n<span class=\"giallo-l\"><span>      );<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>      config<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #79B8FF;\"> JSON<\/span><span>.<\/span><span style=\"color: #B392F0;\">parse<\/span><span>(parametersStoreResponse.Parameter.Value);<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> value<\/span><span style=\"color: #F97583;\"> =<\/span><span> config[name];<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    if<\/span><span> (<\/span><span style=\"color: #F97583;\">!<\/span><span>value) {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">      throw new<\/span><span style=\"color: #B392F0;\"> Error<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">`Config &quot;${<\/span><span>name<\/span><span style=\"color: #9ECBFF;\">}&quot; not found.`<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return<\/span><span> value;<\/span><\/span>\n<span class=\"giallo-l\"><span>  },<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">  getUserCredentials<\/span><span>:<\/span><span style=\"color: #F97583;\"> async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">userAlias<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> credentials<\/span><span style=\"color: #F97583;\"> = await<\/span><span> secretsManagerClient.<\/span><span style=\"color: #B392F0;\">send<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">      new<\/span><span style=\"color: #B392F0;\"> GetSecretValueCommand<\/span><span>({<\/span><\/span>\n<span class=\"giallo-l\"><span>        SecretId:<\/span><span style=\"color: #9ECBFF;\"> `GymBookingAssistant\/Credentials\/${<\/span><span>userAlias<\/span><span style=\"color: #9ECBFF;\">}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>      }),<\/span><\/span>\n<span class=\"giallo-l\"><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return<\/span><span style=\"color: #79B8FF;\"> JSON<\/span><span>.<\/span><span style=\"color: #B392F0;\">parse<\/span><span>(credentials.SecretString);<\/span><\/span>\n<span class=\"giallo-l\"><span>  },<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  \/\/ other functions...<\/span><\/span>\n<span class=\"giallo-l\"><span>};<\/span><\/span><\/code><\/pre>\n<p>The <code>gymApiClient<\/code> modules bundles all the logic that interacts with the Gym's API server and exports 3 methods:<\/p>\n<ul>\n<li>a <code>getHttpClient<\/code> method that yields an instance of the HTTP client (built on top of <a rel=\"external\" href=\"https:\/\/axios-http.com\/\">Axios<\/a>) to be used with the Gym's API, already enriched with a bunch of interceptors meant for request\/response logging or to set some defaults HTTP headers. The returned HTTP client instance can be used to fire multiple different API calls, as the search and book interactions that we've seen above.<\/li>\n<li>an <code>isResponseError<\/code> utility that takes an <a rel=\"external\" href=\"https:\/\/axios-http.com\/docs\/res_schema\">Axios HTTP response<\/a> as parameters and tells whether it contains an error or not;<\/li>\n<li>And a <code>login<\/code> method, that reuses the above <code>getHttpClient<\/code>, and fires a login call to the Gym's server.<\/li>\n<\/ul>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 1<\/span><span style=\"color: #79B8FF;\">module<\/span><span>.<\/span><span style=\"color: #79B8FF;\">exports<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 2<\/span><span style=\"color: #B392F0;\">  login<\/span><span>:<\/span><span style=\"color: #F97583;\"> async function<\/span><span> (<\/span><span style=\"color: #FFAB70;\">username<\/span><span>,<\/span><span style=\"color: #FFAB70;\"> password<\/span><span>) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 3<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> APPLICATION_ID<\/span><span style=\"color: #F97583;\"> = await<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">getConfig<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;applicationId&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 4<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> LOGIN_DOMAIN<\/span><span style=\"color: #F97583;\"> = await<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">getConfig<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;loginDomain&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 5<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 6<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> loginRequest<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 7<\/span><span>      method:<\/span><span style=\"color: #9ECBFF;\"> &quot;POST&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 8<\/span><span>      url:<\/span><span style=\"color: #9ECBFF;\"> `${<\/span><span style=\"color: #79B8FF;\">CORE_API_BASE_URI<\/span><span style=\"color: #9ECBFF;\">}\/Application\/${<\/span><span style=\"color: #79B8FF;\">APPLICATION_ID<\/span><span style=\"color: #9ECBFF;\">}\/Login`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\"> 9<\/span><span>      data: {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">10<\/span><span>        domain:<\/span><span style=\"color: #79B8FF;\"> LOGIN_DOMAIN<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">11<\/span><span>        keepMeLoggedIn:<\/span><span style=\"color: #79B8FF;\"> true<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">12<\/span><span>        username: username,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">13<\/span><span>        password: password,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">14<\/span><span>      },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">15<\/span><span>    };<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">16<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">17<\/span><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> loginResponse<\/span><span style=\"color: #F97583;\"> = await<\/span><span style=\"color: #79B8FF;\"> module<\/span><span>.<\/span><span style=\"color: #79B8FF;\">exports<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">18<\/span><span>      .<\/span><span style=\"color: #B392F0;\">getHttpClient<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">19<\/span><span>      .<\/span><span style=\"color: #B392F0;\">request<\/span><span>(loginRequest);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">20<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">21<\/span><span style=\"color: #F97583;\">    if<\/span><span> (<\/span><span style=\"color: #79B8FF;\">module<\/span><span>.<\/span><span style=\"color: #79B8FF;\">exports<\/span><span>.<\/span><span style=\"color: #B392F0;\">isResponseError<\/span><span>(loginResponse)) {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">22<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> errorMsg<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #9ECBFF;\"> `Unable to login: ${<\/span><span style=\"color: #79B8FF;\">JSON<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span style=\"color: #9ECBFF;\">(<\/span><span>loginResponse<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>data<\/span><span style=\"color: #9ECBFF;\">)}. Aborting`<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">23<\/span><span style=\"color: #F97583;\">      await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">24<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">25<\/span><span style=\"color: #F97583;\">      throw new<\/span><span style=\"color: #B392F0;\"> Error<\/span><span>(errorMsg);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">26<\/span><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">27<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">28<\/span><span style=\"color: #F97583;\">    return<\/span><span> loginResponse.data;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">29<\/span><span>  },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">30<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">31<\/span><span style=\"color: #B392F0;\">  getHttpClient<\/span><span>:<\/span><span style=\"color: #F97583;\"> function<\/span><span> () {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">32<\/span><span style=\"color: #F97583;\">    let<\/span><span> client<\/span><span style=\"color: #F97583;\"> =<\/span><span> axios.<\/span><span style=\"color: #B392F0;\">create<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">33<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">34<\/span><span>    client.interceptors.request.<\/span><span style=\"color: #B392F0;\">use<\/span><span>(<\/span><span style=\"color: #F97583;\">async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">request<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">35<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> CLIENT_ID<\/span><span style=\"color: #F97583;\"> = await<\/span><span> utils.<\/span><span style=\"color: #B392F0;\">getConfig<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;clientId&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">36<\/span><span>      request.headers[<\/span><span style=\"color: #9ECBFF;\">&quot;x-mwapps-client&quot;<\/span><span>]<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #79B8FF;\"> CLIENT_ID<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">37<\/span><span style=\"color: #F97583;\">      return<\/span><span> request;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">38<\/span><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">39<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">40<\/span><span>    client.interceptors.request.<\/span><span style=\"color: #B392F0;\">use<\/span><span>(<\/span><span style=\"color: #F97583;\">async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">request<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">41<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> maskedPayload<\/span><span style=\"color: #F97583;\"> =<\/span><span> maskData.<\/span><span style=\"color: #B392F0;\">maskJSON2<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">42<\/span><span>        request.data,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">43<\/span><span style=\"color: #79B8FF;\">        JSON_MASKING_CONFIG<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">44<\/span><span>      );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">45<\/span><span style=\"color: #F97583;\">      await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">46<\/span><span style=\"color: #9ECBFF;\">        `&gt;&gt;&gt; ${<\/span><span>request<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>method<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">toUpperCase<\/span><span style=\"color: #9ECBFF;\">()} ${<\/span><span>request<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>url<\/span><span style=\"color: #9ECBFF;\">}<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">47<\/span><span style=\"color: #79B8FF;\">        \\n<\/span><span style=\"color: #9ECBFF;\">Params: ${<\/span><span style=\"color: #79B8FF;\">JSON<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span style=\"color: #9ECBFF;\">(<\/span><span>request<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>params<\/span><span style=\"color: #9ECBFF;\">,<\/span><span style=\"color: #79B8FF;\"> null<\/span><span style=\"color: #9ECBFF;\">,<\/span><span style=\"color: #79B8FF;\"> 2<\/span><span style=\"color: #9ECBFF;\">)}<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">48<\/span><span style=\"color: #79B8FF;\">        \\n<\/span><span style=\"color: #9ECBFF;\">Body:<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">49<\/span><span style=\"color: #79B8FF;\">        \\n<\/span><span style=\"color: #9ECBFF;\">${<\/span><span style=\"color: #79B8FF;\">JSON<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span style=\"color: #9ECBFF;\">(<\/span><span>maskedPayload<\/span><span style=\"color: #9ECBFF;\">,<\/span><span style=\"color: #79B8FF;\"> null<\/span><span style=\"color: #9ECBFF;\">,<\/span><span style=\"color: #79B8FF;\"> 2<\/span><span style=\"color: #9ECBFF;\">)}`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">50<\/span><span>      );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">51<\/span><span style=\"color: #F97583;\">      return<\/span><span> request;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">52<\/span><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">53<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">54<\/span><span>    client.interceptors.response.<\/span><span style=\"color: #B392F0;\">use<\/span><span>(<\/span><span style=\"color: #F97583;\">async<\/span><span> (<\/span><span style=\"color: #FFAB70;\">response<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">55<\/span><span style=\"color: #F97583;\">      const<\/span><span style=\"color: #79B8FF;\"> maskedPayload<\/span><span style=\"color: #F97583;\"> =<\/span><span> maskData.<\/span><span style=\"color: #B392F0;\">maskJSON2<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">56<\/span><span>        response.data,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">57<\/span><span style=\"color: #79B8FF;\">        JSON_MASKING_CONFIG<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">58<\/span><span>      );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">59<\/span><span style=\"color: #F97583;\">      await<\/span><span> logging.<\/span><span style=\"color: #B392F0;\">debug<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">60<\/span><span style=\"color: #9ECBFF;\">        `&lt;&lt;&lt; ${<\/span><span>response<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>status<\/span><span style=\"color: #9ECBFF;\">} ${<\/span><span>response<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>request<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>method<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">toUpperCase<\/span><span style=\"color: #9ECBFF;\">()} ${<\/span><span>response<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>config<\/span><span style=\"color: #9ECBFF;\">.<\/span><span>url<\/span><span style=\"color: #9ECBFF;\">}<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">61<\/span><span style=\"color: #79B8FF;\">        \\n<\/span><span style=\"color: #9ECBFF;\">Body:<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">62<\/span><span style=\"color: #79B8FF;\">        \\n<\/span><span style=\"color: #9ECBFF;\">${<\/span><span>utils<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">truncateString<\/span><span style=\"color: #9ECBFF;\">(<\/span><span style=\"color: #79B8FF;\">JSON<\/span><span style=\"color: #9ECBFF;\">.<\/span><span style=\"color: #B392F0;\">stringify<\/span><span style=\"color: #9ECBFF;\">(<\/span><span>maskedPayload<\/span><span style=\"color: #9ECBFF;\">,<\/span><span style=\"color: #79B8FF;\"> null<\/span><span style=\"color: #9ECBFF;\">,<\/span><span style=\"color: #79B8FF;\"> 2<\/span><span style=\"color: #9ECBFF;\">),<\/span><span style=\"color: #79B8FF;\"> RESPONSE_BODY_MAX_SIZE_LOGGED<\/span><span style=\"color: #9ECBFF;\">)}<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">63<\/span><span style=\"color: #79B8FF;\">        \\n\\n<\/span><span style=\"color: #9ECBFF;\">`<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">64<\/span><span>      );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">65<\/span><span style=\"color: #F97583;\">      return<\/span><span> response;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">66<\/span><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">67<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">68<\/span><span style=\"color: #F97583;\">    return<\/span><span> client;<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">69<\/span><span>  },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">70<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">71<\/span><span style=\"color: #B392F0;\">  isResponseError<\/span><span>: (<\/span><span style=\"color: #FFAB70;\">response<\/span><span>)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">72<\/span><span style=\"color: #F97583;\">    return<\/span><span> (<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">73<\/span><span>      response.status<\/span><span style=\"color: #F97583;\"> &lt;<\/span><span style=\"color: #79B8FF;\"> 200<\/span><span style=\"color: #F97583;\"> ||<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">74<\/span><span>      response.status<\/span><span style=\"color: #F97583;\"> &gt;=<\/span><span style=\"color: #79B8FF;\"> 300<\/span><span style=\"color: #F97583;\"> ||<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">75<\/span><span>      (response.data<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #79B8FF;\"> null<\/span><span style=\"color: #F97583;\"> &amp;&amp;<\/span><span> response.data.errors<\/span><span style=\"color: #F97583;\"> !=<\/span><span style=\"color: #79B8FF;\"> null<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">76<\/span><span>    );<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">77<\/span><span>  },<\/span><\/span>\n<span class=\"giallo-l\"><span aria-hidden=\"true\" class=\"giallo-ln\" style=\"color: #444D56;\">78<\/span><span>};<\/span><\/span><\/code><\/pre>\n<p>As visible on lines <code>41-44<\/code> and <code>55-58<\/code>, while logging HTTP request and responses the code is leveraging a masking module (<a rel=\"external\" href=\"https:\/\/www.npmjs.com\/package\/maskdata\">maskdata<\/a>) to avoid leaking PII or other sensitive information, with the help of the following configuration:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">const<\/span><span style=\"color: #79B8FF;\"> JSON_MASKING_CONFIG<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>  passwordFields: [<\/span><span style=\"color: #9ECBFF;\">&quot;password&quot;<\/span><span>],<\/span><\/span>\n<span class=\"giallo-l\"><span>  uuidFields: [<\/span><span style=\"color: #9ECBFF;\">&quot;data.userContext.credentialId&quot;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &quot;data.credentialId&quot;<\/span><span>],<\/span><\/span>\n<span class=\"giallo-l\"><span>  emailFields: [<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">    &quot;username&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">    &quot;data.userContext.accountUsername&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">    &quot;data.userContext.email&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>  ],<\/span><\/span>\n<span class=\"giallo-l\"><span>  phoneFields: [<\/span><span style=\"color: #9ECBFF;\">&quot;data.userContext.mobilePhoneNumber&quot;<\/span><span>],<\/span><\/span>\n<span class=\"giallo-l\"><span>  genericStrings: [<\/span><\/span>\n<span class=\"giallo-l\"><span>    {<\/span><\/span>\n<span class=\"giallo-l\"><span>      fields: [<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;token&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.firstName&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.address1&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.lastName&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.nickName&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.birthDate&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.displayBirthDate&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.pictureUrl&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &quot;data.userContext.thumbPictureUrl&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>      ],<\/span><\/span>\n<span class=\"giallo-l\"><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span>  ],<\/span><\/span>\n<span class=\"giallo-l\"><span>};<\/span><\/span><\/code><\/pre><h3 id=\"all-that-glitters-is-not-gold\">All that glitters is not gold<\/h3>\n<p>While great in theory, <em>Common layers<\/em> came with a few traps.<\/p>\n<p>Firstly, I had to structure the code to mimic what the <em>Lambda<\/em> wrapper was expecting: as visible in the source code, an extra <code>nodejs<\/code> directory is required to bundle the common modules. Why one should bother with this Lambda-specific implementation detail while developing locally?<\/p>\n<p>Most importantly, the fact that the code was expecting common modules to be globally available on the host made the development experience a bit frustrating. To make the code work locally, I had to play with <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/gym-booking-assistant\/blob\/main\/justfile#L11\">symbolic links<\/a> \ud83e\udee4.<\/p>\n<p>Finally, let's have a look at what I did for delivering the SMS with the information about the completed booking.<\/p>\n<h2 id=\"sms-notification\">SMS notification<\/h2>\n<p>As mentioned already in a previous chapter I decided to implement the notification step simply with the help of an EventBridge rule and SNS Topic:<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-07-28_automating-my-gym-bookings-with-a-serverless-assistant\/eventbridge-completed-bookings.png\" alt=\"EventBridge rule for ClassBookingCompleted events for the user andrea\" \/><\/p>\n<p>The rule targets an SNS Topic, whose input is manipulated by an <em>EventBridge<\/em> <a rel=\"external\" href=\"https:\/\/docs.aws.amazon.com\/eventbridge\/latest\/userguide\/eb-input-transformer-tutorial.html\">Input transformer<\/a> on AWS, to create a human friendly message:<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-07-28_automating-my-gym-bookings-with-a-serverless-assistant\/eventbridge-completed-bookings-sns-target.png\" alt=\"EventBridge rule for ClassBookingCompleted events for the user andrea, SNS target\" \/><\/p>\n<p>Before reaching the SNS topic the input is transformed with the help of an <em>EventBridge<\/em> <a rel=\"external\" href=\"https:\/\/docs.aws.amazon.com\/eventbridge\/latest\/userguide\/eb-input-transformer-tutorial.html\">Input transformer <\/a> on AWS, to create a human friendly message. The transformer creates an intermediate JSON object with the following structure and values:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"json\"><span class=\"giallo-l\"><span>{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;classId&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;$.detail.class.id&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;className&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;$.detail.class.name&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">  &quot;classStartDate&quot;<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;$.detail.class.startDate&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>and ultimately builds a human friendly SMS payload with the following template:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span>La classe &lt;className&gt; in programma il &lt;classStartDate&gt; \u00e8 stata prenotata! Se non vuoi partecipare, ricorda di annullare la prenotazione per evitare penalizzazioni.<\/span><\/span><\/code><\/pre><h3 id=\"quota-request-increase-what\">Quota request increase what?<\/h3>\n<p>After a bunch of successful deliveries, all of sudden all my SMSes started failing. After enabling some basic logging (Why isn't that enabled by default?) I realized that I had reached my monthly budget for SMSes!<\/p>\n<p>How could that be true, considering that:<\/p>\n<ul>\n<li>I was using the Sandbox environment and only sending to a trusted target (my mobile number)<\/li>\n<li>I had sent between 5 and 10 SMSes<\/li>\n<\/ul>\n<p>Very surprisingly, it turned out that the default monthly budget for sending SMS is 1 USD! \ud83d\ude05<\/p>\n<p>So, I had to raise a budget increase request and after a few ping-pong chats with the AWS support I was ultimately able to convince them that there's wasn't a company, there wasn't a site and much less there wasn't a process to opt-out for those messages...<\/p>\n<h1 id=\"final-thoughts\">Final thoughts<\/h1>\n<p>That's it!<\/p>\n<p>While the project I\u2019ve built may not be memorable or flawless, I genuinely enjoyed the process of creating it. Though I may not use it regularly to book my classes, it provided me with an opportunity to dive deeper into the serverless space on AWS, beyond what I typically encounter at work. Writing about this experience also encouraged me to reflect critically on the decisions I made, considering both their benefits and limitations. I may not be entirely proud of the final product, but I\u2019m certainly happy with the valuable lessons I\u2019ve learned along the way.<\/p>\n"},{"title":"Asynchronous programming in .NET","pubDate":"Mon, 08 Apr 2024 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/","guid":"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/","description":"<p><img src=\"\/assets\/images\/posts\/2024-04-08_asynchronous-programming-in-dotnet\/concurrency.png\" alt=\"Concurrency\" \/>\n<em>An abstract representation of concurrency, as hallucinated by OpenAI<\/em><\/p>\n<!-- no toc -->\n<h1 id=\"table-of-contents\">Table of contents <!-- omit in toc --><\/h1>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#introduction\">Introduction<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#mind-refresher\">Mind refresher<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#processes\">Processes<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#threads\">Threads<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#concurrency\">Concurrency<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#parallelism\">Parallelism<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#net-threadpool-class\">.NET ThreadPool class<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#worker-and-completionport-threads\">Worker and CompletionPort threads<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#what-is-an-io-completion-port\">What is an IO Completion port?<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#why-not-simply-offloading-work-to-a-worker-thread\">Why not simply offloading work to a worker thread?<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#what-about-unix\">What about UNIX?<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#default-configuration\">Default configuration<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#custom-values\">Custom values<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#minimum-values-are-too-high-resource-starvation\">Minimum values are too high: resource starvation<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#minimum-values-are-too-low-throttling-delays\">Minimum values are too low: throttling delays<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#how-to-check-your-app-configuration\">How to check your app configuration<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#when-does-threadpool-come-into-play\">When does ThreadPool come into play<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#writing-asynchronous-code\">Writing asynchronous code<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#io-bound-scenario\">IO-bound scenario<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#cpu-bound-scenario\">CPU-bound scenario<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#avoid-calling-anything-async\">Avoid calling anything Async<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#writing-your-own-async-method\">Writing your own async method<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#configureawait-what\">ConfigureAwait what?<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h1 id=\"introduction\">Introduction<\/h1>\n<p>At work, I recently had the opportunity to contribute to our internal AMQP pub-sub reference library for .NET Core applications. Specifically, I upgraded the library's main dependency and <a rel=\"external\" href=\"https:\/\/github.com\/EasyNetQ\/EasyNetQ\/issues\/1758\">opted-in for some concurrency optimizations<\/a> at the AMQP consumers level.<\/p>\n<p>Although the update itself was relatively straightforward, I realized that I wasn't comfortable enough with the topic of concurrency in the .NET. Therefore, I spent some time reading various documentation and articles related to the subject.<\/p>\n<p>While I was reading, I wrote a private note to organize and solidify the terms and concepts, that I'm now turning into this blog post.<\/p>\n<p>I'm sharing this as a small gift to my future self and as a helpful reference for anyone who may need to investigate the same problem space.<\/p>\n<h1 id=\"mind-refresher\">Mind refresher<\/h1>\n<p>Before diving into .NET specific stuff - although already familiar with the basic terminology - I invested a few minutes to refresh some basic terminology and concepts.<\/p>\n<h2 id=\"processes\">Processes<\/h2>\n<p>A process represents an application or program. As simple as that.<\/p>\n<h2 id=\"threads\">Threads<\/h2>\n<p>A <em>thread<\/em> is a fundamental unit to which the OS allocates processor time inside a <em>process<\/em>.<\/p>\n<p>Most of the times, when talking about threads we talk about <em>Operating system<\/em> or <em>Platform<\/em> threads. <strong>To not be confused<\/strong> with abstractions provided by some languages like <a rel=\"external\" href=\"https:\/\/www.infoworld.com\/article\/3678148\/intro-to-virtual-threads-a-new-approach-to-java-concurrency.html\">Java's virtual threads<\/a><\/p>\n<p>OS threads are limited in number and parallel execution by the underlying host\/hardware:<\/p>\n<ol>\n<li>The max number of threads that a process can <em>spawn<\/em> is usually in the tens of thousands (On my Mac <code>kern.num_threads: 20480<\/code>). To avoid resources starvation, applications usually introduce <em>their<\/em> limits with a  thread pool.<\/li>\n<li>The max number of threads that can truly <em>run in parallel<\/em> is way lower and determined - with some approximation - by the number of CPU cores (8 on my laptop)<\/li>\n<\/ol>\n<p>Threads usually have a state associated (a <em>Context<\/em> to use the .NET terminology), that is extra baggage that might be required to resume a task at a later stage. This is not always explicitly mentioned in docs or included in code blocks, but it's something to be aware of as we'll see later. In .NET<\/p>\n<h2 id=\"concurrency\">Concurrency<\/h2>\n<p><em>Concurrency<\/em> means executing multiple tasks at the same time, but not necessarily simultaneously. This definition implies that concurrent tasks are tasks that can be interrupted and awaited if needed.<\/p>\n<p>When thinking about concurrency I figure myself preparing a risotto:<\/p>\n<ul>\n<li>I put some water to boil and forget about it<\/li>\n<li>While I wait for the water to heat, I clean and cut leek, onions, carrots, parsley, sometimes a potato, etc. Once done I put them in the pot, and forget about it for 1 hour approximately<\/li>\n<li>While I wait for the broth to be ready... I measure the rice to use, and then I start preparing a bit of onions for the sauce<\/li>\n<li>And so on...<\/li>\n<\/ul>\n<p>As you can see a lot of tasks, that I'm taking care of concurrently, but not in parallel! I've only 2 hands in the end.<\/p>\n<p>So, in a nutshell, <strong>concurrency is more about optimizing time and resources than going faster<\/strong>. The requirement for concurrency to happen is having tasks where you have to wait for some condition to happen (the water to boil for instance, or an HTTP response to be returned by an API). A performance improvement is usually a natural consequence of it.<\/p>\n<h2 id=\"parallelism\">Parallelism<\/h2>\n<p>By contrast <em>parallelism<\/em> means doing executing multiple tasks simultaneously. To reuse the same image we used before, we can imagine all the steps to happen at the same time.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-04-08_asynchronous-programming-in-dotnet\/cutting-vegetables-parallel.png\" alt=\"Cutting vegetables in parallel\" \/>\n<em>A drawing of a young boy cutting vegetables in parallel, as hallucinated by OpenAI<\/em><\/p>\n<h1 id=\"net-threadpool-class\">.NET ThreadPool class<\/h1>\n<p>Let's now start diving in .NET specific contents.<\/p>\n<p>As mentioned in one of the refresher's pins, applications are usually limiting their degree of parallelism with the help of a thread pool.<\/p>\n<p>In .NET, this concept is implemented with the <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/api\/system.threading.threadpool?view=net-8.0\">ThreadPool<\/a> class, a utility that's meant to ease threads management and to mitigate the risk of accidental resource starvation.<\/p>\n<p>Threads in .NET can be either <em>foreground<\/em> and <em>background<\/em> threads. Foreground threads have the ability to prevent the current application from terminating.<\/p>\n<p>All threads held by the <code>ThreadPool<\/code> class are background threads and <strong>there is only one <code>ThreadPool<\/code> instance per process\/application.<\/strong><\/p>\n<h2 id=\"worker-and-completionport-threads\">Worker and CompletionPort threads<\/h2>\n<p>The <code>ThreadPool<\/code> keeps 2 different pools of threads internally: one for the <em>worker<\/em> threads and one for <em>completionPort<\/em> threads. While there\u2019s technically no difference between worker and completion threads, they\u2019re used for different purposes.<\/p>\n<p><em><strong>Worker threads<\/strong><\/em> refer to any thread other than the main thread of your application and can be used to do any kind of work, including waiting for some I\/O to complete.<\/p>\n<p><em><strong>CompletionPort or I\/O threads<\/strong><\/em> are threads that the Threadpool reserves to dispatch callbacks from the IO completion port. But\u2026<\/p>\n<h2 id=\"what-is-an-io-completion-port\">What is an IO Completion port?<\/h2>\n<p>An <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/windows\/win32\/fileio\/i-o-completion-ports\">I\/O Completion Port (IOCP)<\/a> is a queue-like operating system object that can be used to simultaneously manage multiple I\/O operations, <strong>natively available on Windows NT operating system.<\/strong><\/p>\n<p>An important component of IOCPs are <em>file handles<\/em>: handles to any asynchronous-able I\/O endpoint - a file, a stream, a network socket\u2026<\/p>\n<p>Whenever an operation on any of the\u00a0<em>file handles<\/em>\u00a0completes, and I\/O completion packet is queued into the IOCP. This is where IOCP Threads come into play: they are in charge of completing the operation that was parked on the completion port.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2024-04-08_asynchronous-programming-in-dotnet\/completion-port.png\" alt=\"Completion port representation\" \/>\n<em>Completion port representation<\/em><\/p>\n<h3 id=\"why-not-simply-offloading-work-to-a-worker-thread\">Why not simply offloading work to a worker thread?<\/h3>\n<p>An alternative to using IOCP threads would be to just use worker threads, but doing so we would block one of our workers just because we\u2019re waiting for an event (on the file handle).<\/p>\n<p>The <code>ThreadPool<\/code> class automatically monitor and manage IOCP threads<\/p>\n<h3 id=\"what-about-unix\">What about UNIX?<\/h3>\n<p>As mentioned above, IOCP is a Windows-specific component, and as such not available <em>as-is<\/em> on UNIX systems.\nThe Common Language Runtime (CLR) takes care of creating an abstraction of IOCP in UNIX systems as well. Delving into the host-specific implementation of IOCPs seemed out of scope for my objectives.<\/p>\n<h2 id=\"default-configuration\">Default configuration<\/h2>\n<p>The thread pool provides new worker threads or I\/O completion threads on demand until it reaches the minimum for each category. <strong>By default, the minimum number of threads is set to the number of processors on a system<\/strong> (<code>Environment.ProcessorCount<\/code> property)<\/p>\n<p>The maximum value configured by default might depend on multiple factors, but In general, it is insanely high and a program should\u00a0<em>never<\/em>\u00a0get close.<\/p>\n<p>These are the results I've found while running a test I did on my 8-Core M1 MacBook Pro:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span>Environment.ProcessorCount: 8<\/span><\/span>\n<span class=\"giallo-l\"><span>MinWorkerThreads: 8<\/span><\/span>\n<span class=\"giallo-l\"><span>MinCompletionPortThreads: 1<\/span><\/span>\n<span class=\"giallo-l\"><span>MaxWorkerThreads: 32767<\/span><\/span>\n<span class=\"giallo-l\"><span>MaxCompletionPortThreads: 1000<\/span><\/span><\/code><\/pre><h2 id=\"custom-values\">Custom values<\/h2>\n<p>The ThreadPool class comes with 2 static utility that helps updating the minimum and maximum values for worker and IOCP threads, namely <code>SetMinThreads<\/code> and <code>SetMaxThreads<\/code>.<\/p>\n<h3 id=\"minimum-values-are-too-high-resource-starvation\">Minimum values are too high: resource starvation<\/h3>\n<p>Higher number of minimum threads will increase the amount of context-switching needed when jumping from thread to thread. CPU and Memory usage will be affected.<\/p>\n<h3 id=\"minimum-values-are-too-low-throttling-delays\">Minimum values are too low: throttling delays<\/h3>\n<p>This <a rel=\"external\" href=\"https:\/\/gist.github.com\/JonCole\/e65411214030f0d823cb#recommendation\">note<\/a> clarifies that the minimum number of worker\/IOCP threads should line up with the expected burst of requests that your application can handle. Otherwise, throttling kicks-in adding a delay of ~500ms for each newly created thread.<\/p>\n<p>This can lead to significant issues when your application receives a burst of requests greater than your minimum values.<\/p>\n<h2 id=\"how-to-check-your-app-configuration\">How to check your app configuration<\/h2>\n<p>To investigate your app\u2019s ThreadPool configuration one can leverage the <code>GetMinThreads<\/code> and <code>GetMaxThreads<\/code> ThreadPool\u2019s static methods.<\/p>\n<p>Also <code>Environment.ProcessorCount<\/code> property can be checked for additional context.<\/p>\n<h2 id=\"when-does-threadpool-come-into-play\">When does ThreadPool come into play<\/h2>\n<p>Quoting the .NET <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/standard\/threading\/the-managed-thread-pool\">docs<\/a>, the ThreadPool class is used in many different places:<\/p>\n<blockquote>\n<p>.NET uses thread pool threads for many purposes, including Task Parallel Library (TPL) operations, asynchronous I\/O completion, timer callbacks, registered wait operations, asynchronous method calls using delegates, and System.Net socket connections.<\/p>\n<\/blockquote>\n<p>The easiest way to leverage it is to use the <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/standard\/parallel-programming\/task-parallel-library-tpl\">Task Parallel Library (TPL)<\/a>.<\/p>\n<p>Basically, whenever your using a <code>Task<\/code> or <code>awaiting<\/code> something async in your code you're already taking advantages of it. Coming from a Java background, I have to admit that this sounded a bit unexpected but reassuring, when I read it for the first time! It turns out that when doing asynchronous programming .NET, developers can be sort-of safe without necessarily encountering headaches due to thread management problems.<\/p>\n<h1 id=\"writing-asynchronous-code\">Writing asynchronous code<\/h1>\n<p>There are usually 2 macro scenarios where you want to write asynchronous code. Again, I can't help but quoting the very well written official docs:<\/p>\n<blockquote>\n<p>The core of async programming is the <code>Task<\/code> and <code>Task&lt;T&gt;<\/code> objects, which model asynchronous operations. They are supported by the <code>async<\/code> and <code>await<\/code> keywords. The model is fairly simple in most cases:<\/p>\n<p>For I\/O-bound code, you await an operation that returns a <code>Task<\/code> or <code>Task&lt;T&gt;<\/code> inside of an async method.<\/p>\n<p>For CPU-bound code, you await an operation that is started on a background thread with the <code>Task.Run<\/code> method.<\/p>\n<\/blockquote>\n<p>Let's see a couple of examples to clarify further.<\/p>\n<h2 id=\"io-bound-scenario\">IO-bound scenario<\/h2>\n<p>An IO-bound scenario manifests whenever your awaiting on I\/O: a file, a network call...\nLet's say your code integrates an external REST API to fetch weather forecast information.<\/p>\n<p>A naive, blocking implementation of an hypothetical <code>GetWeatherForecast<\/code> function would be something like this:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"csharp\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">public<\/span><span style=\"color: #B392F0;\"> Forecast GetWeatherForecast<\/span><span>(<\/span><span style=\"color: #F97583;\">string<\/span><span style=\"color: #B392F0;\"> city<\/span><span>){<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> request<\/span><span style=\"color: #F97583;\"> = new<\/span><span style=\"color: #B392F0;\"> HttpRequestMessage<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>      HttpMethod.Get,<\/span><span style=\"color: #9ECBFF;\"> $&quot;https:\/\/weather-forecast.com\/api\/city={<\/span><span>city<\/span><span style=\"color: #9ECBFF;\">}&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span>\t  <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> response<\/span><span style=\"color: #F97583;\"> =<\/span><span> _httpClient.<\/span><span style=\"color: #B392F0;\">Send<\/span><span>(request);<\/span><\/span>\n<span class=\"giallo-l\"><span>\t\t<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> responseBody<\/span><span style=\"color: #F97583;\"> = new<\/span><span style=\"color: #B392F0;\"> StreamReader<\/span><span>(response.Content.<\/span><span style=\"color: #B392F0;\">ReadAsStream<\/span><span>()).<\/span><span style=\"color: #B392F0;\">ReadToEnd<\/span><span>();      <\/span><\/span>\n<span class=\"giallo-l\"><span>\t\t<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/ deserialise the response body into a Forecast instance and return it<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return<\/span><span> responseBody.<\/span><span style=\"color: #B392F0;\">ToForecast<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>This code would completely block the execution of the current thread until a response from the weather-forecast.com API is returned, and new requests would queue up: not very efficient.<\/p>\n<p>What we could do instead is leverage the asynchronous <code>GetAsync<\/code> method offered available on the <code>HttpClient<\/code> type, and free up the current thread's execution,\nwhile awaiting an HTTP response from the external API. To do so, we have to slightly amend our code:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"csharp\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">public async static<\/span><span style=\"color: #B392F0;\"> Task<\/span><span>&lt;<\/span><span style=\"color: #B392F0;\">Forecast<\/span><span>&gt;<\/span><span style=\"color: #B392F0;\"> GetWeatherForecastAsync<\/span><span>(<\/span><span style=\"color: #F97583;\">string<\/span><span style=\"color: #B392F0;\"> city<\/span><span>){<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> response<\/span><span style=\"color: #F97583;\"> = await<\/span><span> _httpClient<\/span><\/span>\n<span class=\"giallo-l\"><span>      .<\/span><span style=\"color: #B392F0;\">GetAsync<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">$&quot;https:\/\/weather-forecast.com\/api\/city={<\/span><span>city<\/span><span style=\"color: #9ECBFF;\">}&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> responseBody<\/span><span style=\"color: #F97583;\"> = await<\/span><span> response.Content.<\/span><span style=\"color: #B392F0;\">ReadAsStringAsync<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/ deserialise the response body into a Forecast instance and return it<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return<\/span><span> responseBody.<\/span><span style=\"color: #B392F0;\">ToForecast<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Notice how we replaced the blocking codes with a few <code>async<\/code> and <code>await<\/code> keywords. First of all we await on the network call to be completed.\nThen, we also await when reading the raw response content to a string. Last but not least, to be able to use the <code>await<\/code> keyword we need\nto mark our signature with the <code>async<\/code> keyword and wrap the Forecast type into a <code>Task<\/code> wrapper. It's also a good practice to add the <code>Async<\/code>\nsuffix to our original <code>GetWeatherForecast<\/code> method.<\/p>\n<p>In this last case, our code leverage native async methods offered by the .NET runtime via the <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/api\/system.net.http.httpclient.getasync?view=net-8.0\">HttpClient<\/a> and <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/api\/system.net.http.httpcontent.readasstringasync?view=net-8.0\">HttpContent<\/a> types.<\/p>\n<p>The <code>await<\/code> keyword included in our new code hides some sort of magic. When that is declared, the execution of the awaited task (the HTTP call or the serialization bits in the above case) are offloaded to <a href=\"https:\/\/adilisio.com\/posts\/asynchronous-programming-in-dotnet\/#worker-and-completionport-threads\">IO Threads<\/a> and the current thread is freed up and can serve other requests.<\/p>\n<p>Hopefully this explains why I said earlier that leveraging async code is <strong>more about optimizing resources usage<\/strong>, and only secondarily making your code go faster.<\/p>\n<h2 id=\"cpu-bound-scenario\">CPU-bound scenario<\/h2>\n<p>The classic example of a CPU-bound scenario is when you have to run some expensive computation. In this case, you want to keep your service\/app responsive, <em>while<\/em> running the numbers.<\/p>\n<p>Unlike the IO-bound, here there's not much to wait on, most of the work needs to happen locally on your host and cores, so what you should do is offloading the expensive task to a worker thread.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"csharp\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">public async static<\/span><span style=\"color: #B392F0;\"> Task<\/span><span>&lt;<\/span><span style=\"color: #B392F0;\">Result<\/span><span>&gt;<\/span><span style=\"color: #B392F0;\"> ComputeResult<\/span><span>(){<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> result<\/span><span style=\"color: #F97583;\"> = await<\/span><span> Task.<\/span><span style=\"color: #B392F0;\">Run<\/span><span>(()<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      \/\/ an expensive, time consuming calculation...  <\/span><\/span>\n<span class=\"giallo-l\"><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return<\/span><span> result<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>The content of the function passed to the <code>Task.Run<\/code> method gets offloaded to one of the ThreadPool's worker thread, while the current thread is freed up and can accept and process other requests without undesired side effects on the experienced responsiveness\/latency.<\/p>\n<h2 id=\"avoid-calling-anything-async\">Avoid calling anything Async<\/h2>\n<p>Notice how in the las <code>ComputeResult<\/code> method I did not add an <code>Async<\/code> suffix. What it initially could look like an omission, is indeed a deliberate choice. <a rel=\"external\" href=\"https:\/\/www.ben-morris.com\/why-you-shouldnt-create-asynchronous-wrappers-with-task-run\/\">As brilliantly explained on this post<\/a>, one should mark as <code>Async<\/code> whatever is supposed to leverage IO threads! As I hope I've clearly explained at this point, concurrency != parallelism.<\/p>\n<p>An individual interested in concurrency improvements is primarily concerned with efficiency rather than execution time: the focus is on making sure that the application can scale and handle as much load as it can.<\/p>\n<p>Wrapping a call to <code>Task.Run(...)<\/code> inside a method ending with the <code>Async<\/code> is probably causing the exact opposite of the what a consumer interested in concurrency looks like.<\/p>\n<p>A consumer invoking the following code would simply consume another (worker) thread, rather than reusing the existing resources:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"csharp\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">public async static<\/span><span style=\"color: #B392F0;\"> Task<\/span><span>&lt;<\/span><span style=\"color: #B392F0;\">Result<\/span><span>&gt;<\/span><span style=\"color: #B392F0;\"> ComputeResultAsync<\/span><span>(){<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    var<\/span><span style=\"color: #B392F0;\"> result<\/span><span style=\"color: #F97583;\"> = await<\/span><span> Task.<\/span><span style=\"color: #B392F0;\">Run<\/span><span>(()<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      \/\/ an expensive, time consuming calculation...  <\/span><\/span>\n<span class=\"giallo-l\"><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return<\/span><span> result<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre><h2 id=\"writing-your-own-async-method\">Writing your own async method<\/h2>\n<p>Most of the times our async methods (like the previous <code>GetWeatherForecastAsync<\/code>) leverage APIs that <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/visual-basic\/programming-guide\/concepts\/async\/#BKMK_APIAsyncMethods\">are <em>natively<\/em> async<\/a>.<\/p>\n<p>One looking to implement their own asynchronous method should take advantages of the <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/api\/system.threading.tasks.taskcompletionsource-1?view=net-8.0\">TaskCompletionSource&lt;TResult&gt; class<\/a>.<\/p>\n<p>How to use the <code>TaskCompletionSource&lt;TResult&gt;<\/code> class goes beyond the objective of this post.<br \/>\nThe HttpContent's <code>ReadAsStringAsync()<\/code> <a rel=\"external\" href=\"https:\/\/github.com\/microsoft\/referencesource\/blob\/51cf7850defa8a17d815b4700b67116e3fa283c2\/System\/net\/System\/Net\/Http\/HttpContent.cs#L60\">implementation<\/a> offers a juicy example though.<\/p>\n<h2 id=\"configureawait-what\">ConfigureAwait what?<\/h2>\n<p>In .NET each thread has an associated state, called <em>Context<\/em>. In a .NET application hosting a REST API, a context is for instance the HTTP request associated to an API call.<\/p>\n<p>How this context behave can be controlled via the <a rel=\"external\" href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/api\/system.threading.tasks.task.configureawait?view=net-8.0\">Task<\/a> class. By default, when a task is awaited, its state is captured by the OS, and when a task resumes its original context is re-attached.<\/p>\n<p>Resuming a thread with its original state might be useful for UI updates, but it's often (at least in my experience) not required.\nAse these lookup and re-attach operations have consequences the performances of our thread management solution, the general advice is to capture\nthe Thread's context only when required.<\/p>\n<p>To opt-out from the default thread's context behavior you can define use the ConfigureAwait(false) method. With reference to one of the code snippets above:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"csharp\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">var<\/span><span style=\"color: #B392F0;\"> forecast<\/span><span style=\"color: #F97583;\"> = await<\/span><span style=\"color: #B392F0;\"> GetWeatherForecastAsync<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;Milan&quot;<\/span><span>).<\/span><span style=\"color: #B392F0;\">ConfigureAwait<\/span><span>(<\/span><span style=\"color: #79B8FF;\">false<\/span><span>);<\/span><\/span><\/code><\/pre>"},{"title":"Elevate your Prometheus alerts with the help of unit tests","pubDate":"Fri, 06 Oct 2023 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/","guid":"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/","description":"<p><img src=\"\/assets\/images\/posts\/2023-10-06_elevate_your_prometheus_alerts_with_the_help_of_unit_tests\/peace.jpg\" alt=\"Peace of mind\" \/>\n<br\/><\/p>\n<!-- no toc -->\n<h1 id=\"table-of-contents\">Table of contents <!-- omit in toc --><\/h1>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#introduction\">Introduction<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#what-is-an-alert\">What is an alert?<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#how-can-you-test-it\">How can you test it?<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#what-about-unit-testing\">What about unit testing?<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#a-basic-example\">A basic example<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#errors-increase-for-multiple-service-instances\">Errors increase for multiple service instances<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#introducing-an-alert-duration-for-higher-precision\">Introducing an alert duration for higher precision<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#side-effects-of-using-a-duration\">Side effects of using a duration<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#alert-on-slos-with-missing-scrapes\">Alert on SLOs with missing scrapes<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#our-availability-definition\">Our availability definition<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#first-attempt\">First attempt<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#missing-scrapes\">Missing scrapes<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#different-policies-depending-on-the-time-of-the-day\">Different policies depending on the time of the day<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#alert-definition\">Alert definition<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#test-the-different-in-and-off-hours-policies\">Test the different in and off-hours policies<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#key-takeaways-and-unsolved-problems\">Key takeaways and unsolved problems<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/elevate-your-prometheus-alerts-with-the-help-of-unit-tests\/#additional-resources\">Additional resources<\/a><\/li>\n<\/ul>\n<h1 id=\"introduction\">Introduction<\/h1>\n<p>Monitoring services and reacting to outages promptly is crucial for online success.<\/p>\n<p>If your company offers online services you can\u2019t afford not being operationally excellent: your service being down for minutes (sometimes just seconds!) can create non-negligible damage to your customers and sometimes severely impact your reputation and trust. What <em>excellent<\/em> means might vary a lot, depending on the use case, the scale of your company, the maturity of your products, and your agreed SLAs, but the point is that if one of your systems experiences an outage, you\u2019re expected to be the first to know about it, to inform your customers while the incident is still ongoing, solve the issues (as soon as possible) and last but not least share a detailed report including what happened and what measures you have put in place to prevent that from happening again in future.<\/p>\n<figure>\n  <img src=\"\/assets\/images\/posts\/2023-10-06_elevate_your_prometheus_alerts_with_the_help_of_unit_tests\/operational_excellence.png\" alt=\"Operational excellence\">\n  <figcaption>Usual steps involved in an incident management process<\/figcaption>\n<\/figure>\n<p>Even if you are a company like Google, with the best software architectures and a pool of incredibly skilled engineers, it's important to recognize that failures can still occur. It's crucial to be well-prepared for such scenarios and not overlook the possibility of them happening.<\/p>\n<p>The prerequisite to meet the above expectations is to leverage a monitoring and alerting system that gives you the power to decide what metrics are really relevant for your systems\/services health and how to alert if bad symptoms show up. That said, even the most powerful monitoring tool is practically useless if you don\u2019t ingest the relevant service metrics and don\u2019t define top-quality alerts.<\/p>\n<p>What I\u2019ll talk about here mostly relates to the <em>Monitor<\/em> and <em>Identity<\/em> steps depicted in the above drawing, and it is specific to the <a rel=\"external\" href=\"https:\/\/prometheus.io\/\">Prometheus<\/a> monitoring system, which I had the chance to work with during the last years, other than being one of the most adopted open-source solutions for monitoring purposes.<\/p>\n<p>Testing that your alerting works end-to-end is fairly complicated, considering the many components <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/introduction\/overview\/#architecture\">usually involved in an alerting system<\/a>, and as such a little too much for this article.<\/p>\n<p>Here I\u2019ll focus on how to write better alerts for your systems with the help of unit tests.<\/p>\n<h1 id=\"what-is-an-alert\">What is an alert?<\/h1>\n<p>An alert is a condition that when met should trigger some sort of action in your systems. You can define an alert in Prometheus with the help of an <em>alerting rule<\/em>:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rules<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> alert<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> InternalServerErrors<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    expr<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> http_requests_total{service=&quot;payments-api&quot;, status=&quot;500&quot;} &gt; 0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There have been internal server errors in the last minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/my-playbook.com\/500-what-to-check&quot;<\/span><\/span><\/code><\/pre>\n<p>The key components of an alert in Prometheus are:<\/p>\n<ul>\n<li>An <strong>expression<\/strong> written in PromQL language, defined on a metric, which describes the condition under which the alert should fire<\/li>\n<li>A <strong>severity<\/strong> indicator, representing how bad the specific alert is and what urgency of action requires<\/li>\n<li><strong>Context<\/strong> information, useful for troubleshooting purposes and ease of readability. You can even define custom annotations to include your team\u2019s runbooks!<\/li>\n<li>Optionally, a <strong>duration<\/strong> clause describing how long to wait before treating what\u2019s defined in the expression as an actual alert (not present in the above example)<\/li>\n<\/ul>\n<p>In the above example, we are asking Prometheus to send an alert whenever a <code>500<\/code> HTTP status code is returned by any of the endpoints exposed by our <code>payments-api<\/code> service.<\/p>\n<p>Note that the above example is deliberately oversimplified: usually, we\u2019re interested in error rates\/increases, coming from multiple instances of our service. We\u2019ll see later in this article a more realistic sample using the <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\/#rate\">rate<\/a> function and <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/operators\/#aggregation-operators\">sum<\/a> operator offered by Prometheus.<\/p>\n<h1 id=\"how-can-you-test-it\">How can you test it?<\/h1>\n<p>Once we have defined our alert, how can we double-check our expectations before shipping that to production and having to wait for real issues? Is there a way to verify that our alert will fire when our condition is met, and only in that case?  In other words, can we check if our alert will notify us when an issue is there and at the same time won\u2019t piss people off by being too noisy during an off-hour on-call shift?<\/p>\n<p>Of course, we can.<\/p>\n<p>A naive approach could be to release your alert in your test environment and alter the responses of your services so that we can hit the alert condition (the <code>payments-api<\/code> in our previous example). Such a test helps you get confidence about your alerting system as a whole because you would have a chance to test not just the alert per se, but its routing which is a crucial thing for the quality of your on-call support.<\/p>\n<p>The biggest drawback of this approach is that oftentimes is time-consuming:<\/p>\n<ul>\n<li>You might have to release a temporary code change on your services to return an altered response just to meet your alert criteria;<\/li>\n<li>Depending on the alert you might need to prepare some sort of load testing;<\/li>\n<li>if you\u2019re in a team of a few engineers all working on the same services you have to pause releases in your development\/test environment until you\u2019re done with your tests;<\/li>\n<li>You could end up wasting a significant amount of time because one of <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/introduction\/overview\/#architecture\">the many components<\/a> of your alerting stack works slightly differently from your live environment, or does not work at all in development.<\/li>\n<\/ul>\n<p>Sometimes this approach is not applicable either, because your alerting system is fully working only in production (usually not a good idea).<\/p>\n<p>Another approach could be to slightly tweak your alert expression to easily meet the alert condition while developing and testing, and change it back to the final value before releasing it to production. This is surely easier than the first option presented, but at the end of the day you\u2019re not really testing your alert in its final version.<\/p>\n<h1 id=\"what-about-unit-testing\">What about unit testing?<\/h1>\n<p>It turns out there is a smarter and funnier option when you want to simply test your alerting rule, and that is offered by Prometheus itself: you can unit test your alert with the help of the <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/unit_testing_rules\/\">promtool<\/a> CLI and an additional YAML file where you can define simulated events (time series) and your assertions.<\/p>\n<h2 id=\"a-basic-example\">A basic example<\/h2>\n<p>A basic alert test file looks like the following:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rule_files<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #9ECBFF;\"> .\/alerts.yaml<\/span><span style=\"color: #6A737D;\"> # The YAML resource containing your alerting rule(s)<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\"># An array of tests, where you can define multiple tests for one or more <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\"># alerting rules<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>: <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  # Each element is a specific test <\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><span style=\"color: #6A737D;\"> # this one defines the interval each value of below time series<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # The simulated metrics that will trigger our alert. In this case we&#39;re simulating 500 errors<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_requests_total{service=&quot;payments-api&quot;, status=&quot;500&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # This means that we&#39;ll start with 0 errors, 1 at the first minute, and then 0 again on the second<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Note that the distance between values here is defined by the above interval<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0 1 0<\/span><span> <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # Here we define our expectations for the alert depending on the time our <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # rule is evaluated. <\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><span> <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # What we expect to happen at the first minute. This lines up<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # with the first 1 in the values above<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> InternalServerErrors<\/span><span style=\"color: #6A737D;\"> # The name of the rule we&#39;re evaluating<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><span style=\"color: #6A737D;\"> # What\/if alerts are expected<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">          # At the first minute we expect 1 alert to be triggered with below details<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              service<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> payments-api<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              status<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 500<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There have been internal server errors in the last minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/my-playbook.com\/500-what-to-check&quot;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><span style=\"color: #6A737D;\"> # What we expect to happen at the second minute<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # We do not expect alerts here, as the errors are reset to 0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> InternalServerErrors<\/span><\/span><\/code><\/pre>\n<p>With the help of a similar resource, you can define one or more assertions on one or multiple alerts. As you can see, you can verify that your alerts not only fire but also reset as expected! Moreover, you can test that the contextual information matches your team's needs during on-call shifts.<\/p>\n<p>The above test sample and the related alert are oversimplified, as mentioned. Let\u2019s see how unit tests can even help us more with increasingly complex (and more realistic!) alerting rules.<\/p>\n<h2 id=\"errors-increase-for-multiple-service-instances\">Errors increase for multiple service instances<\/h2>\n<p>In the previous example, we defined an alert on a simple counter, namely <code>http_requests_total<\/code>. The problem is that <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/concepts\/metric_types\/#counter\">counters<\/a> are ever-growing metrics, and they are usually reset when your application restarts. Counters per se are probably not really useful to check the <em>current<\/em> number of errors returned by your service, a much better choice is to use them in combination with Prometheus\u2019s <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\/#rate\">rate function<\/a>, which helps you calculate the per-second average rate of increase of errors. Last but not least, you likely have multiple instances of your services running in parallels, therefore you\u2019re probably interested in aggregating results and this is where the <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/operators\/#aggregation-operators\">sum operator<\/a> comes to help:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span>-<\/span><span style=\"color: #85E89D;\"> alert<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpike<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # Alert if there&#39;s a spike of 401 errors &gt; 10%<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum(rate(http_requests_total{service=&quot;payments-api&quot;, status=&quot;401&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum(rate(http_requests_total{service=&quot;payments-api&quot;}[2m])) <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      &gt; 0.1<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> warning<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There has been a spike of unauthorized requests in the last 2 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        There has been a spike of unauthorized requests in the last 2 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        Current value is {{ $value | humanizePercentage }}.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/what-to-check-in-case-of-401-spikes&quot;<\/span><\/span><\/code><\/pre>\n<p>With the above snippet, we\u2019re telling Prometheus to send an alert whenever the 401 error increase (over the total of requests) in the last 2 minutes is higher than 10%.<\/p>\n<p>Let\u2019s test this!<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rule_files<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #9ECBFF;\"> .\/alerts.yaml<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  # 2 minutes of requests growth without errors, then a first spike of errors at minute 2,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  # and then another one starting at minute 4 <\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_requests_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, status=&quot;200&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 10 20 50 100 150<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0+10x2 50+50x2<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_requests_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, status=&quot;401&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 0 3 3 20 20<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0x1 3x1 20x2<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><span style=\"color: #6A737D;\"> # Note that 1m represents the second data point in the time series (10 requests, 0 401)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpike<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 1, there haven&#39;t bee 401 errors yet. So we do not expect alerts.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpike<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 2, we have a total of 23 requests, of which 3 are 401s, so the ratio is 13.04%.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # as the time series start with 0 requests, this represents the actual rate computed by Prometheus.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Hence we expect a meaningful alert!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> warning<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There has been a spike of unauthorized requests in the last 2 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                There has been a spike of unauthorized requests in the last 2 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                Current value is 13.04%.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/what-to-check-in-case-of-401-spikes&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 3m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpike<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 3, the instant values are 50 requests and 3 unauthorized errors. That said, since our rate function&#39;s<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # window is 2m, the actual values considered are 3-0=3 401 errors and a total of<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 50-10-+3 = 43 requests. As the ratio is below our threshold, no alerts are expected this time.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span><\/code><\/pre>\n<p>Check out the <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/prometheus-rules-tests\/blob\/main\/03.unauthorized_errors_increase\/tests.yaml\">entire test case<\/a> to see what happens when counters reset!<\/p>\n<h3 id=\"introducing-an-alert-duration-for-higher-precision\">Introducing an alert duration for higher precision<\/h3>\n<p>On Prometheus, you can add a <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/alerting_rules\/#defining-alerting-rules\">duration<\/a> parameter (<code>for<\/code> clause) to the alert condition to avoid alerts firing unless the value remains above the threshold defined for some time.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span>-<\/span><span style=\"color: #85E89D;\"> alert<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpike<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # Alert if there&#39;s a spike of 401 errors &gt; 10<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # and it lasts for 5 minutes at least<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum(rate(http_requests_total{service=&quot;payments-api&quot;, status=&quot;401&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum(rate(http_requests_total{service=&quot;payments-api&quot;}[2m])) <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      &gt; 0.1<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    for<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 5m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> warning<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There has been a spike of unauthorized requests in the last 5 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        There has been a spike of unauthorized requests in the last 5 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        Current value is {{ $value | humanizePercentage }}.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/what-to-check-in-case-of-401-spikes&quot;<\/span><\/span><\/code><\/pre>\n<p>The above example is almost identical to the previous one, the only difference is that it won\u2019t fire unless the alert condition is true for at least 5 minutes. Duration is sometimes useful to improve alert precision: if you get paged, it's more likely that issues are persistent and require your focus, compared to the previous example which was subject to some level of brittleness:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rule_files<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #9ECBFF;\"> .\/alerts.yaml<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  # No errors during the first 2 minutes, then high rates of unauthorized requests for 10 minutes <\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_requests_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, status=&quot;200&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 10 20 50 100 150 200 250 300 350 400<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0+10x2 50+50x7<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_requests_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, status=&quot;401&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 0 15 30 45 60 75 90 0 0 0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0x1 15+15x5 0x2<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpikeWithDuration<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 1, there haven&#39;t bee 401 errors yet. So we do not expect alerts.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 5m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpikeWithDuration<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 5, the computed 401 increase over the request total is &gt;20%, but the increase of error <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # only started 3 minutes before, so no alerts are expected<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 7m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpikeWithDuration<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 7, we have the same increase computed as minute 5 but this time the alert is on since 5 minutes, <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Therefore we actually expect an alert!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> warning<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There has been a spike of unauthorized requests in the last 5 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                There has been a spike of unauthorized requests in the last 5 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                Current value is 23.08%.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/what-to-check-in-case-of-401-spikes&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 8m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpikeWithDuration<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 8, errors are decreasing but we still expect an alert<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> warning<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;There has been a spike of unauthorized requests in the last 5 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                There has been a spike of unauthorized requests in the last 5 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                Current value is 13.04%.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/what-to-check-in-case-of-401-spikes&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 9m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> UnauthorizedRequestsSpikeWithDuration<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 9, the rate of errors over the last 2m window interval is 0, so <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # no alerts are expected<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span><\/code><\/pre><h3 id=\"side-effects-of-using-a-duration\">Side effects of using a duration<\/h3>\n<p>Beware of the drawbacks of using an alert duration though! Adding a duration might affect your incident response times in a significant way and Google engineers do a great job telling more about that in <a rel=\"external\" href=\"https:\/\/sre.google\/workbook\/alerting-on-slos\/\">this chapter of the SRE book<\/a>.<\/p>\n<p>The purpose of this subchapter is not to suggest one or the other approaches, but instead to show how unit tests could help you focus more on the actual differences and to find out what\u2019s best for your particular use case.<\/p>\n<h2 id=\"alert-on-slos-with-missing-scrapes\">Alert on SLOs with missing scrapes<\/h2>\n<p>In this example, we\u2019ll see another standard alert example, where we want to alert if we are not meeting our agreed service level objective (SLO) of 99.9% internal availability for our <code>payments-api<\/code>, and also see what happens in case no metrics are available!<\/p>\n<h3 id=\"our-availability-definition\">Our availability definition<\/h3>\n<p>Availability here is not defined as mere uptime, but more as request availability: we\u2019re not interested in checking that our service is up and running - or better say, not just that - but that our API actually returns successful responses (almost) all the time. With <em>internal<\/em> availability we mean the availability of our <code>payments-api<\/code> and its dependencies, excluding the ones that are external to our infrastructure (Like a bank, for instance).<\/p>\n<p>More formally, we define the internal availability as:\n$$\n\\frac{Successful\\ requests} {Total\\ request - Invalid\\ requests - External\\ dependency\\ errors}\n$$<\/p>\n<p>In the below samples, we\u2019ll use a custom counter metrics called <code>http_server_sli_total<\/code>, coming with the usual service label and an additional <code>sli_error_type<\/code> label that will help us differentiate between:<\/p>\n<ul>\n<li>Successes \u2192 Empty value<\/li>\n<li>Invalid requests \u2192 <code>invalid_request_error<\/code><\/li>\n<li>Internal dependency errors \u2192 <code>internal_dependency_error<\/code><\/li>\n<li>External dependency errors \u2192<code>external_dependency_error<\/code><\/li>\n<\/ul>\n<h3 id=\"first-attempt\">First attempt<\/h3>\n<p>Let\u2019s start with a basic sample, assuming for simplicity that we always have metrics available, or - in other words - our service has significant traffic already.<\/p>\n<p>A sample alert rule would be like this:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span>-<\/span><span style=\"color: #85E89D;\"> alert<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # Alert if we&#39;re not meeting our SLO of 99.9% availability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type=&quot;&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type!~&quot;invalid_request_error|external_dependency_error&quot;}[2m])) <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      &lt; 0.999<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    for<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;payments-api has had a low internal availability for the operation {{ $labels.operation}} in the last 3 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        payments-api has had a low internal availability for the operation {{ $labels.operation}} in the last 3 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        Current value is {{ $value | humanizePercentage }}.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/low-avaiability&quot;<\/span><\/span><\/code><\/pre>\n<p>And, its test:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rule_files<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #9ECBFF;\"> .\/alerts.yaml<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 50 100 150 200 250 300<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 50+50x5<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;invalid_request_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 10 20 30 40 50<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0 10+10x4<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;internal_dependency_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 0 15 30 45 0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0x1 15+15x2 0<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;external_dependency_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 0 0 10 10 0 0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 0x1 10x1 0x1<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # At minute 1, only 2 invalid request errors are thrown and our availability scores 100%. No alerts expected<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Things start to change at minute 2, were we observe 10 internal_dependency_error. Our computed availability is 100\/(100+15)=86.96%<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # That said it&#39;s the first time that we do no meet our SLO, there we do not alert yet! <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 3m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Availability is below standard but the 2 minutes duration is not found yet. No alerts<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 4m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Alerts are expected here! At minute 4, the computed availability is 100\/(100+30)=76.92% <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              operation<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> CreatePayment<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;payments-api has had a low internal availability for the operation CreatePayment in the last 3 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                payments-api has had a low internal availability for the operation CreatePayment in the last 3 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                Current value is 76.92%.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/low-avaiability&quot;<\/span><\/span><\/code><\/pre>\n<p>Our alert seems to actually do what we want we want, but in our alert rule, we always assume some level of HTTP traffic for our <code>payments-api<\/code>. It\u2019s a fair and positive assumption, but perhaps something that we can\u2019t take for granted on day 1 of our API. How is our alert above doing in case of low or missing traffic?<\/p>\n<p>Let\u2019s have a look.<\/p>\n<h3 id=\"missing-scrapes\">Missing scrapes<\/h3>\n<p>Let\u2019s see what happens if we see some missing metrics scrape: let\u2019s imagine that our service has low traffic and we just released a new version of our API, which led to counter resets. This scenario is very close to the one that I tried to describe in one of my <a rel=\"external\" href=\"https:\/\/adilisio.com\/posts\/fooled-by-prometheus-rate-function\/\">previous posts<\/a>.<\/p>\n<p>Our application does not <a rel=\"external\" href=\"https:\/\/www.doit.com\/making-peace-with-prometheus-rate\/\">initialize counters to 0<\/a> when new services are started, which turns out to be a fairly common thing.<\/p>\n<p>Let\u2019s try to change our test time series to account for missing scrapes and see how our alert behaves. Let\u2019s zoom in on the <code>input_series<\/code> block for a bit:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #6A737D;\"># ... <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ 1 4 7 10 13 16<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x3 1+3x5<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;invalid_request_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ _ _ 1 1 1 0 0 <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x5 1x2 0x1<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;internal_dependency_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ 1 2 0 0 0 0 0 <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x3 1+1x1 0x4<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;external_dependency_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ _ _ _ _ _ _ _ <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x10<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\"># ... <\/span><\/span><\/code><\/pre>\n<p>With the above time series, we\u2019re simulating a scenario where our service starts to get API requests from the third minute. On minute 3 we have a success and an internal error, hence we would expect an exact 50% availability score. At minute 4 we have a total increase of 6 requests, made of 2 successes and 4 internal errors with a computed availability of roughly 66.67%. Finally, at minute 5 our results increased by 6 successes, 1 invalid request (in theory irrelevant to our computation), and 1 internal error (due to how Prometheus does extrapolation). So, we would expect roughly an 85.71% availability at minute 6. Since our availability threshold was lower than the expected one for 2 minutes (remember the <code>for<\/code> duration clause), we should get an alert now! Let\u2019s complete the test and give it a try:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rule_files<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #9ECBFF;\"> .\/alerts.yaml<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ 1 4 7 10 13 16<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x3 1+3x5<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;invalid_request_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ _ _ 1 1 1 0 0 <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x5 1x2 0x1<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;internal_dependency_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ 1 2 _ _ _ _ _ <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x3 1+1x1 _x5<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;http_server_sli_total{service=&quot;payments-api&quot;, pod=&quot;payments-service-abcd&quot;, operation=&quot;CreatePayment&quot;, sli_error_type=&quot;external_dependency_error&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # _ _ _ _ _ _ _ _ _ _ <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> _x10<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 5m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Availability is below standard but the 2 minutes duration is not found yet. No alerts<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              operation<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> CreatePayment<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;payments-api has had a low internal availability for the operation CreatePayment in the last 3 minutes!&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                payments-api has had a low internal availability for the operation CreatePayment in the last 3 minutes!<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                Current value is 85.71%.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/low-avaiability&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 6m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span><\/code><\/pre>\n<p>Surprisingly, the above test fails!<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"shellscript\"><span class=\"giallo-l\"><span style=\"color: #B392F0;\">Unit<\/span><span style=\"color: #9ECBFF;\"> Testing:  06.slo_based_alerts_missing_scrapes\/tests.yaml<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">  FAILED:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    alertname:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability, time: 5m,<\/span><span> <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        exp:[<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            0:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">              Labels:<\/span><span style=\"color: #9ECBFF;\">{alertname=<\/span><span style=\"color: #B392F0;\">&quot;LowInternalAvailability&quot;,<\/span><span style=\"color: #9ECBFF;\"> operation=&quot;CreatePayment&quot;, severity=&quot;critical&quot;}<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">              Annotations:<\/span><span style=\"color: #9ECBFF;\">{description=<\/span><span style=\"color: #B392F0;\">&quot;payments-api has had a low internal availability for the operation CreatePayment in the last 2 minutes! Current value is 85.71%.\\n&quot;,<\/span><span style=\"color: #9ECBFF;\"> runbook_url=&quot;https:\/\/your-oncall-docs.com\/low-avaiability&quot;, summary=&quot;payments-api has had a low internal availability for the operation CreatePayment in the last 2 minutes!&quot;}<\/span><\/span>\n<span class=\"giallo-l\"><span>            ], <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        got:[]<\/span><\/span><\/code><\/pre>\n<p>But\u2026 why?<\/p>\n<p>Let\u2019s leverage an additional test utility offered by Prometheus which gives us the option to actually assert the expected value of our expression at any given time, rather than the presence of an alert. Welcome <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/unit_testing_rules\/#promql_test_case\">promql_expr_test<\/a>, an extra assertion block that we can define besides the <code>alert_rule_test<\/code> one.<\/p>\n<p>With the help of this new block, we can translate our expectations above into actual <code>promtool<\/code> assertions:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rule_files<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #9ECBFF;\"> .\/alerts.yaml<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>: <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # ... unchanged ...<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    promql_expr_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # At minute 3 we expect 50% availability<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type=&quot;&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type!~&quot;invalid_request_error|external_dependency_error&quot;}[2m]))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 3m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_samples<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> value<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 0.5<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # At minute 4 we expect roughly 66.67% availability. The increase of <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # successes over the last 2 minutes is computed with the help of some <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # extrapolation mechanism: Prometheus applies the same rate experienced in <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # the last minute, cutting off at 0. So the number of successful requests<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # that Prometheus computes for minute 2 is 0 (instead of -2)<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type=&quot;&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type!~&quot;invalid_request_error|external_dependency_error&quot;}[2m]))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 4m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_samples<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> value<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 0.6666666666666666<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            labels<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;{operation=&quot;CreatePayment&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # At minute 5 we expect roughly 75% availability<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type=&quot;&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type!~&quot;invalid_request_error|external_dependency_error&quot;}[2m]))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 5m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_samples<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> value<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 0.75<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            labels<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;{operation=&quot;CreatePayment&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # At minute 6 we expect roughly 100% availability<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type=&quot;&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">          sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type!~&quot;invalid_request_error|external_dependency_error&quot;}[2m]))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 6m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_samples<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> value<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 1<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            labels<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;{operation=&quot;CreatePayment&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # ... unchanged ...<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span>\t\t\t<\/span><\/span><\/code><\/pre>\n<p>By running again our test we should now get a new - more explicit - error:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">expr<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;sum by(operation) (rate(http_server_sli_total{service=<\/span><span style=\"color: #79B8FF;\">\\&quot;<\/span><span style=\"color: #9ECBFF;\">payments-api<\/span><span style=\"color: #79B8FF;\">\\&quot;<\/span><span style=\"color: #9ECBFF;\">, sli_error_type=<\/span><span style=\"color: #79B8FF;\">\\&quot;\\&quot;<\/span><span style=\"color: #9ECBFF;\">}[2m])) \/ sum by(operation) (rate(http_server_sli_total{service=<\/span><span style=\"color: #79B8FF;\">\\&quot;<\/span><span style=\"color: #9ECBFF;\">payments-api<\/span><span style=\"color: #79B8FF;\">\\&quot;<\/span><span style=\"color: #9ECBFF;\">, sli_error_type!~<\/span><span style=\"color: #79B8FF;\">\\&quot;<\/span><span style=\"color: #9ECBFF;\">invalid_request_error|external_dependency_error<\/span><span style=\"color: #79B8FF;\">\\&quot;<\/span><span style=\"color: #9ECBFF;\">}[2m]))<\/span><span style=\"color: #79B8FF;\">\\n<\/span><span style=\"color: #9ECBFF;\">&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">  time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 3m,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">  exp<\/span><span>: {}<\/span><span style=\"color: #9ECBFF;\"> 5E-01<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">  got<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> nil<\/span><\/span><\/code><\/pre>\n<p>This is basically saying that our expectations for what should have happened at minute 3 are wrong: we expected the computed rate to be 50% but it turns out that Prometheus can\u2019t compute any value!<\/p>\n<p>The error we made was to <em>assume<\/em> that a missing scrape would be considered as a 0 by Prometheus, but that is not the case: a missing scrape is more simply an undefined value. <a rel=\"external\" href=\"https:\/\/www.metricfire.com\/blog\/understanding-the-prometheus-rate-function\/\">As the rate function requires at least 2 data points to compute a value<\/a>, it can\u2019t actually find one in this case. So, the correct assertion for minute 3 is as follows:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span>-<\/span><span style=\"color: #85E89D;\"> expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type=&quot;&quot;}[2m])) \/<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      sum by(operation) (rate(http_server_sli_total{service=&quot;payments-api&quot;, sli_error_type!~&quot;invalid_request_error|external_dependency_error&quot;}[2m]))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 3m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    exp_samples<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # we do not expect any value here!<\/span><\/span><\/code><\/pre>\n<p>Finally, due to the missing increase of errors at minute 3, our alert condition only stays true for 1 minute. Therefore there won\u2019t be any alert firing, because the alert duration that we declared with the help of the for clause is 2 minutes! To get the test passing, we should then modify our <code>alert_rule_test<\/code> assertions to never expect an alert!<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # No errors nor requests at this stage. No alerts expected<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 5m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # Availability is below standard but the 2 minutes duration<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # threshold is not reached yet. No alerts expected<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 6m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> LowInternalAvailability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # In order for a metric to disappear from Prometheus we need <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # at least 5 missing scrapes. Therefore the same value as minute 5 <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # is considered for internal errors, hence no increase is found.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # At the same time successful requests are growing so the availability<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # goes back to 100%. No alerts are expected!<\/span><\/span><\/code><\/pre>\n<p>At this point, I hope that you\u2019re convinced a bit more of how unit testing rules can shed light the on internal of Prometheus and help you set exact expectations for your production systems!<\/p>\n<p>The entire alert and test example related to this last scenario can be found <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/prometheus-rules-tests\/blob\/main\/06.slo_based_alerts_missing_scrapes\/tests.yaml\">here<\/a>.<\/p>\n<h2 id=\"different-policies-depending-on-the-time-of-the-day\">Different policies depending on the time of the day<\/h2>\n<p>Let\u2019s now see one last example where unit tests can come into help: setting different alerting policies based on the time of the day. This is particularly useful when you want to treat minor issues very noisily during working hours but to turn a blind eye to the same when happening at night, to not bother your on-callers for low-impact glitches.<\/p>\n<h3 id=\"alert-definition\">Alert definition<\/h3>\n<p>I\u2019ve been using this setup recently to alert on AMQP dead-lettered messages:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">rules<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> alert<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesInHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;{{ $labels.queue }} has {{ $value }} dead-lettered messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      description<\/span><span>:<\/span><span style=\"color: #F97583;\"> |<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        There are {{ $value }} dead-lettered messages on the `{{ $labels.queue }}` queue in the `{{ $labels.vhost }}` virtual host.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/dead-lettered-messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # During in-hours shifts (Monday to Friday from 10AM UTC to 4:59PM UTC) we want a noisy<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # policy for alerts: even one single message in DLQ for at least 2 minutes should page us<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # Note that:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # - the amqp_message_ready metric is a counter that represents the messages which are sitting<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    #   on a queue ready to be consumed <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # - we&#39;re using the max operator to aggregate metrics coming from multiple service instances\/pods<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> |<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      max(amqp_messages_ready{vhost=&quot;payments&quot;, queue=&quot;notifications.deadletter&quot;}) by (queue, vhost) &gt; 0<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      and on() ((hour() &gt;= 10 &lt; 17) and (day_of_week() &gt;= 1 &lt; 6))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    for<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> alert<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesOffHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;{{ $labels.queue }} has {{ $value }} dead-lettered messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      description<\/span><span>:<\/span><span style=\"color: #F97583;\"> |<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        There are {{ $value }} dead-lettered messages on the `{{ $labels.queue }}` queue in the `{{ $labels.vhost }}` virtual host.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/dead-lettered-messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # During off-hours shifts (Monday to Friday before 10AM and from 5PM, and at every <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # hour during the weekend) we want a more relaxed policy: we should only page the on-caller<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    # if the number of dead-lettered messages is greater than 3.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    expr<\/span><span>:<\/span><span style=\"color: #F97583;\"> |<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      max(amqp_messages_ready{vhost=&quot;payments&quot;, queue=&quot;notifications.deadletter&quot;}) by (queue, vhost) &gt; 3<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">      and on() (((hour() &gt;= 17 or hour() &lt; 10) and (day_of_week() &gt;= 1 &lt; 6)) or (day_of_week() == 0 or day_of_week() &gt;= 6))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    for<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span><\/code><\/pre>\n<p>In the above alert, I have defined a noisy alert policy for dead-lettered messages during the in-hours shift (Monday to Friday from 10 AM to 5 PM) and a more relaxed one for the off-hours shift (Monday to Friday before 10 AM and after 5 PM, at every hour during the weekend): the condition for paging someone is at least 1 message during the in-hours shift, and at least 4 of them at night or during the weekend.<\/p>\n<h3 id=\"test-the-different-in-and-off-hours-policies\">Test the different in and off-hours policies<\/h3>\n<p>Let\u2019s now define a test to verify our expectations during the off-hours shift first: we don\u2019t want to spoil our sleep!<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  # Off-hours test<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # We simulate to different series of metrics produced by different pods  <\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;amqp_messages_ready{vhost=&quot;payments&quot;, queue=&quot;notifications.deadletter&quot;, pod=&quot;abcd&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;10x3&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;amqp_messages_ready{vhost=&quot;payments&quot;, queue=&quot;notifications.deadletter&quot;, pod=&quot;efgh&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;0x3&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesOffHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # No errors are expected at this stage. The alert condition changed to true just now<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesOffHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # The alert condition has been true on 2 consecutive minutes, <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # so an alert is now expected!<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              queue<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> notifications.deadletter<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              vhost<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> payments<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;notifications.deadletter has 10 dead-lettered messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                There are 10 dead-lettered messages on the `notifications.deadletter` queue in the `payments` virtual host.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/dead-lettered-messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesInHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # we do not expect alerts from the in-hours policy as the defined shift starts at 10AM UTC<\/span><\/span><\/code><\/pre>\n<p>It\u2019s important to note how the <code>eval_time<\/code> definition is relative to midnight UTC and to the current day.\nFinally, let\u2019s also check that even a single dead-lettered message is enough to alert us at work:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">tests<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  # In-hours test<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> interval<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    input_series<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">      # We simulate to different series of metrics produced by different pods  <\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;amqp_messages_ready{vhost=&quot;payments&quot;, queue=&quot;notifications.deadletter&quot;, pod=&quot;abcd&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # 10 hours is 600 minutes...<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;1x600 1x1&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> series<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;amqp_messages_ready{vhost=&quot;payments&quot;, queue=&quot;notifications.deadletter&quot;, pod=&quot;efgh&quot;}&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        values<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &#39;0x600 0x1&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    alert_rule_test<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 10h<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesInHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # No errors are expected at this stage. The alert condition changed to true just now (because of the hour change)<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 10h1m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesInHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # Same as the above one, our alert condition has not been true for 2 minutes (yet)<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 10h2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesInHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # The alert condition has been true on 2 consecutive minutes, <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # so an alert is now expected!<\/span><\/span>\n<span class=\"giallo-l\"><span>          -<\/span><span style=\"color: #85E89D;\"> exp_labels<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              severity<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> critical<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              queue<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> notifications.deadletter<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              vhost<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> payments<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">            exp_annotations<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              summary<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;notifications.deadletter has 1 dead-lettered messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              description<\/span><span>:<\/span><span style=\"color: #F97583;\"> &gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                There are 1 dead-lettered messages on the `notifications.deadletter` queue in the `payments` virtual host.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">              runbook_url<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> &quot;https:\/\/your-oncall-docs.com\/dead-lettered-messages&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>      -<\/span><span style=\"color: #85E89D;\"> eval_time<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> 10h2m<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        alertname<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> DeadLetteredMessagesOffHours<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exp_alerts<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        # we do not expect alerts from the off-hours policy as the defined shift starts at 10AM UTC<\/span><\/span><\/code><\/pre>\n<p>Isn\u2019t this a cool way of raising confidence in your alerts, even at night? You can focus on preparing your dinner instead of making sure you won't be woken up by a negligible glitch tonight.<\/p>\n<h1 id=\"key-takeaways-and-unsolved-problems\">Key takeaways and unsolved problems<\/h1>\n<p>By now, I hope you are really looking forward to adding unit tests to your Prometheus alerts!<\/p>\n<p>In my experience, they have been really helpful in improving the operational readiness of my team: they did not just help set correct alerting rules but also, and I\u2019d say almost equally importantly, they encouraged my team to always decorate each and every alert with all the meaningful context, from clear summaries and descriptions to links to our on-call playbooks.<\/p>\n<p>Last but not least these tests have been probably my favorite way to understand a bit more about how Prometheus works, especially when it comes to the rate functions and its extrapolation magic as well as counter resets!<\/p>\n<p>That being said, it\u2019s important to note that this type of test is partial: the tests shown in this article mostly focus on making sure your Prometheus rules are working as you would expect.<\/p>\n<p>They do not necessarily help you write simpler and\/or more performant queries, and with this kind of test, you can\u2019t actually check that the metrics you are asserting actually exist on your Prometheus database! From a unit test perspective, you can have a state-of-the-art alert, but if you type <code>http_request_total<\/code> (singular) instead of <code>http_requests_total<\/code> (plural) you might get a very unpleasant and unexpected call from one of your customers.<\/p>\n<p>This category of problems requires different testing strategies and tools. One example is <a rel=\"external\" href=\"https:\/\/cloudflare.github.io\/pint\/\">pint<\/a>, developed by the Cloudflare engineering team. I may write about it in a future post here!<\/p>\n<h1 id=\"additional-resources\">Additional resources<\/h1>\n<p><a rel=\"external\" href=\"https:\/\/github.com\/dili91\/prometheus-rules-tests\">Here<\/a> you can find all the alerts and tests shown in this article.<\/p>\n<p>Moreover, a list of of resources I've either directly referenced here or used to prepare this post:<\/p>\n<ul>\n<li><a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/introduction\/overview\/#what-are-metrics\">Prometheus official documentation<\/a><\/li>\n<li><a rel=\"external\" href=\"https:\/\/www.youtube.com\/watch?v=7uy_yovtyqw\">PromLabs YouTube chanel - Understanding Counter Rates and Increases in PromQL<\/a><\/li>\n<li><a rel=\"external\" href=\"https:\/\/www.doit.com\/making-peace-with-prometheus-rate\/\">Doit.com - Making peace with Prometheus rate()<\/a><\/li>\n<li><a rel=\"external\" href=\"https:\/\/samber.github.io\/awesome-prometheus-alerts\/rules.html\">Awesome Prometheus alerts - Collection of alerting rules<\/a><\/li>\n<li><a rel=\"external\" href=\"https:\/\/sre.google\/workbook\/on-call\/\">Google SRE Book - On-Call section<\/a><\/li>\n<li><a rel=\"external\" href=\"https:\/\/blog.cloudflare.com\/monitoring-our-monitoring\">Cloudflare blog - Monitoring our monitoring<\/a><\/li>\n<\/ul>\n"},{"title":"Sequentially starting containers in a Kubernetes pod","pubDate":"Sun, 19 Mar 2023 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/sequentially-starting-containers-in-a-kubernetes-pod\/","guid":"https:\/\/adilisio.com\/posts\/sequentially-starting-containers-in-a-kubernetes-pod\/","description":"<p><img src=\"\/assets\/images\/posts\/2023-03-19_sequentially_starting_containers_in_a_kubernetes_pod\/kubernetes.jpg\" alt=\"Kubernetes\" \/>\n<br\/><\/p>\n<p>When running multiple Docker containers in a Kubernetes pod, it's important to start them in the correct order to avoid any potential issues or downtime. One way to achieve this is by using Kubernetes' <code>postStart<\/code> hooks. In this blog post, we'll explore how to use <code>postStart<\/code> hooks to sequentially start Docker containers in a Kubernetes pod.<\/p>\n<h2 id=\"the-containers\">The Containers<\/h2>\n<p>Let's assume we have three Docker containers: <code>db<\/code>, <code>api<\/code>, and <code>worker<\/code>. The <code>db<\/code> container is responsible for running the database, <code>api<\/code> is responsible for serving the API endpoints, and <code>worker<\/code> is responsible for running background tasks.<\/p>\n<h2 id=\"the-pod-definition\">The Pod Definition<\/h2>\n<p>To start these containers sequentially, we'll create a Kubernetes pod definition with <code>initContainers<\/code> and <code>postStart<\/code> hooks. The <code>initContainers<\/code> section will ensure that the <code>db<\/code> container starts up and initializes the database before the other containers are started. The <code>postStart<\/code> hook for the <code>api<\/code> container will ensure that the <code>worker<\/code> container only starts once the API is available.<\/p>\n<p>Here's an example of what the pod definition might look like:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"yaml\"><span class=\"giallo-l\"><span style=\"color: #85E89D;\">apiVersion<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> v1<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">kind<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> Pod<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">metadata<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">  name<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> my-app<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">spec<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">  initContainers<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> name<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> db<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    image<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> my-db-image<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    command<\/span><span>: [<\/span><span style=\"color: #9ECBFF;\">&#39;bash&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;-c&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;echo &quot;Starting database...&quot; &amp;&amp; sleep 5&#39;<\/span><span>]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">  containers<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> name<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> api<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    image<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> my-api-image<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    command<\/span><span>: [<\/span><span style=\"color: #9ECBFF;\">&#39;bash&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;-c&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;echo &quot;Starting API...&quot; &amp;&amp; sleep 5&#39;<\/span><span>]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    ports<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span>    -<\/span><span style=\"color: #85E89D;\"> containerPort<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 8080<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    readinessProbe<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      httpGet<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        path<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> \/healthz<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        port<\/span><span>:<\/span><span style=\"color: #79B8FF;\"> 8080<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    lifecycle<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">      postStart<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">        exec<\/span><span>:<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">          command<\/span><span>: [<\/span><span style=\"color: #9ECBFF;\">&#39;bash&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;-c&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;until $(curl --output \/dev\/null --silent --head --fail http:\/\/localhost:8080\/healthz); do echo &quot;Waiting for API...&quot; &amp;&amp; sleep 1; done&#39;<\/span><span>]<\/span><\/span>\n<span class=\"giallo-l\"><span>  -<\/span><span style=\"color: #85E89D;\"> name<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> worker<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    image<\/span><span>:<\/span><span style=\"color: #9ECBFF;\"> my-worker-image<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #85E89D;\">    command<\/span><span>: [<\/span><span style=\"color: #9ECBFF;\">&#39;bash&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;-c&#39;<\/span><span>,<\/span><span style=\"color: #9ECBFF;\"> &#39;echo &quot;Starting worker...&quot; &amp;&amp; sleep 5&#39;<\/span><span>]<\/span><\/span><\/code><\/pre><h2 id=\"how-it-works\">How it Works<\/h2>\n<p>In the above pod definition, the <code>db<\/code> container will start first and initialize the database. Once the database is up and running, the <code>api<\/code> container will start and wait until its <code>readinessProbe<\/code> endpoint returns a successful response. Finally, the <code>worker<\/code> container will start and wait until the API is available before running any background tasks.<\/p>\n<p>The <code>postStart<\/code> hook for the <code>api<\/code> container uses a command that sends a curl request to the API's <code>readinessProbe<\/code> endpoint. It will keep retrying until it receives a successful response, indicating that the API is ready to handle requests. This ensures that the <code>worker<\/code> container only starts once the API is available.<\/p>\n<p>By using <code>postStart<\/code> hooks, we can ensure that our Docker containers start up in the correct order and avoid any potential issues or downtime in our Kubernetes environment.<\/p>\n"},{"title":"Implementing a rate limiter for our API in Rust","pubDate":"Mon, 06 Feb 2023 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/","guid":"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/","description":"<p><img src=\"\/assets\/images\/posts\/2023-01-15_implementing-a-rate-limiter-for-our-api-in-rust\/speed.jpg\" alt=\"Speed\" \/>\n<br\/><\/p>\n<p>In this blog post I want to talk about rate limiting and why you should probably care in the context of APIs development.<\/p>\n<p>The first half I'll focus on the principles of rate limiting and the most common solutions. Finally, I'll go through how to build a rate limiter, written in Rust and based on Redis.<\/p>\n<!-- no toc -->\n<h1 id=\"table-of-contents\">Table of contents <!-- omit in toc --><\/h1>\n<p>Here is the complete list of topics I'll touch in this article:<\/p>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#what-is-rate-limiting\">What is rate limiting<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-focus\">Our focus<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#why-rate-limiting-is-important\">Why rate limiting is important<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#a-shield-for-our-infrastructure\">A shield for our infrastructure<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#to-meet-our-quality-standards\">To meet our quality standards<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#for-service-quotas\">For service quotas<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-billing-guard\">Our billing guard<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#local-vs-distributed\">Local vs distributed<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#known-algorithms\">Known algorithms<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#token-bucket\">Token bucket<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#leaky-bucket\">Leaky bucket<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#fixed-window\">Fixed window<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#sliding-window\">Sliding window<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#expressiveness-at-the-api-level\">Expressiveness at the API level<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#building-a-rate-limiter\">Building a rate limiter<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-requirements\">Our requirements<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-tech-stack\">Our tech stack<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#remote-state-management\">Remote state management<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#why-rust\">Why Rust<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#github-repository\">Github repository<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#implementation-details\">Implementation details<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-main-trait\">Our main trait<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-rate-limiter-instance\">Our rate limiter instance<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-algorithm\">Our algorithm<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#on-the-shoulder-of-redis\">On the shoulder of Redis<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-trait-implemented\">Our trait implemented<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#the-beauty-of-error-handling-in-rust\">The beauty of error handling in Rust<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#testing-our-library\">Testing our library<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#using-it\">Using it<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-pilot-project\">Our pilot project<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#building-blocks\">Building blocks<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#actix-middlewares-a-tldr\"><em>Actix<\/em> middlewares, a <code>TLDR;<\/code><\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#injecting-our-rate-limiter\">Injecting our rate limiter<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#configuring-it\">Configuring it<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#testing-our-setup\">Testing our setup<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#basic-manual-tests\">Basic, manual tests<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#simulating-a-burst-of-request\">Simulating a burst of request<\/a>\n<ul>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#docker-compose-setup\">Docker compose setup<\/a><\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#generating-load-with-k6\">Generating load with <code>k6<\/code><\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#bonus-points-sliding-window-algorithm\">Bonus points: <em>Sliding window<\/em> algorithm<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#final-considerations\">Final considerations<\/a><\/li>\n<\/ul>\n<p>Let's start!<\/p>\n<h1 id=\"what-is-rate-limiting\">What is rate limiting<\/h1>\n<p>Rate limiting is a technique used to make sure that an operation\/service is executed\/requested\nwithin certain limits. Basically, it's a defensive measure that prevents the excessive consumption of\na resource.<\/p>\n<h2 id=\"our-focus\">Our focus<\/h2>\n<p>Whilst the principles of rate limiting can be applied to a wide range of disciplines,\nin this article I'll focus on its use in the context of software engineering, specifically in\nthe context of API development.<\/p>\n<h2 id=\"why-rate-limiting-is-important\">Why rate limiting is important<\/h2>\n<p>There are multiple reasons to adopt rate limiting on your services, let's see the main ones.<\/p>\n<h3 id=\"a-shield-for-our-infrastructure\">A shield for our infrastructure<\/h3>\n<p>The most common and basic need is to protect our services from resources starvation. Let's say that one of our REST endpoint internally deals with some heavy database query, or does an expensive network call to an enterprise service bus, which is constantly under high pressure - not a very infrequent scenario. In such conditions, rate limiting is a simple yet powerful trick to prevent undesired, self-induced denial of service and cascading failures.<\/p>\n<h3 id=\"to-meet-our-quality-standards\">To meet our quality standards<\/h3>\n<p>Keeping control of the maximum allowed requests on our services does not simply shield the underlying infrastructure, but also help us meeting the desired availability and performance goals, thus the quality of our service as experienced by our customers. If we know what's our system breaking point, there's no point in letting our traffic reaching that, as we know that our services would not behave as good as when in normal circumstances.<\/p>\n<h3 id=\"for-service-quotas\">For service quotas<\/h3>\n<p>Rate limiting relates as well to the concept of service quotas. Perhaps we want our service to allow a different rate of requests based on the pricing plan on which our users are running. Or, since we know that our service is used worldwide by millions of people, we want to provide everybody a reasonable and fair experience, without affecting other consumers.<\/p>\n<h3 id=\"our-billing-guard\">Our billing guard<\/h3>\n<p>Last but not least, rate limiting is also our friend when it comes to save some money. This is particularly relevant in the context of infrastructures hosted on cloud providers, that give us\nthe option to <em>infinitely<\/em> scale our services, either vertically or horizontally. Defining a maximum of parallel requests for our service is a simple yet effective way to prevent an auto scaling rule to completely drain our budget.<\/p>\n<h2 id=\"local-vs-distributed\">Local vs distributed<\/h2>\n<p>In the world of distributed systems, we must differentiate between local and distributed rate limiters implementations: <em>local<\/em> refers to solutions which are self-contained on a single service instance or replica, thus having a local state. With <em>distributed<\/em> instead we mean a component whose state is remote and shared across different service units, independent from each other.<\/p>\n<figure>\n  <img src=\"\/assets\/images\/posts\/2023-01-15_implementing-a-rate-limiter-for-our-api-in-rust\/rate_limiter_local.png\" alt=\"Local rate limiter\">\n  <figcaption>Local state rate limiters<\/figcaption>\n<\/figure>\n<p>Local rate limiters are usually simpler to implement: their state is kept in the local, ephemeral memory of a specif service instance.<\/p>\n<p>As each rate limiter instance is isolated, we don't have to care about synchronizing accesses\/changes to the internal state across multiple services, but <em>just<\/em> across multiple threads\/listeners. Choosing a language that offers a strong support for working in a multi threaded context (like Rust, for instance) is already half the battle.<\/p>\n<p>Local\/in-memory rate limiters might be a good fit if:<\/p>\n<ul>\n<li>In the unlikely case you run a single instance of your service;<\/li>\n<li>You have some sort of session stickiness in place that lets you route requests coming from the same origin to the same instances;<\/li>\n<li>You're mostly concerned with protecting your infrastructure and keep control of your cloud provider billing; You don't care much if one of your customers exceeds its service quotas\/limits;<\/li>\n<li>You're doing a <em>POC<\/em> and you want to start simple and progressively improve;<\/li>\n<\/ul>\n<p>Distributed rate limiters are instead a different beast to tame. First of all, unlike local ones, they have to store their state in a shared cache or a low-latency storage system, like a NoSQL database.<\/p>\n<p>As such, distributed rate limiters usually come with an extra provisioning effort as well as an operational and hosting cost. Moreover, compared to in-memory solutions, they're way more exposed to the likelihood of race conditions, as the same origins\/clients can fire multiple simultaneous requests to different servers which are internally competing for the same resources. Locking, either pessimistic or optimistic, or both combined, is an opinionated approach mitigate\/solve this kind of issues.<\/p>\n<p>That being said, distributed rate limiters are a much better fit for proper service quotas checks, and generally in service oriented\/microservices architectures, usually made of ephemeral, stateless and round-robin load balanced service instances\/pods.<\/p>\n<figure>\n  <img src=\"\/assets\/images\/posts\/2023-01-15_implementing-a-rate-limiter-for-our-api-in-rust\/rate_limiter_distributed.png\" alt=\"Distributed rate limiter\">\n  <figcaption>Remote state rate limiters<\/figcaption>\n<\/figure>\n<h2 id=\"known-algorithms\">Known algorithms<\/h2>\n<p>As far as I'm aware, there are 4 main, well-known algorithms for building a rate limiter.<\/p>\n<h3 id=\"token-bucket\">Token bucket<\/h3>\n<p>The <em>Token bucket<\/em> algorithm is probably the simplest to reason about amongst the 4 algorithms. A token bucket is a container with a prefixed size, where each token relates to a request. The bucket is refilled at a predefined, fixed rate and it can contain a number of tokens up to its size. When a request comes in, if there's at least one token, it consumes a token and executes normally. Alternatively, if there are no tokens to consume, the request is dropped.<\/p>\n<p>The biggest advantage of this algorithm is its flexibility: it's a great fit for both regular traffic and bursts of requests in short periods of time. On the flip side, the implementation of its refill mechanism might not be so simple, and its <em>bucket size<\/em> and <em>refill rate<\/em> configurations might be a bit tricky to tune.<\/p>\n<p>It's widely adopted in the industry, including by big players like <a rel=\"external\" href=\"https:\/\/stripe.com\/blog\/rate-limiters\">Stripe<\/a> or <a rel=\"external\" href=\"https:\/\/partners.deliveroo.com\/docs\/#data-api-response-codes\">Deliveroo<\/a>.<\/p>\n<h3 id=\"leaky-bucket\">Leaky bucket<\/h3>\n<p>The <em>Leaky bucket<\/em> algorithm is similar to the <em>Token bucket<\/em> one, but it executes requests at a regular rate with the help of a processing <em>FIFO<\/em> queue. When a request comes in, the algorithm checks the state of the queue: if it's full, the request is dropped, otherwise it's added at the end of the queue.<\/p>\n<p>Similarly to the previous algorithm, it comes with two parameters: <em>queue size<\/em> and <em>outflow rate<\/em>, representing respectively the maximum number of requests that a rate limiter can handle and the (constant) execution rate of our requests.<\/p>\n<p>Due to its internals, this algorithm is a perfect fit for use cases with a kind of stable rate of requests, or when you can afford processing requests at constant rate, regardless of the traffic experienced at the edges of your service. If your application has to deal with frequent and irregular burst of requests, by using a leaky bucket rate limiter you might end up dropping a non negligible amount of requests.<\/p>\n<p><a rel=\"external\" href=\"https:\/\/shopify.dev\/api\/usage\/rate-limits#the-leaky-bucket-algorithm\">Shopify<\/a> uses this algorithm on their APIs.<\/p>\n<h3 id=\"fixed-window\">Fixed window<\/h3>\n<p>Differently from the above 2 approaches, this algorithms divides the timeline into <em>fix-sized time windows<\/em> with a predefined maximum of requests supported for each of them. When a request arrives, it increments a counter representing the number of requests executed in the same window. If the updated counter is higher than the predefined threshold, the request is dropped, otherwise it can execute.<\/p>\n<p>The biggest advantages of this approach are its simplicity and its low memory footprint. On the other side, burst of traffic at the edges of two adjacent time windows can lead to the execution of more requests than the ones defined by our quotas. With the help of an example, let's imagine a window size 5 and a duration of 1 minute. As time windows are independent from each other, there's nothing that could prevent 10 requests happening between <code>17:03:58<\/code> and <code>17:04:05<\/code>. From the perspective of our algorithm such traffic would be totally legit, as we would handle five request in each of the <code>17:04<\/code> and <code>17:05<\/code> windows, but we would end up accepting 10 requests in approximately 7 seconds, which might be inconvenient if not unacceptable for our use case.<\/p>\n<figure>\n  <img src=\"\/assets\/images\/posts\/2023-01-15_implementing-a-rate-limiter-for-our-api-in-rust\/rate_limiter_overflow.png\" alt=\"Limits overflow\">\n  <figcaption>request limits overflow sample<\/figcaption>\n<\/figure>\n<h3 id=\"sliding-window\">Sliding window<\/h3>\n<p>The <em>Sliding window<\/em> algorithm tries to improve the previous approach by keeping track of the history of requests. When a request comes in, it subtracts the window duration to the current timestamp and removes all the existing requests which are older than the newly computed window. Then, it adds the new timestamp to the history of requests and finally checks whether the request can execute or not: if the counter of request, including the current one, is same or lower than the the allowed maximum, the request is accepted. Otherwise, it's dropped.<\/p>\n<figure>\n  <img src=\"\/assets\/images\/posts\/2023-01-15_implementing-a-rate-limiter-for-our-api-in-rust\/rate_limiter_sliding_window.png\" alt=\"Sliding window\">\n  <br><figcaption>Sliding window algorithm<\/figcaption>\n<\/figure>\n<p>The major drawback of this algorithm is its potential impact on memory, especially especially if you have a long window duration (eg: hours).<\/p>\n<p><a rel=\"external\" href=\"https:\/\/blog.cloudflare.com\/counting-things-a-lot-of-different-things\/\">Cloudflare<\/a> leverages a sliding window - though improved, memory efficient - algorithm to mitigate L7 DDoS attacks.<\/p>\n<h2 id=\"expressiveness-at-the-api-level\">Expressiveness at the API level<\/h2>\n<p>Whilst there are multiple ways to implement a rate limiter, how rate limiting errors are expressed at the API level is quite standard.<\/p>\n<p>Back in 2012 the <a rel=\"external\" href=\"https:\/\/www.ietf.org\/\">IETF<\/a> officially introduced a new <code>429<\/code> HTTP status code and a <code>Retry-in<\/code> response header precisely for this purpose.<\/p>\n<p><a rel=\"external\" href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc6585#section-4\">RFC 6585<\/a> states that<\/p>\n<blockquote>\n<p>The 429 status code indicates that the user has sent too many\nrequests in a given amount of time (\"rate limiting\").<\/p>\n<p>The response representations <strong>SHOULD<\/strong> include details explaining the condition, and <strong>MAY<\/strong> include a <code>Retry-After<\/code> header indicating how long to wait before making a new request.<\/p>\n<\/blockquote>\n<p>The combination of the HTTP status code and header makes the error very explicit for clients and helps building very accurate user experiences and\/or efficient API integrations.<\/p>\n<h1 id=\"building-a-rate-limiter\">Building a rate limiter<\/h1>\n<p>Now that we have discussed some of the theoretic fundamentals of rate limiting, we can finally get our hands dirty with some code. In this hands-on section of the article we'll see how to build a rate limiter, test it and finally use it in a sample API project.<\/p>\n<h2 id=\"our-requirements\">Our requirements<\/h2>\n<p>First and foremost, let's define some requirement for our rate limiter:<\/p>\n<ul>\n<li>It should be used on an API to keep control of incoming requests. As such, it will be a server-side component\/library;<\/li>\n<li>It should limit requests based on the IP of the caller;<\/li>\n<li>It should allow up to 5 requests per minute from the same IP address;<\/li>\n<li>Our limits should be respected regardless of how the rate limiter is hosted. In other words, our checks should be accurate even if our service is hosted on multiple service replicas at the same time;<\/li>\n<li>In case of rate limiter internal errors, it will be up to the overarching API to decide whether to allow or throttle the request.<\/li>\n<li>If a request is allowed, it should return to the users the remaining request budget. If a request is throttled, it should suggest when to retry the next request.<\/li>\n<\/ul>\n<h2 id=\"our-tech-stack\">Our tech stack<\/h2>\n<p>The fact that our rate limiter has to be distributed gives us already an important implementation constraint: in order for our requests checks to be accurate, the state of the rate limiter must be remote. Moreover, as we do not have strict service quota checks to respect, it might be sensible to start with a simple implementation.<\/p>\n<p>In its first iteration our rate limiter will be an implementation of the <em>Fixed window<\/em> algorithm and we'll leverage <a rel=\"external\" href=\"https:\/\/redis.io\/\">Redis<\/a> for our remote state management.<\/p>\n<h3 id=\"remote-state-management\">Remote state management<\/h3>\n<p>Redis is a in-memory data store, designed for low latency and bundled with a bunch of neat features that we can really leverage to keep our implementation simple and powerful at the same time.<\/p>\n<p>Thanks to its basic features, and the more advanced <code>WATCH<\/code> and <code>MULTI<\/code> commands, Redis is a very nice fit for a simple solution yet resilient to concurrency issues.<\/p>\n<h3 id=\"why-rust\">Why Rust<\/h3>\n<p>For this experiment I decided to use Rust.<\/p>\n<p>As I said in the first part of this post, picking a language that has a first-class support for concurrency is half of the job when building a rate limiter. Whilst this is very true for Rust, I can't say it was the real driver for my decision in this case: we'll solve most of the concurrency challenges with the help of Redis commands.<\/p>\n<p>I decided to go with Rust here mostly for my personal interest: I wanted to get a bit more familiar with the language as well as ecosystem and the tooling. I wanted to test my productivity with something which was outside the comfort zone provided by my current company code bases and internal libraries (Yes, <a rel=\"external\" href=\"https:\/\/www.youtube.com\/watch?v=1nKC505_uTU\">we write a lot of Rust in TrueLayer<\/a>).<\/p>\n<h3 id=\"github-repository\">Github repository<\/h3>\n<p>All code snippets below are taken from my public <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\">rate-limiting<\/a> project on Github - Feel free to contribute and\/or giving it a \u2b50\ufe0f if you like.<\/p>\n<h2 id=\"implementation-details\">Implementation details<\/h2>\n<p>Let's start by defining the basic <strike>interface<\/strike> trait of our rate limiter component.<\/p>\n<h3 id=\"our-main-trait\">Our main trait<\/h3>\n<p>Simply put, our rate limiter component should accept a request and return whether that can be executed or not.<\/p>\n<p>This is how we can model our basic functionality:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">pub trait<\/span><span style=\"color: #B392F0;\"> RateLimiter<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ Method that checks whether a request is allowed or should<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ be throttled instead.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ Returns an error if unable to check, usually due to<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ issues connecting to the underlying Redis instance.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    fn<\/span><span style=\"color: #B392F0;\"> check_request<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        &amp;<\/span><span style=\"color: #79B8FF;\">self<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>        request_identifier<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> RequestIdentifier<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    )<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #B392F0;\"> Result<\/span><span>&lt;<\/span><span style=\"color: #B392F0;\">RateLimiterResponse<\/span><span>,<\/span><span style=\"color: #B392F0;\"> RateLimiterError<\/span><span>&gt;;<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>One of the strengths of Rust is its type system. Rust <strong>does not have exceptions<\/strong> and thanks to the <a rel=\"external\" href=\"https:\/\/doc.rust-lang.org\/std\/result\/\">Result<\/a>\ntype we can really design an expressive interface for our rate limiter component.<\/p>\n<p>Our <code>check_request<\/code> function will accept an enum <code>RequestIdentifier<\/code> which can model either an IP address (with the help of the standard <a rel=\"external\" href=\"https:\/\/doc.rust-lang.org\/std\/net\/enum.IpAddr.html\">IpAddr<\/a> type) or a custom request identifier, made of a key value pair.<\/p>\n<p>When I originally wrote the code, I wanted to give some more flexibility to our rate limiter going beyond the sole IP address use case, but in the context of this article, we can completely ignore the <code>Custom<\/code> variant.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #6A737D;\">\/\/\/ Enum that represents the possible input types for our rate limiter<\/span><\/span>\n<span class=\"giallo-l\"><span>#[derive(<\/span><span style=\"color: #B392F0;\">Clone<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">pub enum<\/span><span style=\"color: #B392F0;\"> RequestIdentifier<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ An Ip address. Used when we want to rate limit requests based on<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ the Ip address from which the request was fired<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    Ip<\/span><span>(<\/span><span style=\"color: #B392F0;\">IpAddr<\/span><span>),<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ A custom identifier in a string format. Used when we want to rate<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ limit based on custom criteria, like a client identifier.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    Custom<\/span><span> { key<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> String<\/span><span>, value<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> String<\/span><span> },<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>The same <code>check_request<\/code> method will return either a <code>RateLimiterResponse<\/code> enumeration to model proper rate limiter responses or a <code>RateLimiterError<\/code> enumeration in case of internal errors while computing our limits checks.<\/p>\n<p>Each variant included in the <code>RateLimiterResponse<\/code> enum will have their own, specific fields.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">pub enum<\/span><span style=\"color: #B392F0;\"> RateLimiterResponse<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ variant for requests that are allowed<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    RequestAllowed<\/span><span>(<\/span><span style=\"color: #B392F0;\">RequestAllowed<\/span><span>),<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ variant for requests that are throttled<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    RequestThrottled<\/span><span>(<\/span><span style=\"color: #B392F0;\">RequestThrottled<\/span><span>),<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>In case of a successful request check, we want our users to know how many requests they can fire after the one just happened. We're conveying this information with the help of a <code>remaining_request_counter<\/code> field on our <code>RequestAllowed<\/code> variant.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #6A737D;\">\/\/\/ Struct for requests that are allowed by the rate limiter<\/span><\/span>\n<span class=\"giallo-l\"><span>#[derive(<\/span><span style=\"color: #B392F0;\">Debug<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">pub struct<\/span><span style=\"color: #B392F0;\"> RequestAllowed<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ the updated counter of available requests for the given ip\/custom<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ request id<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    pub<\/span><span> remaining_request_counter<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> u64<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Alternatively, if a request cannot go through because our users have already reached the maximum number of requests in the current window, we want to suggest them when\nto expect the next request to succeed again with a <code>retry_in<\/code> field of type <a rel=\"external\" href=\"https:\/\/doc.rust-lang.org\/stable\/std\/time\/struct.Duration.html\">Duration<\/a>, which can then\nbe converted to the desired time unit (usually seconds).<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #6A737D;\">\/\/\/ Struct for requests that are throttled by the rate limiter<\/span><\/span>\n<span class=\"giallo-l\"><span>#[derive(<\/span><span style=\"color: #B392F0;\">Debug<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">pub struct<\/span><span style=\"color: #B392F0;\"> RequestThrottled<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ a duration representing when the user should retry the request<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    pub<\/span><span> retry_in<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> Duration<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Finally, if we're unable to check the request limits due to an internal error, we'll let our users know with a <code>RateLimiterError<\/code> type, which can surface\na few different internal errors.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #6A737D;\">\/\/\/ Enum that represent the error potentially returned by the rate<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">\/\/\/ limiter component<\/span><\/span>\n<span class=\"giallo-l\"><span>#[derive(thiserror<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Error<\/span><span>,<\/span><span style=\"color: #B392F0;\"> Debug<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">pub enum<\/span><span style=\"color: #B392F0;\"> RateLimiterError<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>    #[error(<\/span><span style=\"color: #9ECBFF;\">&quot;Init error&quot;<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    InitError<\/span><span>(#[source]<\/span><span style=\"color: #B392F0;\"> RedisError<\/span><span>),<\/span><\/span>\n<span class=\"giallo-l\"><span>    #[error(<\/span><span style=\"color: #9ECBFF;\">&quot;Compute error&quot;<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    ComputeError<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    #[error(<\/span><span style=\"color: #9ECBFF;\">&quot;Connect error: {0}&quot;<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    IoError<\/span><span>(#[source]<\/span><span style=\"color: #B392F0;\"> RedisError<\/span><span>),<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Whilst this list is only partial, it's probably more than enough for our first iteration of the rate limiter. Note how this struct is decorated with\nshorthands included in the <a rel=\"external\" href=\"https:\/\/docs.rs\/thiserror\/latest\/thiserror\/\">thiserror<\/a> crate to improve the readability of inner errors.<\/p>\n<p>Last but not least, note how the above type and function declarations are kept very generic: we don't want to pollute our contracts with any implementation details.<\/p>\n<h3 id=\"our-rate-limiter-instance\">Our rate limiter instance<\/h3>\n<p>So far we spoke about our contracts but we haven't touched on our concrete type yet. As our rate limiter will implement the <em>Fixed window<\/em> algorithm,\nwe should expect two attributes on our main struct: our window should have a <strong>size<\/strong> and <strong>duration<\/strong>.<\/p>\n<p>Lastly, as our component will use Redis under the hood, we should probably as well expect a <strong>client<\/strong> to be included in the same rate limiter concrete type. We'll use this <a rel=\"external\" href=\"https:\/\/docs.rs\/redis\/latest\/redis\/\">redis crate<\/a> to interact with our instance.<\/p>\n<p>With all this requirement in mind we can finally define our concrete <code>FixedWindowRateLimiter<\/code> struct:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span>#[derive(<\/span><span style=\"color: #B392F0;\">Clone<\/span><span>)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">pub struct<\/span><span style=\"color: #B392F0;\"> FixedWindowRateLimiter<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ The size of the window, that is the maximum number<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ of requests that the rate limiter will allow for a time equal<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ to the _window_duration_<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    pub<\/span><span> window_size<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> u64<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ Represents how long the window should be considered valid.<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ This can be considered as the equivalent of the refill rate<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    pub<\/span><span> window_validity<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> Duration<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ The internal client that will be used to fire requests<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ against Redis<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    pub<\/span><span> redis_client<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> redis<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Client<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>To initiate our rate limiter instance we can simply go ahead with the classic struct initialization syntax in Rust, or doing something\nfancier with the help of a <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/rate-limiter-rs\/src\/builders\/fixed_window.rs\">builder pattern<\/a>. We're gonna skip these bits here, as not really relevant for what we want to build.<\/p>\n<p>Instead, we can now move to the juicy details of our specific rate limiter implementation.<\/p>\n<h3 id=\"our-algorithm\">Our algorithm<\/h3>\n<p>As mentioned, we'll leverage Redis native commands to implement our checks. Let's now try to design our algorithm, conscious\nof the <a rel=\"external\" href=\"https:\/\/redis.io\/commands\/\">Redis offering<\/a>.<\/p>\n<p>We can use a Redis key as counter for our requests: its <strong>name<\/strong> will be made of the IP address of the request, and its <strong>value<\/strong> will keep\nthe updated counter of requests received from the same IP address in the current window.<\/p>\n<p>When a request comes in we have to:<\/p>\n<ol>\n<li>Increment our key by 1. if not existing, create a key with an initial value of 0;<\/li>\n<li>Get the updated counter and\n<ol>\n<li>Block the request if the counter is above our window <strong>size<\/strong>, and suggest when to retry;<\/li>\n<li>Otherwise, let the request execute and return the updated request budget.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>To save some memory we'll also associate an expiration to our key, equals to the window <strong>duration<\/strong>, so that we can eventually clean up our state, even if no request are coming after the first one. We can place this step soon after the first instruction above.<\/p>\n<p>Notice how our implementation slightly diverges from the definition in the <a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#fixed-window\">first part of this article<\/a>:\nthe start of our window won't be bound to the beginning of a minute, but it will be equal to the timestamp of the first request received from an IP address. We're doing this mostly because we're leveraging Redis keys expirations. Only accidentally, such choice will also mitigate a bit more - not solve! - the limits overflow problem described in the <em>Fixed window<\/em> algorithm section.<\/p>\n<h4 id=\"on-the-shoulder-of-redis\">On the shoulder of Redis<\/h4>\n<p>Translated into Redis instructions, the above steps will look like this:<\/p>\n<ol>\n<li><code>INCR $key_name<\/code>: Increment <code>$key_name<\/code> by 1. If not existing, create a <code>$key_name<\/code> and set it to 0<\/li>\n<li><code>EXPIRE $key_name $window_duration_seconds NX<\/code>: associate an expiry of <code>$window_duration_seconds<\/code> to the key, if not existing<\/li>\n<li><code>TTL $key_name<\/code>: get the updated expiry of the key<\/li>\n<\/ol>\n<p>Finally, as mentioned already, one of the reasons why I picked Redis was its support for locking, to help us dealing with concurrent updates.\nTo make the above sequence resilient to race conditions, we have to basically treat that as an atomic command, and make sure we prevent concurrent modifications\nto the same key.<\/p>\n<p>We can do that with the help of <a rel=\"external\" href=\"https:\/\/redis.io\/docs\/manual\/transactions\/#:~:text=Redis%20Transactions%20allow%20the%20execution,are%20serialized%20and%20executed%20sequentially.\">transactions<\/a>!<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span>WATCH $key_name<\/span><\/span>\n<span class=\"giallo-l\"><span>MULTI<\/span><\/span>\n<span class=\"giallo-l\"><span>... our commands ...<\/span><\/span>\n<span class=\"giallo-l\"><span>EXEC<\/span><\/span>\n<span class=\"giallo-l\"><span>UNWATCH<\/span><\/span><\/code><\/pre>\n<p>We can surround our 4 statements with a <code>MULTI<\/code> \/ <code>EXEC<\/code> block to implement atomicity, and with a <code>WATCH<\/code> \/ <code>UNWATCH<\/code> guard on our key we can stop any potential concurrent modification from happen: if our key is modified after the <code>WATCH<\/code> instruction above by another process before being <code>UNWATCH<\/code>ed, the whole transaction aborts.<\/p>\n<p>Therefore, our sequence of commands will eventually look like this:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"plain\"><span class=\"giallo-l\"><span>WATCH rl:ip_192.168.224.6<\/span><\/span>\n<span class=\"giallo-l\"><span>MULTI<\/span><\/span>\n<span class=\"giallo-l\"><span>INCR rl:ip_192.168.224.6<\/span><\/span>\n<span class=\"giallo-l\"><span>EXPIRE rl:ip_192.168.224.6 60 NX<\/span><\/span>\n<span class=\"giallo-l\"><span>TTL rl:ip_192.168.224.6<\/span><\/span>\n<span class=\"giallo-l\"><span>EXEC<\/span><\/span>\n<span class=\"giallo-l\"><span>UNWATCH<\/span><\/span><\/code><\/pre>\n<p>Note how <code>rl:ip_192.168.224.6<\/code> is our key name here: the prefix says that key refers to rate limiting(we might use Redis for other things as well!) and that <code>ip<\/code> word tell us a bit more on how we have identified the request.<\/p>\n<h4 id=\"our-trait-implemented\">Our trait implemented<\/h4>\n<p>Now we just have to reproduce the same exact sequence of instruction from our trait implementation for the <code>FixedWindowRateLimiter<\/code> struct:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">impl<\/span><span style=\"color: #B392F0;\"> RateLimiter<\/span><span style=\"color: #F97583;\"> for<\/span><span style=\"color: #B392F0;\"> FixedWindowRateLimiter<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    fn<\/span><span style=\"color: #B392F0;\"> check_request<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        &amp;<\/span><span style=\"color: #79B8FF;\">self<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>        request_identifier<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> RequestIdentifier<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    )<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #B392F0;\"> Result<\/span><span>&lt;<\/span><span style=\"color: #B392F0;\">RateLimiterResponse<\/span><span>,<\/span><span style=\"color: #B392F0;\"> RateLimiterError<\/span><span>&gt; {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> key<\/span><span style=\"color: #F97583;\"> = &amp;<\/span><span style=\"color: #79B8FF;\">self<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">build_request_key<\/span><span>(request_identifier);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let mut<\/span><span> con<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #79B8FF;\"> self<\/span><span style=\"color: #F97583;\">.<\/span><span>redis_client<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">get_connection<\/span><span>()<\/span><span style=\"color: #F97583;\">?<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> (executed_request_counter, expire_in_seconds)<\/span><span style=\"color: #F97583;\">:<\/span><span> (<\/span><span style=\"color: #B392F0;\">u64<\/span><span>,<\/span><span style=\"color: #B392F0;\"> u64<\/span><span>)<\/span><span style=\"color: #F97583;\"> =<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            redis<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">transaction<\/span><span>(<\/span><span style=\"color: #F97583;\">&amp;mut<\/span><span> con,<\/span><span style=\"color: #F97583;\"> &amp;<\/span><span>[key],<\/span><span style=\"color: #F97583;\"> |<\/span><span>con, pipe<\/span><span style=\"color: #F97583;\">|<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                pipe<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">cmd<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;INCR&quot;<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">arg<\/span><span>(key)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">cmd<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;EXPIRE&quot;<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">arg<\/span><span>(key)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">arg<\/span><span>(<\/span><span style=\"color: #79B8FF;\">self<\/span><span style=\"color: #F97583;\">.<\/span><span>window_validity<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>())<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">arg<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;NX&quot;<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">ignore<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">cmd<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;TTL&quot;<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">arg<\/span><span>(key)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">query<\/span><span>(con)<\/span><\/span>\n<span class=\"giallo-l\"><span>            })<\/span><span style=\"color: #F97583;\">?<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> response<\/span><span style=\"color: #F97583;\"> = if<\/span><span> executed_request_counter<\/span><span style=\"color: #F97583;\"> &lt;=<\/span><span style=\"color: #79B8FF;\"> self<\/span><span style=\"color: #F97583;\">.<\/span><span>window_size {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            RateLimiterResponse<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">RequestAllowed<\/span><span>(<\/span><span style=\"color: #B392F0;\">RequestAllowed<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                remaining_request_counter<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #79B8FF;\"> self<\/span><span style=\"color: #F97583;\">.<\/span><span>window_size <\/span><span style=\"color: #F97583;\">-<\/span><span> executed_request_counter<\/span><span style=\"color: #F97583;\"> as<\/span><span style=\"color: #B392F0;\"> u64<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>            })<\/span><\/span>\n<span class=\"giallo-l\"><span>        }<\/span><span style=\"color: #F97583;\"> else<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            RateLimiterResponse<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">RequestThrottled<\/span><span>(<\/span><span style=\"color: #B392F0;\">RequestThrottled<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                retry_in<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> Duration<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">from_secs<\/span><span>(expire_in_seconds),<\/span><\/span>\n<span class=\"giallo-l\"><span>            })<\/span><\/span>\n<span class=\"giallo-l\"><span>        };<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        Ok<\/span><span>(response)<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Our redis crate syntax is actually self explanatory:<\/p>\n<ol>\n<li>First of all we get a connection to our Redis instance;<\/li>\n<li>We create a transaction using the acquired connection and key;<\/li>\n<li>We define and execute our sequence of commands with the help of the <code>query<\/code> utility;<\/li>\n<li>We finally return the rate limiter response object to the caller, including the details which will help us satisfying our initial <a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#our-requirements\">requirements<\/a>.<\/li>\n<\/ol>\n<p>I did not talk much about the <code>build_request_key<\/code> function, but its responsibility should be clear \ud83d\ude03. The interested reader can find its <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/rate-limiter-rs\/src\/lib.rs#L19\">default implementation<\/a> on the <code>RateLimiter<\/code> trait.<\/p>\n<h4 id=\"the-beauty-of-error-handling-in-rust\">The beauty of error handling in Rust<\/h4>\n<p>The bit I'd highlight here instead is the use of the magic <code>?<\/code> operator, as it's a chance to see how error handling is yet another strength of Rust.<\/p>\n<p>Without being verbose at all in this method, we're basically asking the compiler to propagate upstream any <code>RedisError<\/code> returned by the <a rel=\"external\" href=\"https:\/\/docs.rs\/redis\/latest\/redis\/struct.RedisError.html\">crate we depend upon<\/a>. But it's not just that! Our method signature states that it will return a <code>RateLimiterError<\/code> type (described above) in case of any error.<\/p>\n<p>So... What kind of magic is happening here ?<\/p>\n<p>In order for our code to compile, the compiler has to know how to map a <code>RedisError<\/code> into a <code>RateLimiterError<\/code>. Without any information about this conversion, the compiler will reject building:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"shellscript\"><span class=\"giallo-l\"><span style=\"color: #B392F0;\">$<\/span><span style=\"color: #9ECBFF;\"> cargo build<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">~&gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">   ...<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">   Compiling<\/span><span style=\"color: #9ECBFF;\"> rate-limiter-rs v0.1.0<\/span><span> (\/Users\/andrea.dilisio\/Documents\/Personal\/Projects\/rate-limiting\/rate-limiter-rs)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">error[E0277]:<\/span><span style=\"color: #9ECBFF;\"> `<\/span><span style=\"color: #F97583;\">?<\/span><span style=\"color: #9ECBFF;\">`<\/span><span style=\"color: #B392F0;\"> couldn&#39;t convert the error to `RateLimiterError`<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">  --&gt; src\/builders\/fixed_window.rs:60:15<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">   |<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">60 |             })?;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">   |               ^ the trait `From&lt;RedisError&gt;` is not implemented for `RateLimiterError`<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">   |<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">   = note: the question mark operation (`?`) implicitly performs a conversion on the error value using the `From` trait<\/span><\/span>\n<span class=\"giallo-l\"><\/span><\/code><\/pre>\n<p>So, as suggested by the compiler itself \ud83d\ude32, we have to implement the <code>From&lt;RedisError&gt;<\/code> trait for our <code>RateLimiter<\/code> error:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">impl<\/span><span style=\"color: #B392F0;\"> From<\/span><span>&lt;<\/span><span style=\"color: #B392F0;\">RedisError<\/span><span>&gt;<\/span><span style=\"color: #F97583;\"> for<\/span><span style=\"color: #B392F0;\"> RateLimiterError<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    fn<\/span><span style=\"color: #B392F0;\"> from<\/span><span>(redis_error<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> RedisError<\/span><span>)<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #79B8FF;\"> Self<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        match<\/span><span> redis_error<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">kind<\/span><span>() {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            redis<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">ErrorKind<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">InvalidClientConfig<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">              RateLimiterError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">InitError<\/span><span>(redis_error),<\/span><\/span>\n<span class=\"giallo-l\"><span>            _<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span style=\"color: #B392F0;\"> RateLimiterError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">IoError<\/span><span>(redis_error),<\/span><\/span>\n<span class=\"giallo-l\"><span>        }<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>With the above hint the compiler knows how to finally map a <code>RedisError<\/code> into one of our custom error variants. We could be way more accurate in discriminating <code>RedisError<\/code>s in the <code>from<\/code> function, but we're happy with the <code>_<\/code> catch-all in this first version of our library.<\/p>\n<p>Notice the verbosity the compiler in this case. At first you might get bored of the compiler due to its pedantry, but its output is actually one of the most powerful way of learning and getting familiar with Rust \ud83d\ude00. And <code>cargo<\/code> is surely one of the most accurate and complete build tool \/ package manager I've ever worked with.<\/p>\n<h2 id=\"testing-our-library\">Testing our library<\/h2>\n<p>Now that we went through the main implementation details of our library, we can approach some tests. I've skipped some information about how we could structure the project but the interested can jump on the <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/tree\/main\/rate-limiter-rs\">source code<\/a> available of my Github to clear any open point.<\/p>\n<p>To be comfortable using our library, let's define a few tests to:<\/p>\n<ul>\n<li>make sure that the rate limiting algorithm actually works \ud83d\ude05<\/li>\n<li>verify that error scenarios are handled properly<\/li>\n<\/ul>\n<p>Let's start by checking that in case of connection errors to a Redis instance, our <code>check_request<\/code> function returns a proper <code>RateLimiterError<\/code>. I think it's easier to start with, as we're testing an interim step in our algorithm. Moreover it comes handy, as we did not talk yet about how booting a Redis instance locally. We'll touch that later.<\/p>\n<p>One of the features of testing in Rust is that you can include your test functions on the same file where the productive code is defined. Moreover, with the help of a special <code>#[cfg(test)]<\/code> directive we can make sure our tests won't be compiled into our library when released.<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span>#[cfg(test)]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">mod<\/span><span style=\"color: #B392F0;\"> test<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>  #[test]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">  fn<\/span><span style=\"color: #B392F0;\"> should_yield_a_connection_error<\/span><span>() {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        \/\/arrange<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> rate_limiter<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> RateLimiterFactory<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">fixed_window<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">with_redis_settings<\/span><span>(<\/span><span style=\"color: #B392F0;\">RedisSettings<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                host<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #9ECBFF;\"> &quot;redis&quot;<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to_string<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span>                port<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #79B8FF;\"> 6379<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>            })<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">build<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">unwrap<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> ip<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> generate_random_ip<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> request_identifier<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> RequestIdentifier<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Ip<\/span><span>(ip);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        \/\/act<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> res<\/span><span style=\"color: #F97583;\"> =<\/span><span> rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">check_request<\/span><span>(request_identifier);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        \/\/assert<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        assert!<\/span><span>(res<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">is_err<\/span><span>());<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        assert!<\/span><span>(<\/span><span style=\"color: #B392F0;\">matches!<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>            res<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">unwrap_err<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            RateLimiterError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">IoError<\/span><span>(<\/span><span style=\"color: #B392F0;\">redis<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">RedisError<\/span><span> {<\/span><span style=\"color: #F97583;\"> ..<\/span><span> })<\/span><\/span>\n<span class=\"giallo-l\"><span>        ))<\/span><\/span>\n<span class=\"giallo-l\"><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">  \/\/ utility test method to generate random IP addresses<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">  fn<\/span><span style=\"color: #B392F0;\"> generate_random_ip<\/span><span>()<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #B392F0;\"> IpAddr<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let mut<\/span><span> rng<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> rand<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">thread_rng<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        IpAddr<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">V4<\/span><span>(<\/span><span style=\"color: #B392F0;\">Ipv4Addr<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">new<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>            rng<\/span><span style=\"color: #F97583;\">.gen<\/span><span>(), rng<\/span><span style=\"color: #F97583;\">.gen<\/span><span>(), rng<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">gen<\/span><span>(), rng<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">gen<\/span><span>()))<\/span><\/span>\n<span class=\"giallo-l\"><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Let's test our assertion again <code>cargo test<\/code> and verify that our test passes.<\/p>\n<p>Let's now test our <code>check_request<\/code> behavior with a Redis instance available. We can use the basic, official Docker image for this purpose:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"shellscript\"><span class=\"giallo-l\"><span style=\"color: #B392F0;\">docker<\/span><span style=\"color: #9ECBFF;\"> run<\/span><span style=\"color: #79B8FF;\"> -p<\/span><span style=\"color: #9ECBFF;\"> 6379:6379<\/span><span style=\"color: #79B8FF;\"> --name<\/span><span style=\"color: #9ECBFF;\"> redis<\/span><span style=\"color: #79B8FF;\"> -d<\/span><span style=\"color: #9ECBFF;\"> redis<\/span><\/span><\/code><\/pre>\n<p>Then, on the same <code>.rs<\/code> file as before, let's include a new test for the main algorithm behavior:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span>#[test]<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">fn<\/span><span style=\"color: #B392F0;\"> should_check_request_eligibility<\/span><span>() {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/arrange<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> window_size<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #79B8FF;\"> 5<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> window_duration<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> Duration<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">from_secs<\/span><span>(<\/span><span style=\"color: #79B8FF;\">60<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> rate_limiter<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> RateLimiterFactory<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">fixed_window<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">with_window_size<\/span><span>(window_size)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">with_window_duration<\/span><span>(window_duration)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">build<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">unwrap<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> request_identifier<\/span><span style=\"color: #F97583;\"> =<\/span><span> <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        RequestIdentifier<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Ip<\/span><span>(<\/span><span style=\"color: #B392F0;\">generate_random_ip<\/span><span>());<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    for<\/span><span> n<\/span><span style=\"color: #F97583;\"> in<\/span><span style=\"color: #79B8FF;\"> 1<\/span><span style=\"color: #F97583;\">..=<\/span><span style=\"color: #79B8FF;\">2<\/span><span style=\"color: #F97583;\"> *<\/span><span> window_size {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">        \/\/act<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> res<\/span><span style=\"color: #F97583;\"> =<\/span><span> rate_limiter<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">check_request<\/span><span>(request_identifier<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">clone<\/span><span>())<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">unwrap<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        if<\/span><span> n<\/span><span style=\"color: #F97583;\"> &lt;=<\/span><span> window_size {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            let<\/span><span> allowed_res<\/span><span style=\"color: #F97583;\"> =<\/span><span> res<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_allowed<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            assert_eq!<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>                allowed_res<\/span><span style=\"color: #F97583;\">.<\/span><span>remaining_request_counter,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                cmp<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">max<\/span><span>(<\/span><span style=\"color: #79B8FF;\">0<\/span><span>, window_size<\/span><span style=\"color: #F97583;\"> as<\/span><span style=\"color: #B392F0;\"> i64<\/span><span style=\"color: #F97583;\"> -<\/span><span> n<\/span><span style=\"color: #F97583;\"> as<\/span><span style=\"color: #B392F0;\"> i64<\/span><span>)<\/span><span style=\"color: #F97583;\"> as<\/span><span style=\"color: #B392F0;\"> u64<\/span><\/span>\n<span class=\"giallo-l\"><span>            )<\/span><\/span>\n<span class=\"giallo-l\"><span>        }<\/span><span style=\"color: #F97583;\"> else<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            let<\/span><span> tolerance_secs<\/span><span style=\"color: #F97583;\"> =<\/span><span> window_duration<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>()<\/span><span style=\"color: #F97583;\"> *<\/span><span style=\"color: #79B8FF;\"> 5<\/span><span style=\"color: #F97583;\"> \/<\/span><span style=\"color: #79B8FF;\"> 100<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            let<\/span><span> throttled_res<\/span><span style=\"color: #F97583;\"> =<\/span><span> res<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_throttled<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            let<\/span><span> retry_in_secs<\/span><span style=\"color: #F97583;\"> =<\/span><span> throttled_res<\/span><span style=\"color: #F97583;\">.<\/span><span>retry_in<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            assert!<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>                retry_in_secs<\/span><span style=\"color: #F97583;\"> &gt;<\/span><span style=\"color: #79B8FF;\"> 0<\/span><span style=\"color: #F97583;\"> &amp;&amp;<\/span><span> retry_in_secs <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    &lt;=<\/span><span> window_duration<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                &quot;retry in is not in valid range&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>            );<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            assert!<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>                window_duration<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>() <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    -<\/span><span> throttled_res<\/span><span style=\"color: #F97583;\">.<\/span><span>retry_in<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>() <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    &lt;=<\/span><span> tolerance_secs,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                &quot;retry_in suggestion is greater than tolerance of {0}s&quot;<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>                tolerance_secs<\/span><\/span>\n<span class=\"giallo-l\"><span>            )<\/span><\/span>\n<span class=\"giallo-l\"><span>        }<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Let's review the contents on the above test:<\/p>\n<ul>\n<li>in the <code>\/\/arrange<\/code> section we create an instance of a rate limiter\nallowing a maximum of 5 requests from the same IP address during a 1 minute time window;<\/li>\n<li>we then have a loop of 10 iterations (quite arbitrary number, twice as <code>window_size<\/code>) where\n<ul>\n<li>we call our main <code>check_request<\/code> always with the same <code>request_identifier<\/code>;<\/li>\n<li>we assert on the returned <code>RateLimiterResponse<\/code> object;<\/li>\n<\/ul>\n<\/li>\n<li>in the first five iterations we call an internal <code>as_allowed()<\/code> utility (available during tests only!)\nwhich verifies (panic otherwise) that the returned variant is of type <code>RequestAllowed<\/code>. Moreover we assert\non the outstanding request budget;<\/li>\n<li>Similarly, in the last five iterations we verify that the returned variant is of type <code>RequestThrottled<\/code> and that the <code>retry_in<\/code> field\nmatches our expectations.<\/li>\n<\/ul>\n<p>By running this test and verifying that all our tests pass, we can be pretty comfortable with our first rate limiter implementation and we can\nthink of finally using it on one of our existing projects.<\/p>\n<p>The interested reader can spy on a few other test improvements which I included in the original source code:<\/p>\n<ul>\n<li>parametrized tests with the help of the <a rel=\"external\" href=\"https:\/\/github.com\/la10736\/rstest\">rstest<\/a> framework;<\/li>\n<li>few other tests on <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/rate-limiter-rs\/src\/builders\/fixed_window.rs#L71\">builders<\/a> and other internal methods used by the library;<\/li>\n<li>cargo <a rel=\"external\" href=\"https:\/\/nexte.st\/\">nextest<\/a> extension for an improved test performances and readability, both locally and at CI time.<\/li>\n<\/ul>\n<h1 id=\"using-it\">Using it<\/h1>\n<p>We can finally approach the last part of this blog post, where we'll see our rate limiter in action.<\/p>\n<h2 id=\"our-pilot-project\">Our pilot project<\/h2>\n<p>For this demo I'll use a very basic API project built on the <a rel=\"external\" href=\"https:\/\/actix.rs\/\">Actix<\/a> web framework.\nOur pilot project, namely <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/tree\/main\/carbon-intensity-api\">carbon-intensity-api<\/a>, has just a couple of endpoints:<\/p>\n<ul>\n<li>a <code>GET \/health_check<\/code> endpoint, that returns <code>200<\/code> whenever the API is up and running;<\/li>\n<li>a <code>GET \/carbon\/intensity<\/code> endpoint, returning an indication (currently fake) of the <a rel=\"external\" href=\"https:\/\/www.nationalgrideso.com\/future-energy\/net-zero-explained\/what-carbon-intensity\">carbon intensity<\/a> at the time the request was fired.<\/li>\n<\/ul>\n<h3 id=\"building-blocks\">Building blocks<\/h3>\n<p>Our project structures is simple as this:<\/p>\n<ul>\n<li>an <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/carbon-intensity-api\/src\/application.rs\">application<\/a> unit, that models the API instance;<\/li>\n<li>the usual library and binary files, very common in Rust projects;<\/li>\n<li>a <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/carbon-intensity-api\/src\/settings.rs\">settings<\/a> class, to give our API some degrees of configurability;<\/li>\n<li>a <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/tree\/main\/carbon-intensity-api\/src\/routes\">routes<\/a> directory, including our two above mentioned endpoint definitions.<\/li>\n<\/ul>\n<p>It would be pointless and counterproductive \ud83d\ude05 to rate limit the <em>health_check<\/em> endpoint. Let's see how we can easily narrow our focus on the carbon intensity endpoint only.<\/p>\n<h2 id=\"actix-middlewares-a-tldr\"><em>Actix<\/em> middlewares, a <code>TLDR;<\/code><\/h2>\n<p>To add rate limiting to our API we'll introduce a <a rel=\"external\" href=\"https:\/\/actix.rs\/docs\/middleware\">middleware<\/a>. Conceptually, a middleware is a component which helps us adding some extra behavior to an existing endpoint, without modifying the endpoint code per se. To quote Actix docs:<\/p>\n<blockquote>\n<p>Typically, middleware is involved in the following actions:<\/p>\n<ul>\n<li>Pre-process the Request<\/li>\n<li>Post-process a Response<\/li>\n<li>Modify application state<\/li>\n<li>Access external services (redis, logging, sessions)<\/li>\n<\/ul>\n<\/blockquote>\n<p>Hence, a perfect fit for our rate limiter checks.<\/p>\n<p>Creating and registering a custom middleware in <em>Actix<\/em> is probably not the best nor the simplest piece of code you'll write, but it's probably the safest option to choose, as it's a way to plug extra behavior to your project without changing its code too much nor polluting the business logic with undesired technical, framework related complexity.<\/p>\n<p>I'm not going to discuss the details of how you could write a middleware: for the purpose of this post it's enough to know that the rate limiter library that we've built above will be invoked in the context of a <code>call<\/code> function with the below signature<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">fn<\/span><span style=\"color: #B392F0;\"> call<\/span><span>(<\/span><span style=\"color: #F97583;\">&amp;<\/span><span style=\"color: #79B8FF;\">self<\/span><span>, req<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> ServiceRequest<\/span><span>)<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #79B8FF;\"> Self<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Future<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    ...<\/span><span> custom logic<\/span><span style=\"color: #F97583;\">...<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>And that our rate limiter instance will be an attribute of the same middleware object.<\/p>\n<p>If you're curious about the Nitty Gritty details of an Actix middleware codebase, <a rel=\"external\" href=\"https:\/\/imfeld.dev\/writing\/actix-web-middleware\">this article<\/a> will surely help you out.<\/p>\n<h2 id=\"injecting-our-rate-limiter\">Injecting our rate limiter<\/h2>\n<p>So, let's see our invocation now:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">fn<\/span><span style=\"color: #B392F0;\"> call<\/span><span>(<\/span><span style=\"color: #F97583;\">&amp;<\/span><span style=\"color: #79B8FF;\">self<\/span><span>, req<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> ServiceRequest<\/span><span>)<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #79B8FF;\"> Self<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Future<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">let<\/span><span> service<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #79B8FF;\"> self<\/span><span style=\"color: #F97583;\">.<\/span><span>service<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">clone<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">let<\/span><span> rate_limiter<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #79B8FF;\"> self<\/span><span style=\"color: #F97583;\">.<\/span><span>rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">clone<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">  async move<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> ip_address<\/span><span style=\"color: #F97583;\"> =<\/span><span> req<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">connection_info<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">realip_remote_addr<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">ok_or_else<\/span><span>(<\/span><span style=\"color: #F97583;\">||<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            ApiError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">InvalidRequest<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                &quot;Missing IP address!&quot;<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to_string<\/span><span>()))<\/span><span style=\"color: #F97583;\">?<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">parse<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">map_err<\/span><span>(<\/span><span style=\"color: #F97583;\">|<\/span><span>e<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> AddrParseError<\/span><span style=\"color: #F97583;\">|<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            ApiError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Internal<\/span><span>(e<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to_string<\/span><span>()))<\/span><span style=\"color: #F97583;\">?<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> request_identifier<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> RequestIdentifier<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Ip<\/span><span>(ip_address);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    let<\/span><span> rate_limiter_response<\/span><span style=\"color: #F97583;\"> =<\/span><span> rate_limiter<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        .<\/span><span style=\"color: #B392F0;\">check_request<\/span><span>(request_identifier);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    return match<\/span><span> rate_limiter_response {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">      Ok<\/span><span>(response)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        return match<\/span><span> response {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            RateLimiterResponse<\/span><span style=\"color: #F97583;\">::<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">              RequestAllowed<\/span><span>(<\/span><span style=\"color: #B392F0;\">RequestAllowed<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                  remaining_request_counter,<\/span><\/span>\n<span class=\"giallo-l\"><span>            })<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">              let mut<\/span><span> inner_service_response<\/span><span style=\"color: #F97583;\"> =<\/span><\/span>\n<span class=\"giallo-l\"><span>                  service<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">call<\/span><span>(req)<\/span><span style=\"color: #F97583;\">.await?<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span>  <\/span><\/span>\n<span class=\"giallo-l\"><span>              inner_service_response<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">headers_mut<\/span><span>()<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">insert<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                  HeaderName<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">from_str<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #79B8FF;\">                      RATE_LIMITER_REMAINING_REQUEST_HTTP_HEADER_NAME<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>                  )<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                  .<\/span><span style=\"color: #B392F0;\">map_err<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                      |<\/span><span>e<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> InvalidHeaderName<\/span><span style=\"color: #F97583;\">|<\/span><span> <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                        ApiError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Internal<\/span><span>(e<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to_string<\/span><span>()),<\/span><\/span>\n<span class=\"giallo-l\"><span>                  )<\/span><span style=\"color: #F97583;\">?<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                  HeaderValue<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">from_str<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>                      remaining_request_counter<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to_string<\/span><span>()<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_str<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span>                  )<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                  .<\/span><span style=\"color: #B392F0;\">map_err<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                      |<\/span><span>e<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> InvalidHeaderValue<\/span><span style=\"color: #F97583;\">|<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                        ApiError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">Internal<\/span><span>(e<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to_string<\/span><span>()),<\/span><\/span>\n<span class=\"giallo-l\"><span>                  )<\/span><span style=\"color: #F97583;\">?<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>              );<\/span><\/span>\n<span class=\"giallo-l\"><span>  <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">              Ok<\/span><span>(inner_service_response)<\/span><\/span>\n<span class=\"giallo-l\"><span>            }<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            RateLimiterResponse<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">RequestThrottled<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                RequestThrottled<\/span><span> { retry_in })<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                    log<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">warn!<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;request throttled<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">                        for ip={}&quot;<\/span><span>, ip_address);<\/span><\/span>\n<span class=\"giallo-l\"><span>        <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    return<\/span><span style=\"color: #B392F0;\"> Err<\/span><span>(<\/span><span style=\"color: #B392F0;\">ApiError<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">RequestThrottled<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                        retry_after_seconds<\/span><span style=\"color: #F97583;\">:<\/span><span> retry_in<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">as_secs<\/span><span>(),<\/span><\/span>\n<span class=\"giallo-l\"><span>                    }<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                    .<\/span><span style=\"color: #B392F0;\">into<\/span><span>());<\/span><\/span>\n<span class=\"giallo-l\"><span>            }<\/span><\/span>\n<span class=\"giallo-l\"><span>        };<\/span><\/span>\n<span class=\"giallo-l\"><span>      }<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">      Err<\/span><span>(_err)<\/span><span style=\"color: #F97583;\"> =&gt;<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        log<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">warn!<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;unable to check rate limit for request coming<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">            from ip={}. Skipping validation&quot;<\/span><span>, ip_address);<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        Ok<\/span><span>(service<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">call<\/span><span>(req)<\/span><span style=\"color: #F97583;\">.await?<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span>      }<\/span><\/span>\n<span class=\"giallo-l\"><span>    };<\/span><\/span>\n<span class=\"giallo-l\"><span>  }<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">  .<\/span><span style=\"color: #B392F0;\">boxed_local<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>Let's go through the 3 main steps of the above code snippet:<\/p>\n<ol>\n<li>We get the request IP address, relying on the <a rel=\"external\" href=\"https:\/\/docs.rs\/actix-web\/3.0.0-alpha.3\/actix_web\/dev\/struct.ConnectionInfo.html#method.realip_remote_addr\">realip_remote_addr<\/a> utility\noffered by Actix. In the unlikely case we're not able to identity the IP, we return an error;<\/li>\n<li>We invoke the rate limiter component. Notice the use of the <code>self<\/code> keyword: the rate limiter is part of our rate limiter instance;<\/li>\n<li>We match the response of the library and based on the returned response variant we decide what to do next:\n<ul>\n<li>if <code>RequestAllowed<\/code> we invoke the carbon intensity endpoint with the <code>service.call(req)<\/code> syntax; Worth highlighting the fact that the target service invocation is completely generic: this will give us the ability to potentially use the same middleware on different endpoints without changing the code!<\/li>\n<li>if <code>RequestThrottled<\/code> we avoid calling the internal endpoint and we build an API error object;<\/li>\n<li>in case of errors returned by the rate limiter library, we log a warning message and we move on with the carbon intensity endpoint invocation, as if the request was allowed.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h2 id=\"configuring-it\">Configuring it<\/h2>\n<p>Finally, let's see how we can configure our rate limiter instance at boot time in our project:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"rust\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">impl<\/span><span style=\"color: #B392F0;\"> Application<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #6A737D;\">    \/\/\/ Builds the main app entrypoint<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    pub fn<\/span><span style=\"color: #B392F0;\"> build<\/span><span>(settings<\/span><span style=\"color: #F97583;\">:<\/span><span style=\"color: #B392F0;\"> AppSettings<\/span><span>)<\/span><span style=\"color: #F97583;\"> -&gt;<\/span><span style=\"color: #79B8FF;\"> Self<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> rate_limiter<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> RateLimiterFactory<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">fixed_window<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">with_window_size<\/span><span>(settings<\/span><span style=\"color: #F97583;\">.<\/span><span>rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span>window_size)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">with_window_duration<\/span><span>(<\/span><span style=\"color: #B392F0;\">Duration<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">from_secs<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>                settings<\/span><span style=\"color: #F97583;\">.<\/span><span>rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span>window_duration_seconds,<\/span><\/span>\n<span class=\"giallo-l\"><span>            ))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">with_redis_settings<\/span><span>(<\/span><span style=\"color: #B392F0;\">RedisSettings<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>                host<\/span><span style=\"color: #F97583;\">:<\/span><span> settings<\/span><span style=\"color: #F97583;\">.<\/span><span>rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span>redis_server<\/span><span style=\"color: #F97583;\">.<\/span><span>host,<\/span><\/span>\n<span class=\"giallo-l\"><span>                port<\/span><span style=\"color: #F97583;\">:<\/span><span> settings<\/span><span style=\"color: #F97583;\">.<\/span><span>rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span>redis_server<\/span><span style=\"color: #F97583;\">.<\/span><span>port,<\/span><\/span>\n<span class=\"giallo-l\"><span>            })<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">build<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">expect<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;unable to setup rate limiter component&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> server<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> HttpServer<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">new<\/span><span>(<\/span><span style=\"color: #F97583;\">move ||<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">            App<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">new<\/span><span>()<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                .<\/span><span style=\"color: #B392F0;\">wrap<\/span><span>(<\/span><span style=\"color: #B392F0;\">Logger<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">default<\/span><span>())<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                .<\/span><span style=\"color: #B392F0;\">route<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;\/health_check&quot;<\/span><span>,<\/span><span style=\"color: #B392F0;\"> web<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">get<\/span><span>()<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to<\/span><span>(health_check))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                .<\/span><span style=\"color: #B392F0;\">service<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">                    web<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">scope<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;\/carbon\/intensity&quot;<\/span><span>)<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                        .<\/span><span style=\"color: #B392F0;\">wrap<\/span><span>(<\/span><span style=\"color: #B392F0;\">RateLimiterMiddlewareFactory<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                            ::<\/span><span style=\"color: #B392F0;\">with_rate_limiter<\/span><span>(<\/span><span style=\"color: #B392F0;\">Rc<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">new<\/span><span>(<\/span><\/span>\n<span class=\"giallo-l\"><span>                                rate_limiter<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">clone<\/span><span>())<\/span><\/span>\n<span class=\"giallo-l\"><span>                            ))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">                        .<\/span><span style=\"color: #B392F0;\">route<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;&quot;<\/span><span>,<\/span><span style=\"color: #B392F0;\"> web<\/span><span style=\"color: #F97583;\">::<\/span><span style=\"color: #B392F0;\">get<\/span><span>()<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">to<\/span><span>(get_intensity)),<\/span><\/span>\n<span class=\"giallo-l\"><span>                )<\/span><\/span>\n<span class=\"giallo-l\"><span>        });<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> actix_server<\/span><span style=\"color: #F97583;\"> =<\/span><span> server<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">bind<\/span><span>((settings<\/span><span style=\"color: #F97583;\">.<\/span><span>http_server<\/span><span style=\"color: #F97583;\">.<\/span><span>host, settings<\/span><span style=\"color: #F97583;\">.<\/span><span>http_server<\/span><span style=\"color: #F97583;\">.<\/span><span>port))<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            .<\/span><span style=\"color: #B392F0;\">expect<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&quot;unable to build app&quot;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> port<\/span><span style=\"color: #F97583;\"> =<\/span><span> actix_server<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">addrs<\/span><span>()[<\/span><span style=\"color: #79B8FF;\">0<\/span><span>]<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">port<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">        let<\/span><span> http_server<\/span><span style=\"color: #F97583;\"> =<\/span><span> actix_server<\/span><span style=\"color: #F97583;\">.<\/span><span style=\"color: #B392F0;\">run<\/span><span>();<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        Application<\/span><span> { http_server, port }<\/span><\/span>\n<span class=\"giallo-l\"><span>    }<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>The above code should be quite self explanatory: we're just building the rate limiter instance and registering\nit on our API. The only bit probably worth highlighting here is the use of the <a rel=\"external\" href=\"https:\/\/doc.rust-lang.org\/std\/rc\/struct.Rc.html\">reference-counting struct<\/a> <code>Rc<\/code>, which is a consequence of both how <a rel=\"external\" href=\"https:\/\/actix.rs\/docs\/application#shared-mutable-state\">Actix HttpServer works<\/a> and how we <a rel=\"external\" href=\"https:\/\/doc.rust-lang.org\/book\/ch15-04-rc.html\">can borrow references in Rust<\/a>.<\/p>\n<h2 id=\"testing-our-setup\">Testing our setup<\/h2>\n<p>Believe it or not, we're basically done with coding. At this point we just have to define a bunch of integration tests to validate our setup. I've written a few of them, which are available <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/carbon-intensity-api\/tests\/integration.rs\">on my repository<\/a>. I'll skip their review here.<\/p>\n<h3 id=\"basic-manual-tests\">Basic, manual tests<\/h3>\n<p>Instead, let's refocus on our initial requirements and validate the expressiveness of our solution with a couple of manual tests. To do so, let's start our API locally and invoke it via a browser or CLI tool:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"shellscript\"><span class=\"giallo-l\"><span style=\"color: #B392F0;\">$<\/span><span style=\"color: #9ECBFF;\"> http :8080\/carbon\/intensity<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">~&gt;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">HTTP\/1.1<\/span><span style=\"color: #79B8FF;\"> 200<\/span><span style=\"color: #9ECBFF;\"> OK<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Connection:<\/span><span style=\"color: #9ECBFF;\"> keep-alive<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Content-Length:<\/span><span style=\"color: #79B8FF;\"> 114<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Content-Type:<\/span><span style=\"color: #9ECBFF;\"> application\/json<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Date:<\/span><span style=\"color: #9ECBFF;\"> Fri,<\/span><span style=\"color: #79B8FF;\"> 03<\/span><span style=\"color: #9ECBFF;\"> Feb<\/span><span style=\"color: #79B8FF;\"> 2023<\/span><span style=\"color: #9ECBFF;\"> 14:35:58 GMT<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Server:<\/span><span style=\"color: #9ECBFF;\"> nginx\/1.23.3<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">x-remaining-request:<\/span><span style=\"color: #79B8FF;\"> 4<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>{<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    &quot;from&quot;<\/span><span style=\"color: #79B8FF;\">:<\/span><span style=\"color: #9ECBFF;\"> &quot;2018-01-20T12:00Z&quot;,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    &quot;intensity&quot;<\/span><span style=\"color: #79B8FF;\">:<\/span><span style=\"color: #9ECBFF;\"> {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        &quot;actual&quot;<\/span><span style=\"color: #79B8FF;\">:<\/span><span style=\"color: #9ECBFF;\"> 263,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        &quot;forecast&quot;<\/span><span style=\"color: #79B8FF;\">:<\/span><span style=\"color: #9ECBFF;\"> 266,<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">        &quot;index&quot;<\/span><span style=\"color: #79B8FF;\">:<\/span><span style=\"color: #9ECBFF;\"> &quot;Moderate&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">    &quot;to&quot;<\/span><span style=\"color: #79B8FF;\">:<\/span><span style=\"color: #9ECBFF;\"> &quot;2018-01-20T12:30Z&quot;<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>As expected, the first request will go through and return some (currently static) carbon intensity data. We can also see the extra information about the remaining budget of 4 request in the current window for our IP address, passed with the <code>x-remaining-request<\/code> HTTP response header: we should expect the same exact response (except for the <code>x-remaining-request<\/code> value \ud83d\ude00) for the next 4 attempts.<\/p>\n<p>If we call our endpoint for 5 more times, on the overall 6th attempt we will get a very explicit error:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"shellscript\"><span class=\"giallo-l\"><span style=\"color: #B392F0;\">$<\/span><span style=\"color: #9ECBFF;\"> http :8080\/carbon\/intensity<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">~&gt;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">HTTP\/1.1<\/span><span style=\"color: #79B8FF;\"> 429<\/span><span style=\"color: #9ECBFF;\"> Too Many Requests<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Connection:<\/span><span style=\"color: #9ECBFF;\"> keep-alive<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Content-Length:<\/span><span style=\"color: #79B8FF;\"> 22<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Date:<\/span><span style=\"color: #9ECBFF;\"> Fri,<\/span><span style=\"color: #79B8FF;\"> 03<\/span><span style=\"color: #9ECBFF;\"> Feb<\/span><span style=\"color: #79B8FF;\"> 2023<\/span><span style=\"color: #9ECBFF;\"> 14:36:19 GMT<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">Server:<\/span><span style=\"color: #9ECBFF;\"> nginx\/1.23.3<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">retry-after:<\/span><span style=\"color: #79B8FF;\"> 40<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #B392F0;\">You&#39;ve been throttled!<\/span><\/span><\/code><\/pre>\n<p>Both the standard <code>429<\/code> HTTP status and the <code>Retry-After<\/code> header should make us feel happy with the result!<\/p>\n<h3 id=\"simulating-a-burst-of-request\">Simulating a burst of request<\/h3>\n<p>Let's try to make this test a bit more robust by simulating much more requests simultaneously.<\/p>\n<h4 id=\"docker-compose-setup\">Docker compose setup<\/h4>\n<p>To do that, we'll boot a slightly more complex deployment locally with the help of <a rel=\"external\" href=\"https:\/\/docs.docker.com\/compose\/\">Docker compose<\/a>. Our deployment will be made of:<\/p>\n<ul>\n<li>1 Redis server;<\/li>\n<li>3 Carbon intensity API replicas;<\/li>\n<li>1 load balancer\/reverse proxy sitting at the edge of our setup, which will receive HTTP requests on port 8080 and forward those to the API.<\/li>\n<\/ul>\n<p>The <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/carbon-intensity-api\/compose.yaml\">compose file<\/a> for the test stack I'm talking about is available on my repository. To have that running locally, we just have to run a <code>docker-compose up<\/code> command from the <code>carbon-intensity-api<\/code> root folder.<\/p>\n<h4 id=\"generating-load-with-k6\">Generating load with <code>k6<\/code><\/h4>\n<p>With the help of <a rel=\"external\" href=\"https:\/\/k6.io\/\">k6<\/a> we'll then generate <code>500<\/code> requests to our <code>GET \/carbon\/intensity<\/code> endpoint <em>simultaneously<\/em>, using <code>25<\/code> virtual users\/threads. If our rate limiting logic works as expected, only 5 requests should succeed and we should get <code>429<\/code>s for the rest.<\/p>\n<p>Let's revive some Javascript skills now and define a <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/carbon-intensity-api\/distributed_test.js\">distributed_test.js<\/a> file:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">import<\/span><span> http<\/span><span style=\"color: #F97583;\"> from<\/span><span style=\"color: #9ECBFF;\"> &#39;k6\/http&#39;<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">import<\/span><span> { Counter }<\/span><span style=\"color: #F97583;\"> from<\/span><span style=\"color: #9ECBFF;\"> &#39;k6\/metrics&#39;<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">import<\/span><span> { check }<\/span><span style=\"color: #F97583;\"> from<\/span><span style=\"color: #9ECBFF;\"> &#39;k6&#39;<\/span><span>;<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">const<\/span><span style=\"color: #79B8FF;\"> api_responses<\/span><span style=\"color: #F97583;\"> = new<\/span><span style=\"color: #B392F0;\"> Counter<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&#39;api_responses&#39;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">export const<\/span><span style=\"color: #79B8FF;\"> options<\/span><span style=\"color: #F97583;\"> =<\/span><span> {<\/span><\/span>\n<span class=\"giallo-l\"><span>    vus:<\/span><span style=\"color: #79B8FF;\"> 25<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    iterations:<\/span><span style=\"color: #79B8FF;\"> 500<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    thresholds: {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &#39;api_responses&#39;<\/span><span>: [<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">            &#39;count == 100&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>        ],<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &#39;api_responses{status:429}&#39;<\/span><span>: [<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">            &#39;count == 495&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>        ],<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &#39;api_responses{status:200}&#39;<\/span><span>: [<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">            &#39;count == 5&#39;<\/span><\/span>\n<span class=\"giallo-l\"><span>        ],<\/span><\/span>\n<span class=\"giallo-l\"><span>    },<\/span><\/span>\n<span class=\"giallo-l\"><span>};<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">export default function<\/span><span> () {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> res<\/span><span style=\"color: #F97583;\"> =<\/span><span> http.<\/span><span style=\"color: #B392F0;\">get<\/span><span>(<\/span><span style=\"color: #9ECBFF;\">&#39;http:\/\/localhost:8080\/carbon\/intensity&#39;<\/span><span>);<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span>    api_responses.<\/span><span style=\"color: #B392F0;\">add<\/span><span>(<\/span><span style=\"color: #79B8FF;\">1<\/span><span>, {status: res.status})<\/span><\/span>\n<span class=\"giallo-l\"><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">    const<\/span><span style=\"color: #79B8FF;\"> output<\/span><span style=\"color: #F97583;\"> =<\/span><span style=\"color: #B392F0;\"> check<\/span><span>(res, {<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #9ECBFF;\">        &#39;Status code is either 200 or 429&#39;<\/span><span>: (r) <\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">            =&gt;<\/span><span> r.status<\/span><span style=\"color: #F97583;\"> ===<\/span><span style=\"color: #79B8FF;\"> 200<\/span><span style=\"color: #F97583;\"> ||<\/span><span> r.status<\/span><span style=\"color: #F97583;\"> ===<\/span><span style=\"color: #79B8FF;\"> 429<\/span><span>,<\/span><\/span>\n<span class=\"giallo-l\"><span>    });<\/span><\/span>\n<span class=\"giallo-l\"><span>}<\/span><\/span><\/code><\/pre>\n<p>By running the above script from the CLI with <code>k6 run distributed_test.js<\/code>, we should be able to validate our\nload test results:<\/p>\n<figure>\n  <img src=\"\/assets\/images\/posts\/2023-01-15_implementing-a-rate-limiter-for-our-api-in-rust\/distributed_test.png\" alt=\"Distributed test\">\n  <br>\n  <figcaption>Load test results on k6<\/figcaption>\n<\/figure>\n<p>And... yay! \ud83c\udf89<\/p>\n<h2 id=\"bonus-points-sliding-window-algorithm\">Bonus points: <em>Sliding window<\/em> algorithm<\/h2>\n<p>Before wrapping up, let's very briefly mention one last potential improvement (for now!) for our library.<\/p>\n<p><a href=\"https:\/\/adilisio.com\/posts\/implementing-a-rate-limiter-for-our-api-in-rust\/#fixed-window\">As we saw<\/a>, the biggest drawback of the <em>Fixed window<\/em> algorithm is that it can't really prevent limits overflows if burst of requests are hitting our API across two adjacent windows.<\/p>\n<p>To overcome this limitation, we could have implemented instead a <em>Sliding window<\/em> algorithm, leveraging Redis <a rel=\"external\" href=\"https:\/\/redis.io\/docs\/data-types\/sorted-sets\/#:~:text=A%20Redis%20sorted%20set%20is,Leaderboards.\">sorted sets<\/a>. Interested readers that are still alive at this point can check a sample<a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\/blob\/main\/rate-limiter-rs\/src\/rate_limiters\/sliding_window.rs\"> sliding window rate limiter implementation<\/a> on the same <code>rate-limiter-rs<\/code> project.<\/p>\n<h1 id=\"final-considerations\">Final considerations<\/h1>\n<p>We finally got to the end of this article\/experiment. The solution I've showed is far from being production ready and surely not free from bugs, but it helped me becoming more comfortable with the topic of rate limiting and a bit more proficient than I was in Rust, which were the objectives that I had in mind when I started this initiative.<\/p>\n<p>What I've realized while doing this exercise - and writing about that! - is that rate limiting is a complex and nuanced problem space, with many tradeoffs that most of the times are not even expected at the beginning of an investigation. As such, whilst there are algorithms and best practices around to drive you towards a very robust solution, it's important to understand that it's up to you to find the approach that best fits your particular case and solve your actual problems.<\/p>\n<p>To conclude, all the code snippets shown on this article are taken from my <a rel=\"external\" href=\"https:\/\/github.com\/dili91\/rate-limiting\">rate-limiting<\/a> repository, which is public and more than open for contributions \ud83d\udc4b.<\/p>\n"},{"title":"Fooled by Prometheus rate function","pubDate":"Mon, 21 Nov 2022 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/fooled-by-prometheus-rate-function\/","guid":"https:\/\/adilisio.com\/posts\/fooled-by-prometheus-rate-function\/","description":"<p>A couple of weeks ago I had to deal with an incident that affected one of the services that my team owns at my job.<\/p>\n<p>It was a minor issue that manifested with a bunch of internal dependency errors thrown by our gRPC internal endpoints. The origin of those errors was a regression introduced by one of our service dependencies, promptly solved by reverting a couple of PRs.<\/p>\n<p>What really got me thinking and inspired further actions (including this blog post), was our availability board on Grafana:<\/p>\n<br>\n<p><img src=\"\/assets\/images\/posts\/2022-11-22_fooled-by-prometheus-rate-function\/grpc_sli.png\" alt=\"gRPC SLI\" \/><\/p>\n<br>\n<p>Unexpectedly, our Service level indicators (SLI), inspired to <a rel=\"external\" href=\"https:\/\/cloud.google.com\/blog\/products\/devops-sre\/sre-fundamentals-slis-slas-and-slos\">the ones defined by Google SREs<\/a>, showed <em>all good<\/em> during the incident time window.<\/p>\n<p>At first I thought of a blip on our Prometheus queries. Our service is relatively new and these kind of issues it's not so uncommon.\nFurther investigations though showed different results.<\/p>\n<p>Let's put our error rates under the microscope.<\/p>\n<h1 id=\"error-rates-explained\">Error rates explained<\/h1>\n<p>This is how an error rate more or less looks like:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #B392F0;\">sum<\/span><span>(<\/span><span style=\"color: #B392F0;\">rate<\/span><span>(grpc_server_sli_total{service<\/span><span style=\"color: #F97583;\">=<\/span><span style=\"color: #9ECBFF;\">&quot;foo&quot;<\/span><span>, sli_error_type!=<\/span><span style=\"color: #9ECBFF;\">&quot;&quot;<\/span><span>}[1m]))<\/span><\/span><\/code><\/pre>\n<p>The main concepts to grasp here are:<\/p>\n<ul>\n<li>the metric identifier, <code>grpc_server_sli_total<\/code> in this case. As the name suggests, it refers to gRPC requests handled by our service. What's included in graph brackets are additional filters to narrow our focus on what really interests us. In this particular case we want to get non successful gRPC requests handled by our service named <em>foo<\/em>;<\/li>\n<li>The <code>rate<\/code> function is used to compute the increase of errors, rather than ever-growing counters, along with its time interval selection of 1 minute. More on <a href=\"https:\/\/adilisio.com\/posts\/fooled-by-prometheus-rate-function\/#prometheus-rate-function\">the next chapter<\/a>;<\/li>\n<li>We aggregate samples produced by different instances of our service <em>foo<\/em> with the help of the <code>sum<\/code> function. If you're familiar with Kubernetes, this will help you merging results produced by different pods belonging to the same deployment.<\/li>\n<\/ul>\n<h1 id=\"prometheus-rate-function\">Prometheus <code>rate<\/code> function<\/h1>\n<p>From <a rel=\"external\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\/#rate\">Prometheus docs<\/a>, the <code>rate<\/code> function<\/p>\n<blockquote>\n<p>calculates the per-second average rate of increase of the time series in the range vector.<\/p>\n<\/blockquote>\n<br>\n<p><img src=\"https:\/\/media.giphy.com\/media\/jpbnkneOMU9UeXXCfd\/giphy.gif\" alt=\"Thinking\" \/><\/p>\n<br>\n<p>Okay. what\u2019s a <em>range vector<\/em>, then ?<\/p>\n<blockquote>\n<p>a set of time series containing a range of data points over time for each time series.<\/p>\n<\/blockquote>\n<p>Unlike instant vectors, range vectors are represented with a list of results, belonging to a specific time interval. We get a range vector with the help of selectors made of the <mark>metric identifier + the interval<\/mark> in which we want to look for:\n<br><\/p>\n<p><img src=\"\/assets\/images\/posts\/2022-11-22_fooled-by-prometheus-rate-function\/range_vector_selector.png\" alt=\"Range vector selector\" \/><\/p>\n<br>\n<p>When I first stumbled across the time interval, I got a bit confused by that. What\u2019s the best value to use ?\nHow does that cope with the time interval usually selected on tools like in Grafana ?<\/p>\n<br>\n<p><img src=\"\/assets\/images\/posts\/2022-11-22_fooled-by-prometheus-rate-function\/grafana_window.png\" alt=\"Grafana time selector\" \/><\/p>\n<br>\n<p>After a bit of research, I found out that the accepted best practice seems to be <a rel=\"external\" href=\"https:\/\/www.robustperception.io\/what-range-should-i-use-with-rate\/\">at least 4X times the scrape interval<\/a>, to be resilient with failed scrapes.<\/p>\n<p>If you use Grafana, as of version 7.2, there's dynamic <code>$__rate_interval<\/code> variable that should help with that.<\/p>\n<h2 id=\"a-very-important-detail\">A very important detail<\/h2>\n<p>A very key bit to remember about the rate function is that <mark>it works properly with at least 2 samples of your metric: the increase from <em>null<\/em> to <em>something<\/em> is 0!<\/mark> This means that the rate function\ndoesn't play well with counters that are not initialized to 0.<\/p>\n<h1 id=\"the-problem-with-uninitialized-counters\">The problem with uninitialized counters<\/h1>\n<p>This is exactly the reason why our boards shown at the beginning of this blog post were so green. We don't initialize error counters to 0 at service startup time, but\nwe increment those counters to 1 (and more) as failed requests happen. Below is how one of our produced error metric looks like:<\/p>\n<pre class=\"giallo\" style=\"color: #E1E4E8; background-color: #24292E;\"><code data-lang=\"javascript\"><span class=\"giallo-l\"><span style=\"color: #F97583;\">...<\/span><\/span>\n<span class=\"giallo-l\"><span>grpc_server_sli_total{service<\/span><span style=\"color: #F97583;\">=<\/span><span style=\"color: #9ECBFF;\">&quot;foo&quot;<\/span><span>, grpc_method<\/span><span style=\"color: #F97583;\">=<\/span><span style=\"color: #9ECBFF;\">&quot;CreateVRPMandate&quot;<\/span><span>, sli_error_type<\/span><span style=\"color: #F97583;\">=<\/span><span style=\"color: #9ECBFF;\">&quot;internal_dependency_error&quot;<\/span><span>, sli_dependency_name<\/span><span style=\"color: #F97583;\">=<\/span><span style=\"color: #9ECBFF;\">&quot;ob_connector&quot;<\/span><span>}<\/span><\/span>\n<span class=\"giallo-l\"><span style=\"color: #F97583;\">...<\/span><\/span><\/code><\/pre>\n<p>This is potentially a problem in 2 scenarios, mostly:<\/p>\n<ol>\n<li>New services shipped to production, still with low traffic<\/li>\n<li>Whenever a new version of a service is released<\/li>\n<\/ol>\n<p>Services with low traffic are mostly impacted by this because more time is required to initialize error counters. Those are also circumstances where even a very low amount of error\nis non negligible at all.<\/p>\n<p>The second scenario is probably more interesting for a broader set of service. To be precise, it does manifest not just during planned new releases rollouts but also\nduring regular service instances rotation. This is a typical scenario in a Kubernetes cluster: whenever k8s decides to shutdown one of your pod because it's consuming too much resources for instance, you might encounter error counters blips.<\/p>\n<p>Below a few screenshots showing the impact of a new service release (<code>v9.3.59<\/code>) on error counters of type <code>internal_dependency_error<\/code>.<\/p>\n<p>At ~18:20 we can see new release kicking in:\n<br><\/p>\n<p><img src=\"\/assets\/images\/posts\/2022-11-22_fooled-by-prometheus-rate-function\/service_release.png\" alt=\"Release\" \/><\/p>\n<p>Below screenshot shows how that impacted the <code>rate<\/code> of errors value (the thin green line). We can see how that\nchanges right before and after the release moment. Moreover, we can see how we had to wait 3.5 hours to see the first <code>internal_dependency_error<\/code> impacting again our computed error rate.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2022-11-22_fooled-by-prometheus-rate-function\/rate_drops.png\" alt=\"Rate drops\" \/><\/p>\n<p>Finally, a view on error counters in isolation. We can see how each counter got reset as soon as the new pods replaced the old ones belonging to deployment <code>v9.3.58<\/code>.\nAt ~20:15 is also clear how the <code>sum<\/code> function is not really of great help in case of counters updated on different pods. Whilst we got 2 appearances of error at that time, since they happened on different pods, each pod individual rate is 0 and so their sum is also 0.<\/p>\n<p><img src=\"\/assets\/images\/posts\/2022-11-22_fooled-by-prometheus-rate-function\/error_counters.png\" alt=\"Error counters\" \/><\/p>\n<h1 id=\"potential-solutions-and-key-takeaways\">Potential solutions and key takeaways<\/h1>\n<p>The correctness of the rate function is a quite largely debated topic by the online community. In the context of my investigation I also stumbled across this <a rel=\"external\" href=\"https:\/\/github.com\/prometheus\/prometheus\/issues\/3806\">5 years old open proposal<\/a> on the Prometheus repository that, if implemented, would actually solve all the headaches.<\/p>\n<p>The most reliable and suggested advice I've found is to initialize your counters to 0. As simple as that!<\/p>\n<p>That said, this might not be the easiest thing to do in your service. Often times gRPC\/HTTP requests counters are updated with the help of stateless middlewares, as soon as those requests are handled by your service. Whilst this is very nice from the perspective of separation of concerns and helps not poisoning your business domain logic with monitoring related tasks, at the same time it makes initialization tasks like the one mentioned above a bit harder to implement.<\/p>\n<p>Depending on the specific scenario, you might think of trading a bit of readability and maintainability of your solution with an improved correctness of your metrics.<\/p>\n<p>Last but not least, it's also worth considering what's the ultimate use of your metrics: if you don't plan to wake somebody up at night for a couple of requests spoiling the rightmost decimals of your SLI indicators, then to lose a bunch data points is probably totally fine.<\/p>\n"},{"title":"Hello World","pubDate":"Thu, 22 Sep 2022 00:00:00 +0000","link":"https:\/\/adilisio.com\/posts\/hello-world\/","guid":"https:\/\/adilisio.com\/posts\/hello-world\/","description":"<p><img src=\"\/assets\/images\/posts\/2022-09-22_hello-world\/first_bday.png\" alt=\"My first birthday\" \/><\/p>\n<p><br\/><br\/><\/p>\n<p>I\u2019ve been thinking about a personal blog quite often over the last years, but it had never happened up until now,\nfor one reason or another.<\/p>\n<p>I eventually took the decision mainly because of these reasons:<\/p>\n<ul>\n<li>I <strong>think<\/strong> I\u2019ll have something worth saying, occasionally. I feel lucky for being part of a florid environment like TrueLayer and for all the other companies, clients, cultures and technology stacks I stumbled across during my career;<\/li>\n<li>I <strong>feel<\/strong> I owe the community something back for all that I learned during these years. Paying my intellectual debt will make me happy and hopefully help someone else out there on the internet;<\/li>\n<li>I <strong>want<\/strong> to improve my technical writing skills;<\/li>\n<li>I <strong>want<\/strong> to crystallize my most relevant experiences;<\/li>\n<li>Last but not least, I <strong>want<\/strong> this blog to be an excuse for learning new topics or deepen my understanding on the ones where I don't feel entirely comfortable.<\/li>\n<\/ul>\n<p>To celebrate this moment, I\u2019m using the 2 most used words in software engineering,\nand also those with which - very originally - I dipped my toes into programming long time ago (<em>almost<\/em> at the time the above photo was captured).<\/p>\n<p>I\u2019ll try to bring cool stuff on the table, from whatever intrigues me from a purely technical perspective to topics that relates more to people than compilers.\nFinding the right topics and time to write will be challenging, but I\u2019m happy to take this bet on me.<\/p>\n<p><strong>Whatever I\u2019ll end up writing on this blog should always be taken with a pinch of salt,\nand it will represent my personal point of view, which not necessarily will align with the one from my employer.<\/strong><\/p>\n<p>See you soon!<\/p>\n<p>Andrea<\/p>\n"}]}}