[{"content":"I’ve been down the UI Rabbit Hole! In one way it feels kind of good, because I have been doing some progress. On the other side, I’ve failed at documenting and writting about the process.\nPlanning and habits Even that I am doing progress, and advancing stuff, there are a lot of things in they way I work that need some improvement.\nEverything is happening in a very unorganized way: no planning, just working on what I think is the next most important thing. I am afraid that with this approach I might miss the big picture, and sometimes miss what is really the next important thing.\nAnother bad effect of not planning is that I am not sharing my time between things that I enjoy doing (mostly the technical part), and things that are not so fun for me (like product discovery, checking how to reach to people to get feedback and so one).\nI also take notes for the blog posts, but I do not find the time to convert those into a readable state. I do not expect this blog to be a very polished publication, but I don’t want either to be a collection of notes and links that make no sense at all.\nI am in the middle of reading Atomic Habits by James Clear, and even I might not religiously follow everything that is in the book I think that I can apply a lot of ideas and get a better usage of my time.\nProbably, once I finish the book I wil start experimenting with tools and processes to test if those make an impact.\nTechnical work on Frontend I do not have an idea of how I want to the app to look, so in a way, I thought that scaffolding the UI can give me an idea of what and how it should would work.\nThe “product” I want to test is sendrules.com, that is designed from a backend developer point of view, based on the problems and use cases that I’ve experienced in previous companies with notifications.\nTextashop’s “send rules” is the thing I want to build to use sendrules, but i cannot work on the backend part, until I know how the features will work in the fronted.\nInstead of doing the full App and API design, i want to work from the UI, then I will figure out the kind of data that I need from my backend, will define the APIs, and then implement the backend that will make use of sendrules.\nFor that I need a fake api that is easy to manipulate and change when I find some UI interaction that feels wrong, or not easy.\nCreating a fake API So I’ve made the easy decission of just use .json files, that can be easily imported to react, and then “mock” the calls with an async function, that will just return the api data based on an api convention.\nAlso, in order to test how the UI would work in case of an slow connection I added the option to tweak the time it takes to return the response.\nI still dont know how to remove this fake api from the production build.\nI’ve been told that the propper way to handle the app state would be to use something like: SimpleR State, that can take care of caching, and sharing state between different components. In that same line, I’ve been recommended to use the Immer Produce library But I do not want to improve parts on the technical side, until I have a clearer idea of how the app should work.\nPagination Another thing that I want to standarize for is how to paginate lists and results.\nI’ve taken a look at shopify API pagination and I am happy to find that they handle the pages with cursors instead of actual “page idx” \/ “page size”\n"pageInfo": { "hasPreviousPage": false, "hasNextPage": true, "startCursor": "eyJsYXN0X2lkIjoxNDIzOTgwNTI3NjM4LCJsYXN0X3ZhbHVlIjoiMjAyMC0wMS0yMCAxNDo0ODoxMS4wMDAwMDAifQ==", "endCursor": "eyJsYXN0X2lkIjoyMzIxMjM5MTQ2NTE4LCJsYXN0X3ZhbHVlIjoiMjAyMC0xMi0xNSAyMzowMDo0NS4wMDAwMDAifQ==" } Insteresting part is that startCursor and endCursor are base64 encoded. If we decode them we get somthing like:\n{"last_id":1423980527638,"last_value":"2020-01-20 14:48:11.000000"} However, I would not rely on that information, just in case they decide to change the formats of their cursors.\nFor the REST API pagination, shopify uses a different format of “next” and “prev”, and provides the information in a header :\nlink: "<https:\/\/{shop}.myshopify.com\/admin\/api\/{api_version}\/customers.json?page_info={page_info}&limit={limit}>; rel={next}, <https:\/\/{shop}.myshopify.com\/admin\/api\/{api_version}\/customers.json?page_info={page_info}&limit={limit}>; rel={previous}" In any case those are cursors too, so, all the components that are in polaris are ready to be used with pages that use cursors, and not page numbers.\nThis is what I have… Some unknowns Content As you can see, at the end of the video above, there is a page where I have to display the content of the notification.\nA notification can have different fields depending on the channel used to send it: for example, an SMS will have only a line, while the email, will have a subject, a text content, and HTML content.\nSo, here comes the question: should I provide a set of pre-made templates where the merchant can only tweak the colors and images to match their brand, or should let them provide the raw content for the notifications ?\nPerhaps I will need to have them both: pre-made and “advanced” ones, where the merchant can directly edit the content or import it from files.\nProviders Another question that bothers me is: should a merchant configure their own providers ? (like, setting their own SendGrid API key), or should textashop try to send emails and SMS on behalf of the merchant.\nThe first option is the safest one for textashop, as I do not need to care for the reputation if the app is used to sendd spam. However, for the merchant will have less value if has to go through the process of setting other accounts, and setting up the configuration.\nCleaning up the code I am just copy \/ pasting components and tweaking them for my usage, and the formatting ends up being a mess. So I found that prettier can easily format the .jsx code.\nDiscovering extra tools and libraries N8N Lately I’ve been hearing about more and more people using n8n to automate different kind of things. I do not have yet a use case for it, but that is perhaps because I haven’t had the chance to work with it. So I decided to give it a try, follow the n8n documentation and see if I can get ideas.\nGoing with the dockerized version In order to just play with it in a local environment I’ve created a docker-compose file, to play with it. Perhaps at some point i can move it to my development server and access it using ssh port forwarding to avoid exposing it to internet. I can setup n8n to use my already running Postgresql database\ndocker pull docker.n8n.io\/n8nio\/n8n:latest I didn’t feel like the opensource \/ free version was very polished. I managed to tunnel the ssh connection, with a valid https certificate and finally signed up\nHowever, the most basic operations that I’ve tried, to connect to google sheets, just to test them, where failing. It didn’t detect when a workflow finished executing, even I managed to add some test rows, I didn’t feel that it was worth the time investment that I’ve put into it.\nSo for now, I will not continue trying to work with it, until I find a real use case where it can really help me automate required work.\nmdxjs.com This is something that came at work, that looks super interesting in order to fix the crappy documentation that I have for sendrules.\nA markdown editor This is another library that I have been pointed at to create content from our own UI: TipTap\nLeads service I am not sure how and at what point I found this Store Leads SaaS that provides information about eCommerces. After watching the recorded demo video and it looks that the basic plan at $75 is a fair price to get a good list of leads.\nMight be interesting to sign up and to try to promote whatever app end up with.\nReferences MDX js : a static generator for react components\nStore Leads: a SaaS for market research of eCommerce stores.\n","permalink":"https:\/\/hontecillas.com\/posts\/2024_11_09_scaffolding_the_polaris_ui\/","summary":"

I’ve been down the UI Rabbit Hole! In one way it feels kind of good, because\nI have been doing some progress. On the other side, I’ve failed at documenting\nand writting about the process.<\/p>\n

Planning and habits<\/h1>\n

Even that I am doing progress, and advancing stuff, there are a lot of things\nin they way I work that need some improvement.<\/p>\n

Everything is happening in a very unorganized way: no planning, just working\non what I think is the next most important thing. I am afraid that with this approach\nI might miss the big picture, and sometimes miss what is really the next important\nthing.<\/p>","title":"Scaffolding the UI with Polaris"},{"content":"In order to provide a useful application, the most important Shopify models to understand are the Customer (that will become the recipient of the messages) and the Product (that will be the main content we want to inform a customer about).\nI need to know which fields I can map, in order to fill the nofifications required fields. There might be missing fields, or fields that might be related to other entities.\nI also need to explore use cases, and how to gather data and events to identify how the sendrules api can be leverages to achieve that, and what are the missing features that prevents to implement those use cases.\nCustomer sensitive data First of all, we have to take into account the privacy requirements (and concerns) of working with customer information that belong to the merchants (the direct textashop customers).\nFor that there is a page in shopify documentation about the three levels of requirements when dealing with that kind of information. In my case, as I am interested in email and phone number to send notifications, I go directly to Level 3.\nThinking about some use cases Initially I want to work with this basic use cases:\nNotify users when there is some product back in stock Notify users when there is a price drop or an special offer Notify users about shipment updates Apps for customer accounts One of the best features of sendrules is that it provides a way to let users select how they want to receive notifications.\nThere will be still a step in the middle, that will be the textashop backend that needs to decide, which notifications are available for customization. A merchant might want to decide that cart recovery notification will always be sent through an SMS, and that does not want to expose the selection to the end customer.\nOne cool thing in shopify is that there is a way to add UI extensions for customer accounts, It looks like there are the “old” or classic customer accounts that do not support that kind of extensions, but the interesting part is the decission of the kind of customer accounts is made by the merchant:\nYou can choose which version of customer accounts to use in your online store.\nand there is an easy way to switch from old to new customer accounts\nThe option for those accounts would be to have a “safe” defaults based on their acceptance of receiving marketing emails or not.\nFor the new ones we can provide a fine grained selection of notification preferences.\nThe interesting part is that those customer ui extension will be available during December, so in some way it feels like I could be a piooner if I am able to finish the app in .. a month !?\nSubscribe to changes All the above sounds great, but to provide some real value we might want to provide a set of rules that automatically send notifications. So I took a look at the kind of events that might be interesting.\nIn shopify, apps can get subscribed to events in shopify : there is a long list of topics that an app can be notified about: Shopify Webhooks Documentation\nSome interesting topics customers_marketing_consent\/update: this is one of the most interesting ones as we might want to update data in sendrules to avoid sending certain types of notifications to the affected customer. discounts\/create: we could create notifications based on a product that a customer added to a wishlist. domains\/create: it looks like the domain includes localization: country, locale, and alternate_locales fulfillment_events\/create: to notify a custmer about his orders ? there are also orders\/create and orders\/fulfilled events, so that would need further investigation. inventory_items\/create: to notify a user that has expressed some interest in products in a certain category ? However, there is also a product_listings\/add that might be covering that case, because a product might be created but not published on the web. Oh! and there is also products\/create ! locales\/create \/ \/update : to check if we have localized all the templates for the added locale. And closesly related with that is the markets\/create that allows to serve idfferent countries \/ regions. segments\/create : yep! this looks like a very nice feature, and something that could be also implemented in sendrules (but for now we might just send a list of users to send the notification to as a payload in sendrules) variants\/in_stock: when a variant becomes on stock, something a customer might want to know. Other interesting things … Localization I looks like localization can be extracted from the market , and we might want to identify to what market a customer belongs.\nB2B: shops that sell to other shops While browsing and reading documentation I found interesting stuff, that might be worth taking a look in the future for some ideas:\nB2B on Shopify could be interesting to target a different profile: we might want to integrate a CRM for businesses that sell to other businesses. ","permalink":"https:\/\/hontecillas.com\/posts\/2024_10_26_shopify_data_models\/","summary":"

In order to provide a useful application, the most important Shopify models\nto understand are the Customer<\/code> (that will become the recipient of\nthe messages) and the Product<\/code> (that will be the main content we want to\ninform a customer about).<\/p>\n

I need to know which fields I can map, in order to fill the nofifications\nrequired fields. There might be missing fields, or fields that might be\nrelated to other entities.<\/p>","title":"Shopify Data Models"},{"content":"SendRules is intended for backend usage and a good API is its primary interface, but Textashop is for a different audience that will require some good UI in order to interact with the notifications.\nShopify App Store Exploration SendRules for shopify (I might need to find a different name for it) should land in the Notification Apps section of the app store, so its a good place to start looking at the minimum requirements to standout.\nLets have a quick look at the page:\nAt first look there are a few details that catch my attention:\nthere are 500 apps (its a very round number, I guess if there are more than that). the first rows are all Ad apps (that might require some investigation once I have an MVP). there is a Built for Shopify badge that stands out. I want to have it! they have star rating and number of installations that are some values that I might want to get as high as possible, but is that part that will be hard to optimize for because I need people to install the app. and last and most important one: it does not look to have a criteria for having that order: I do not know how those apps land in the first page. I’ve actually used different browsers to open the same page, and, while the first row (the Ad apps row) changes, the rest remain the same (actually ‘Shopify Forms’ has a low number of ratings, and an under 4 stars rating and it is in the first place). So even there is no evidence of a sorting \/ listing criteria, lets get into the details of a couple of apps in the front page, and a couple in the last page, to see if we can get some insights.\nFirst page apps Shopify Forms (3.8 stars \/ 156 reviews): the developer is shopify itself, is free, has well produced video, a link to a demo store, and a long enough description (no Built for Shopify badge though). Launched November 1, 2022.\nOmnisend Email Marketing & SMS (4.7 stars \/ 5944 reviews): this is a third party developer “Omnisend” (it only has this app on the market), has a well produced video, a link to a demo store. It does not have a Built for Shopify badge, and if we visit the Omnisend web page, it is really polished, and we can see is supports lots of other eCommerce platforms: woocommerce, wix, shoplazza, bigcommerce… . Launched March 10, 2014.\nLast page apps Accu Web Push Notifications (0.0 stars \/ 0 reviews): the app has no video, only some screenshots of the UI (that does not look integrated into the Shopify Admin), and there is no demo store to showcase it. However if we go t its page, we find other apps like eStore2App Mobile App Builder that has received more “love”. Launched July 21, 2021.\nWDT Bundles & Volume Discounts (5.0 stars \/ 1 revies): this is the only app in the last page that has a review (that makes it a little bit suspicious), has demo store, and some screenshots that do not showcase the product at all. It looks that is actually an upsell and cross-sell app. Launched September 22, 2023.\nSome hypothesis The email \/ sms notifications app market is big but with very well stablished big companies, that have lots of clients and a very polished product. Shopify customers are not going to take advantage of selecting their own providers, they want easy solutions. They are not going to setup the api key of sendgrid, mailgun or any other provider. The actual number of installs might be relevant positioned in the first page. .. and some unknowns … Would be great to know how many installs an app has: we can see the number of reviews, but can we extrapolate the number of installs from that ?\nHow do email marketing apps send emails from the shop owners custom domains ?\n..and some conclusions. There are no built for shopify apps in the last pages: the last built for shopify app is in page 13 of 21 (well, not a very privileged position), and is Wizpop: Data collection popups.\nThe first app in the category with more than 10 reviews is on page 14 of 21, and is Appy: Sales Announcmeent Bar.\nCreating a profitable app for Shopify in this area would be very hard, but lets remember that them main goal is to test the SendRules software that will back the app.\nA secondary goal would be to learn about the process of publishing an app in the Shopify App store.\nTechnical discovery This week I spent some time trying to scaffold how to organize things on the technical side.\nThere are some decissions that have been easy to take, about the technical stack. I am going with:\nGo language for the backend (with PostgreSQL and Redis as storage services): because it what I have more recent experience. I could have tried Python, that I think is useful to iterate fast (and I could be comfortable with too), but given that I use Gon at work, I prefer to not make the context switch harder. React for frontend: mostly because the Polaris components are already there and whatever I choose I will have to learn about it. Polaris is the Design System to make your app look integrated with the shopify UI, and for that there is whole list of components, that you can be used easily with the existing react library. Several App, One Dashboard, multiple UIs There are some things that are not yet decided, because when I think about the project I can see differents parts that will be shown to the user:\nThe Embedded shopify UI: this is the interface that has too look like a “native app” inside the Shopify ecosystem. The Textashop UI: this would be a single app, where a user can access the data from different shopify integrations . The “marketing” web page: mostly static pages, that must be easy indexable, with good SEO and so one, so people can find the product. In case we want to provide a way to have a simple shop, like Bazaari, I would not go with react, but with classic server side rendered webpages. How to organize the app I’ve been thinking about separating different parts of the app in different domains. Something like:\nwww.textashop.com for the marketing web page app.textashop.com for the “external” app management dashboard api.textashop.com for accessing the api (kind of a headless access) shopify.textashop.com for the textashop ui. The alternative would be, using paths:\nwww.textashop.com for the marketing web page www.textashop.com\/app for the “external” app management dashboard www.textashop.com\/api for accessing the api (kind of a headless access) www.textashop.com\/shopify for the textashop ui. I still have to think about it.\nHow is it going ? Mostly fine. This is the third week I document the work I do in my side projects. I must say that writting takes a large amount of time for now, but I think it helps me to keep going and have small goals or things to write about here.\nAt some point, I might change the goal from “keep working on my own stuff” to actually “publish something”. But I will keep the pressure low to continue enjoying the process.\nBy, the way, if you were asking about how textashop progress is going ..\nI managed to have a small proof of concept working:\n","permalink":"https:\/\/hontecillas.com\/posts\/2024_10_20_frontend_required\/","summary":"

SendRules is intended for backend usage and a good API is its primary\ninterface, but Textashop is for a different audience that will require\nsome good UI in order to interact with the notifications.<\/p>\n

Shopify App Store Exploration<\/h1>\n

SendRules<\/a> for shopify (I might need to find\na different name for it) should land in the\nNotification Apps<\/a>\nsection of the app store,\nso its a good place to start looking at the minimum requirements to standout.<\/p>","title":"Frontend Required"},{"content":"Focus is a requirement to success. On the other side, putting all your eggs in the same basket is a risky choice. But what about having an small side project to feel the pain of using your main side project ?\nLet me introduce you Textashop Initially I planned to revisit Sendrules and see if there were any work in progress tasks, close them to have the project in a clean state and start collecting feedback on the idea.\nThat would be the right approach to build a bussiness: first validate the idea, find someone that would pay for it, and then start building it.\nHowever I refrained from doing that just for one reason: I am currently working on building the habit of working on my side projects. I don’t want to start asking people for feedback, and then fail at keeping up working on the project. My priorities are to keep performing at the maximum level at my current job, and to keep exercising regularly. This side project thing is an “extra” effort I want to do without compromising on what I am already doing.\nWith Sendrules I am targeting companies that want to have a well tested notification system, so they do not need to worry about building and maintaining it themselves.\nThe product should work for two types of product:\nThose companies that have their own users, and want to easily manage the messaging preferences, translation templates, while they remain independent of third party service providers. For example a food delivery company: they might have riders, restaurants, customers and internal backoffice users. Each of them might want to receive different kind of notifications: emails, push notifications, SMS .. whatever. But all of those users “belong” to the food delivery company.\nThose companies that have other businesses as clients, and each of their those businesses have their own users. That means have different sets of users, templates and message preferences. I would call this case a “multi-tenant system for notifications”. So, it might need to support different “accounts” in the same system. An example of company with those needs would be a project management app, that have different customers, and each customers enroll their own workers.\nObviously, a company might combine both cases.\nHaving worked on both type of companies I really believe I can build a good product covering both cases. But having a simple product that makes use of sendrules, is a good way to discover issues, or corner cases that should be covered. I would like to experience what are the frictions that might appear when using it.\nThat’s why I decided to create a small Shopify app, that uses send rules to send marketing, news, or shipment updates to customers.\nTaking a look at Shopify apps, I’ve seen that most apps share a “brand”: A single company builds several useful apps under a shared name. I guess is a good way to cross promote apps, or build a good brand that can get revenue from multiple apps. So, I did the same, and made a small webpage for the shopify apps:\nwww.textashop.com\nFirst week progress This week I’ve made a few very very small steps, but I am kind of happy I’ve started working on it. Being the first week of “sideprojecting”, I am not going to be too harsh on myself, .. actually I am not going to put much pressure on me on the progress, because this is a about “showing up” every day and enjoying the process.\nSo this week I’ve updated and put online a webpage for www.textashop.com. This is nothing fancy, just a bunch of static html files (yes, it is not rendered markdown I’ve just wrote the html). I’ve just checked there are no broken links, and have put a small description for a couple of apps: “Send Rules For Shopify” that is the one that I am going to build as a testbed for sendrules.com , and “Catalog Sync” that is just a nice idea, that I am not going to implement, but I might add content to help me be “relevant” for people searching for shopify apps.\nYou might notice that there is also a “subscribe to an email newsletter” that is not working yet. I know that an approach that could work is using and embedded google forms to collect emails. However I spent sometime programming an endpoint to collect emails (so at some point I can evolve it to filter spam requests, etc). However, it is not finished, and I am not able to collect emails yet.\nI also contracted an email provider to have a custom domain email. I want the small steps I take to be “professional”, and having a gmail account is not goint to give confidence to anyone (if you are not able to spend a few bucks and some time to have your branded email, how can I trust you to provide a decent product ?). There are several options to choose from, and I took a look at these: proton mail, zoho, mailbox.org, tuta. I thought that for what I needed mailboc and tuta where perfect valid options, and decided to just give mailbox a try (but I could use tuta for some other project).\nWell, those are not big achievements, and still there is some work to do to say prepare it for “marketing”: the custom domain for the email is not set up, there are no good descriptions for the product, i need to actually collect emails, put some analytics in place to check if I have some traffic, create a robots.txt and try to have it indexed by google, etc, etc .. a lot of non-technical things to help discover the shopify app.\nLets keep it up There is a common believe that it takes at least 21 days to form an habit, but I am not going to consider an achievement working on this for a couple more weeks.\nPerhaps in a month or two I might start writting down actual goals, and deadlines.\nFor now, .. the show must go on !\n","permalink":"https:\/\/hontecillas.com\/posts\/2024_10_13_the_side_side_project\/","summary":"

Focus is a requirement to success. On the other side, putting all your\neggs in the same basket is a risky choice. But what about having\nan small side project to feel the pain of using your main side project ?<\/p>\n

Let me introduce you Textashop<\/h1>\n

Initially I planned to revisit Sendrules<\/a> and\nsee if there were any work in progress tasks, close them to have the\nproject in a clean state and start collecting feedback on the idea.<\/p>","title":"The Side Side-Project"},{"content":"This is a “recycled” post: I intended to publish this at the beginning of the year. However, on the personal side, this 2024 has been a little bit rough, and after a loved one passed away in April, I struggled to make any efforts beyond what was required to perform my work.\nLuckily, on the professional side I had a great support from everybody at KrakenD that helped me navigate those personal issues.\nTrying to pick up some habits At some point I found that even not being in the right mood many days, I would need to make some moves to bring back some habits.\nOn June I decided to try to be more regular at exercising, and decided to hire someone to plan my trainings for running (something that I enjoyed a lot when I was in the right mood). That was great! The obvious benefit is to not need to think about what exercises to do each day. But the best part was becoming accountable: I reported every day to my coach with the output of my heart monitor. If a day I was not in the righ mood, I worked out: not because I was paying to have the planning, but because I respect the work the coach was doing for me (it would not have worked this very same way if I was actually paying for an App to do the plans). On September I was able to finish a half marathon trail for the first time. I want to keep training, and I consider this an acquired habit.\nNow I feel like I am ready to pick up some new habit: it might be a side project, or it might be learning something new. The point is if I will be able to find someone that performs the same role than my running coach.\nFor now, the initial intention is to document whatever I do in this blog. It might not be as effective as having a coach, but is worth giving it a try while I search for a way to be accountable on these other topics.\nSide Projects In the past I thought that having side projects could be a symptom of not being happy or satisfied at work: a side project might be the way to work on “more interesting things”.\nNow, I see that is not the case. I really enjoy my job: working on KrakenD I enjoy my work environment, my very supportive mates, and I really like working on features and improvements on the API Gateway, because I also keep learning new things.\nHowever I still would like to bring some of my own ideas to life (or perhaps learn things that I will not apply to my day to day job).\nOne thing that has been on pause for a long time is my sendrules.com project: a system to send localized and personalized notifications.\nI also have the idea to have a separate small project that makes use of that service: this way I can find pain points to solve, and any usability issues it can have (but that is a topic for another post).\nLearning I’ve been playing from time to time with some Rust, and some embedding programming with it, so I might want to pick it up.\nHowever, initially I will try to focus on the side projects, and depending on how things evolve I will see if I can embrace more stuff.\nI have also have some very old half-written posts, that I need to clean up and made them public (or I should delete them from this repo).\nNo more meta Up to this point, the post has been more a declaration of intentions (I should be ashamed if I do not deliver .. but that was the point, right? to become “accountable” to the blog).\nSome from this point, all that I will write to this blog will be about acomplished things and learnings rather than things I want to do.\nSo, for a minimal intruduction, here it goes “from pelican to hugo”:\nFrom Pelican to Hugo This blog will be powered by Hugo and the PaperMod theme.\nIn the past I had it working with Pelican, that is python based, but having the virtualenv not working on some python updates made me switch to Hugo. A part from that, no had no complains about it.\nI do not plan to spend a lot of time making improvements here: I just want to be able to write some “reports”, post some code, and some images. That’s all !\ntesting the blog feats Like some piece of code that is good:\npub fn run(input: Vec<String>) -> i32 { let mut total_fully_contained: i32 = 0; for ln in input { let mut sp = ln.split(","); let first = sp.next().unwrap(); let second = sp.next().unwrap(); let mut fsp = first.split("-"); let f_a = fsp.next().unwrap(); let f_b = fsp.next().unwrap(); let mut ssp = second.split("-"); let s_a = ssp.next().unwrap(); let s_b = ssp.next().unwrap(); let v: Vec<i32> = vec![ f_a.parse::<i32>().unwrap(), f_b.parse::<i32>().unwrap(), s_a.parse::<i32>().unwrap(), s_b.parse::<i32>().unwrap(), ]; if (v[0] <= v[2] && v[1] >= v[3]) || (v[2] <= v[0] && v[3] >= v[1]) { \/\/ println!("{} {} {} {}", v[0], v[1], v[2], v[3]); total_fully_contained += 1; } } return total_fully_contained; } to check the syntax hightlighting ;)\n","permalink":"https:\/\/hontecillas.com\/posts\/2024_10_05_picking_up_habits\/","summary":"

This is a “recycled” post: I intended to publish this at the beginning of the\nyear. However, on the personal side, this 2024 has been a little bit rough, and\nafter a loved one passed away in April, I struggled to make any efforts beyond\nwhat was required to perform my work.<\/p>\n

Luckily, on the professional side I had a great support from\neverybody at KrakenD<\/a> that helped me navigate\nthose personal issues.<\/p>","title":"Picking Up Habits"},{"content":"Motivation There has been more than a year since I was introduced to Go by the lead architect at the company I was working for.\nI really liked the language: strongly typed, with its very decoupled interface declarations (or protocols, whatever you prefer to call them), and specially the simple concurrency model (deeply rooted in the language with goroutines and channels).\nAfter that I did not give it more attention. But recently I’ve decided that I want to be proficient with it.\nInstead of starting my own project, I decided to try to contribute to an existing one: Krakend. It is an API Gateway. (You can look at the Krakend features in its offical webpage).\nSo, this is the worklog of a small contribution: Unlimit the whitelist depth\nIntroduction to the issue One main concern of the API Gateway is to be fast, and support a lot of connections. That was the reason why the white listing (see the response manipulation documentation ) was only supporting two level depth element selection (e.g: foo.bar, but no foo.bar.baz).\nAdding more depth means more processing, and traversing nodes to check for the field that we want to pick. But even knowing that having deep fields slows down the data processing, the feature could be useful.\nSo, if implemented, it should not perform worse than before for non deep cases.\nHow are response data handled internally in krakend ? The data in krakend is represented as tree of hashmaps, mapping a string to “anything” ( map[string]interface{})\nThe go built-in map type is implemented using a hash table.\nBut, we must take into account that:\nThe specifics of that data structure are an implementation detail of the runtime and are not specified by the language itself.\nInitial approach The first implementation involved creating a new dictionary, and adding there the new fields that matched the whitelist (creating all the intermediate nodes for the data).\nThat worked, however, it allocated a lot of new maps (one for each new node). This is a problem, because it is slower to allocate memory but also because we are adding memory pressure and more work to do to the garbage collector when that memory is not needed anymore.\nThe good approach is to actually delete the nodes that are not part of one found whitelisted element.\nRecursive implementation The initial implementation involves the use of recursive calls. I do not really know at this point how the stack frame is allocated (I’ve read somewhere that the stack is actually allocated in the heap), and how expensive is to create a stack frame for each call.\nAfter the initial implementation these are the numbers for the benchmarks:\ndhontecillas@doozer:~\/prg\/tools\/gopackages\/src\/krakend$ go test -bench=deepWhite -benchtime=3s -run xxx -v krakend\/proxy goos: linux goarch: amd64 pkg: krakend\/proxy BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:1,extraFields:2,extraSiblings:0-8 100000000 56.6 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:3,extraFields:6,extraSiblings:0-8 100000000 55.9 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:7,extraFields:14,extraSiblings:0-8 100000000 56.8 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:1,extraFields:3,extraSiblings:1-8 30000000 150 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:3,extraFields:7,extraSiblings:1-8 20000000 267 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:7,extraFields:15,extraSiblings:1-8 10000000 485 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:1,extraFields:4,extraSiblings:2-8 20000000 215 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:3,extraFields:8,extraSiblings:2-8 10000000 436 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:7,extraFields:16,extraSiblings:2-8 5000000 823 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:1,extraFields:7,extraSiblings:5-8 10000000 466 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:3,extraFields:11,extraSiblings:5-8 5000000 988 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:7,extraFields:19,extraSiblings:5-8 2000000 1932 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:1,extraFields:12,extraSiblings:10-8 5000000 1090 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:3,extraFields:16,extraSiblings:10-8 2000000 2743 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:7,extraFields:24,extraSiblings:10-8 1000000 5707 ns\/op 48 B\/op 1 allocs\/op PASS ok krakend\/proxy 84.534s Recursive implementation optimization try: Another implementation I tried did the following:\nTwo consecutive iterations:\nOne that stores the brothers at each node to be saved (using a list) Another one, that in case there is at least a brother to be saved, deletes the others. If there is no brother to be saved, we simply delegate the job of deleting the full node to the parent. Well, the description is a little vague, but is not important as the result was worse than the previous implementation:\nBenchmark results with new implementation:\ndhontecillas@doozer:~\/prg\/tools\/gopackages\/src\/krakend$ go test -bench=deepWhite -benchtime=3s -run xxx -v krakend\/proxy goos: linux goarch: amd64 pkg: krakend\/proxy BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:1,extraFields:2,extraSiblings:0-8 100000000 64.5 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:3,extraFields:6,extraSiblings:0-8 100000000 58.2 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:7,extraFields:14,extraSiblings:0-8 100000000 61.8 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:1,extraFields:3,extraSiblings:1-8 20000000 221 ns\/op 64 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:3,extraFields:7,extraSiblings:1-8 20000000 329 ns\/op 64 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:7,extraFields:15,extraSiblings:1-8 10000000 569 ns\/op 64 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:1,extraFields:4,extraSiblings:2-8 20000000 322 ns\/op 80 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:3,extraFields:8,extraSiblings:2-8 10000000 536 ns\/op 80 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:7,extraFields:16,extraSiblings:2-8 5000000 963 ns\/op 80 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:1,extraFields:7,extraSiblings:5-8 10000000 676 ns\/op 128 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:3,extraFields:11,extraSiblings:5-8 5000000 1147 ns\/op 128 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:7,extraFields:19,extraSiblings:5-8 2000000 2165 ns\/op 128 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:1,extraFields:12,extraSiblings:10-8 3000000 1617 ns\/op 208 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:3,extraFields:16,extraSiblings:10-8 1000000 3245 ns\/op 208 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:7,extraFields:24,extraSiblings:10-8 500000 6183 ns\/op 208 B\/op 2 allocs\/op PASS ok krakend\/proxy 88.827s Also, it required two allocs per call instead of one.\nRecursive implementation optimization (take two): I could save the wlKeys that should be deleted (instead of the ones that should be maintained), and only used them in case we find we need to maintain any of the whitelisted fields. That improved a little bit on the previous benchmark, but still did not beat the initial recursive version.\ndhontecillas@doozer:~\/prg\/tools\/gopackages\/src\/krakend$ go test -bench=deepWhite -benchtime=3s -run xxx -v krakend\/proxy goos: linux goarch: amd64 pkg: krakend\/proxy BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:1,extraFields:2,extraSiblings:0-8 100000000 68.2 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:3,extraFields:6,extraSiblings:0-8 100000000 70.4 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:0,depth:7,extraFields:14,extraSiblings:0-8 100000000 56.9 ns\/op 48 B\/op 1 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:1,extraFields:3,extraSiblings:1-8 20000000 223 ns\/op 64 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:3,extraFields:7,extraSiblings:1-8 20000000 347 ns\/op 64 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:1,depth:7,extraFields:15,extraSiblings:1-8 10000000 571 ns\/op 64 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:1,extraFields:4,extraSiblings:2-8 20000000 331 ns\/op 80 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:3,extraFields:8,extraSiblings:2-8 10000000 539 ns\/op 80 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:2,depth:7,extraFields:16,extraSiblings:2-8 5000000 962 ns\/op 80 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:1,extraFields:7,extraSiblings:5-8 10000000 660 ns\/op 128 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:3,extraFields:11,extraSiblings:5-8 5000000 1123 ns\/op 128 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:5,depth:7,extraFields:19,extraSiblings:5-8 2000000 2118 ns\/op 128 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:1,extraFields:12,extraSiblings:10-8 3000000 1409 ns\/op 208 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:3,extraFields:16,extraSiblings:10-8 1000000 3013 ns\/op 208 B\/op 2 allocs\/op BenchmarkEntityFormatter_deepWhitelistingFilter\/numTargets:10,depth:7,extraFields:24,extraSiblings:10-8 1000000 6021 ns\/op 208 B\/op 2 allocs\/op PASS ok krakend\/proxy 91.893s Perhap this approach would work better if we have more cases when we do not find any key from the whitelist (that I think would not happen too often).\nGo back to first recursive approach So, I decide to clean up the first implementation and go with it.\nConverting recursive calls to iterative A common undergraduate exercise in computer science is to convert any recursive algorithm into an iterative one. Could it be applied here, to avoid the recursive call cost ?\nSince we know in advance the maximum depth of the elements in the whitelist, we could construct (a single alloc ?) a list of references to ranges, (as long as we can stop iterating a range and start another one, and when finished with that other one continue with the first one).\nThe range is explained in the “For statements with range clause” section of the spec. What I am looking for is something that behaves like C++ iterator, so we can save it for later continue iterating the range.\nWe can look at golang compiler source code to find how range is implemented for maps:\nC++ compiler implementation: mapiterinit golang compiler implementation: walkrange function for TMAP permalink? In this google groups thread, we find someone asking for the same feature, puting as an example what could be a path finding algorithm.\nIn the responses sections there are some alternatives to save the state of the iteration by using function calls (that is what we are actually trying to avoid)\nThe approach would be to store a stack of keys to save, and then delete all keys that are not in there. So, first, instead of iterating over input dictionary keys, we iterate over whitelist candidates at the root level, and we store a list of pointers to strings in the white list.\nAnd then perform the deletion of the ones that are not in the dontDelete slice, for that level.\nBut this is far more complex that the recursive implementation.\nContributing is a way to learn Even I am not used to Golang, I’ve been able to modify the code and run the tests to create the PR.\nSure I spent more time than required for this feature enhancement, but I’ve been learning through the path.\nFrom my experience, contributing to a project written in a language you do not know is a very good way to learn.\nReferences Hashmap Analysis\nGolang Hashmap source code\nGo Range loop internals\n","permalink":"https:\/\/hontecillas.com\/posts\/2018_01_13_golang_dict_whitelisting\/","summary":"Learning Go through contributing to an existing project","title":"Contributing to learn Go"},{"content":"Motivation I have started adding more pieces into my pet project, and I would like to have some visibility about what is happening and in where.\nWe can classify monitoring frameworks in two big groups:\nThose that actively reports their stats to a central server (like statsd). Those that expose their metrics, to allow services to collect that data(like prometheus). I’ve just selected Prometheus, because is widely used and the project is under the umbrella of the Cloud Native Computing Foundation\nAnd also, I like the Grafana dashboards aesthetics :)\nMinimal setup My setup will have two different VMs:\nOne that will host Prometheus and Grafana, that will store the collected data. Another VM with a RabbitMQ server (that will be used as a jobs queue). We have three ways to install Prometheus\nDocker image Official binary Build from source I choose the official binary.\nInstalling Prometheus There is a fine Digital Ocean blog post to setup prometheus.\nI am following the steps from the tutorial except the part of not creating a home dir for it. I prefer prometheus to have its own home to store the binary, and the database.\nadduser prometheus --shell \/bin\/false I did not gave him a shell, so if we have to perform any action as the prometheus user:\nsudo -u prometheus bash\nIn order to have prometheus running at startup, we can create a starup script, or run it under supervisord or similar.\nRunning prometheus as a service (This is copy \/ paste from the DO article).\nSystemd script: \/etc\/systemd\/system\/prometheus.service\n[Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=\/home\/prometheus\/prometheus\/prometheus \\ --config.file \/etc\/prometheus\/prometheus.yml \\ --storage.tsdb.path \/home\/prometheus\/tsdb\/ \\ --web.console.templates=\/home\/prometheus\/prometheus\/consoles \\ --web.console.libraries=\/home\/prometheus\/prometheus\/console_libraries [Install] WantedBy=multi-user.target sudo systemctl daemon-reload sudo systemctl start prometheus sudo systemctl status prometheus Installing RabbitMQ The version that comes in the default Debian 9 \/ Ubuntu Xenial repos are a little bit outdated, so it is better to download the package from the official RabbitMQ downloads page, or directly add the RabbitMQ Debian repository.\nCurrently, Debian 9 default RabbitMQ package is 3.5.x, while the prometheus exporter client requires at least 3.6.x.\nAlso, we will need an up to date erlang version (19.x and above), so again, we need to add the APT repository, following the instructions found in the official erlang page.\nwget https:\/\/packages.erlang-solutions.com\/erlang-solutions_1.0_all.deb sudo dpkg -i erlang-solutions_1.0_all.deb deb https:\/\/packages.erlang-solutions.com\/ubuntu precise contrib Install the Erlang APT repo key\nwget https:\/\/packages.erlang-solutions.com\/ubuntu\/erlang_solutions.asc sudo apt-key add erlang_solutions.asc Install the rabbitmq prometheus exporter RabbitMQ allows to add plugins, and among them there is the Prometheus RabbitMQ plugin\nThe plugins must be downloaded (ez files are compiled erlang modules), and saved into the RabbitMQ plugins directory.\nIn my Debian installation : \/usr\/lib\/rabbitmq\/lib\/rabbitmq-server-[version]\/plugins (or \/usr\/lib\/rabbitmq\/plugins, I’m not sure if updating the server and not having the plugins updated would work :\/ ).\nwget https:\/\/github.com\/deadtrickster\/prometheus_rabbitmq_exporter\/releases\/download\/v3.7.0.1\/accept-0.3.3.ez wget https:\/\/github.com\/deadtrickster\/prometheus_rabbitmq_exporter\/releases\/download\/v3.7.0.1\/prometheus-3.4.4.ez wget https:\/\/github.com\/deadtrickster\/prometheus_rabbitmq_exporter\/releases\/download\/v3.7.0.1\/prometheus_cowboy-0.1.4.ez wget https:\/\/github.com\/deadtrickster\/prometheus_rabbitmq_exporter\/releases\/download\/v3.7.0.1\/prometheus_httpd-2.1.8.ez wget https:\/\/github.com\/deadtrickster\/prometheus_rabbitmq_exporter\/releases\/download\/v3.7.0.1\/prometheus_process_collector-1.3.0.ez wget https:\/\/github.com\/deadtrickster\/prometheus_rabbitmq_exporter\/releases\/download\/v3.7.0.1\/prometheus_rabbitmq_exporter-v3.7.0.1.ez You should be careful when upgrading rabbimq, since those files should need to be updated.\nThen we must enable the plugins:\nrabbitmq-plugins enable prometheus rabbitmq-plugins enable prometheus_cowboy rabbitmq-plugins enable prometheus_httpd rabbitmq-plugins enable prometheus_rabbitmq_exporter So now the metrics can be accessed through the Management API, so it makes sense to tell rabbitmq to use https:\n\/etc\/rabbitmq\/rabbitmq.config\n{rabbitmq_management, [ {listener, [ {port, 15672}, {ssl, true} ]} ]} In rabbitmq we have the guest user, that by default is an admin one, but that is only allowed to make requests from localhost. So we must add a new user:\nrabbitmqctl add_user watcher watcherpass rabbitmqctl set_user_tag watcher monitoring https:\/\/www.rabbitmq.com\/management.html\nWe can test that the data is beeing exported correctly with a command:\ncurl -k -i -u watcher:watcherpass https:\/\/rabbitmq.lh:15672\/api\/metrics (We must use the -k option, because we are using a custom signed certificate).\nWe need a user that can extract the prometheus stats data:\nConfigure prometheus to get the RabbitMQ exported data scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '\/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] - job_name: 'rabbitmq' static_configs: - targets: ['rabbitmq.lh:15672'] metrics_path: \/api\/metrics scheme: https basic_auth: username: gachapin password: gachapin tls_config: server_name: 'rabbitmq.lh' insecure_skip_verify: true Run from the commandline:\nsudo -u prometheus \/home\/prometheus\/prometheus\/prometheus \\ --config.file \/etc\/prometheus\/prometheus.yml \\ --storage.tsdb.path \/home\/prometheus\/tsdb\/ \\ --web.console.templates=\/home\/prometheus\/prometheus\/consoles \\ --web.console.libraries=\/home\/prometheus\/prometheus\/console_libraries Let’s open the browser, and go to http:\/\/watcher.lh:9090\/ to see the prometheus web console. We can check that we are collecting the rabbitmq data by entering the rabbitmq_queues.\nAdd grafana to plot it beautifully We go to the Grafana downloads page, and again we opt for the option of using their APT repository, so we can easily get the updates.\nIt tells us to use this, no matter if Ubuntu or Debian. Add the following line to your \/etc\/apt\/sources.list file\ndeb https:\/\/packagecloud.io\/grafana\/stable\/debian\/ jessie main and curl https:\/\/packagecloud.io\/gpg.key | sudo apt-key add - to add packagecloud key.\nWe should change the admin user \/ pass in the \/etc\/grafana\/grafana.ini file.\nAfter installation we want to enable it on startup:\nsudo \/bin\/systemctl daemon-reload sudo \/bin\/systemctl enable grafana-server After adding the Prometheus data source, we can look for an existing RabbitMQ Grafana Dashboard, and select the one that uses rabbitmq_exporter for prometheus\nSome of the graph requires minor changes, like the rabbitmq_connectionsTotal to rabbitmq_connections\nAccess prometheus only locally Since we aren’t going to access the prometheus console from outside, we can make it serve the data only to localhost, so we can add the :\n--web.listen-addres='127.0.0.1:9090' So the local grafana interface can still access it. And in case we want to use it, we can still use ssh port forwarding.\nMinimal setup done With this we have the minimal setup to monitor how the rabbitmq queues \/ server is doing. Next steps would be set up alarms if the queue is growing too much, if we are running out of memory, or if the CPU is having too much work.\n","permalink":"https:\/\/hontecillas.com\/posts\/2017_12_30_prometheus_grafana\/","summary":"Monitor with Prometheus and Grafana","title":"Set up Prometheus and Grafana"},{"content":"Motivation I have a server that collects some data, and I want to do some offline processing with that in my local computer, and then upload the computation results to ’enhance’ the data in the server.\nThe data is not so small, but also not too big (in the order of tens of megs), and I am currently running an instance in Google Cloud (with free credits).\nThe tool is not for production use, so the first approach I thought was to make something dirty, like PUTing the file into cloud storage, and create a lambda to perform the operation, but at this moment it only supports JavaScript (Node.js).\nI could set up notifications to build a pipeline using Google Cloud Pub-Sub service, but that is too much work for the job.\nSo, I decide to go with a simple HTTP api (I could call it REST), an is a good opportunity to play, and know a little bit more about Python’s AIOHttp library.\nMinimal server At first sight, something that catches my attention is the fact that to declare a route, the function has the HTTP verb to use:\nThis is minimal server script test_server.py:\nfrom aiohttp import web def handler_func(request): return web.json_response({'bar': 2}) def entry_func(args): app = web.Application() app.router.add_get('\/path', handler_func) return app we can use the more generic method, and use * as HTTP verb:\napp.router.add_route('*', '\/path', handler_func) Run the server from command line:\npython -m aiohttp.web -H 0.0.0.0 -P 8080 test_server:run_service [some_extra_args] ... The list of some_extra_args will be passed to the entry_func so, you can get configuration parameters from the commandline.\nFaster iteration We can run the server easily, but a way to ‘hot reload’ the code would be nice (specially when toying with it). So we have aiohttp-devtools package, that gives us exactly that.\npip install aiohttp-devtools We can run the auto-reload server with the adev script installed with the package. But the entry function differs from the one used for aiohttp.web: it does not allow for extra arguments. In this case, we can get the arguments from the sys module, but taking into account, that it wont strip the adev params.\ndef app_factory(): return entry_func(sys.argv) And we launch the dev server with:\nadev runserver test_server.py --app-factory app_factory (There is also another option: using Gunicorn, with the ```–reload`` flag.)\nSending POST data I already have a GET endpoint to retrieve the information, but I need to send the computed data to the server.\nGiven the size of the JSON we are posting, we can directly read all it into memory :\nasync def update_tile(request): print('received request') content = await request.content.read() content = str(content, 'utf-8') And to test it works as expected, I use the curl command:\ncurl -X POST -d @temp\/deleteme.json http:\/\/localhost:8080\/update_tile Going the dirty way For each request we should create a db session \/ connection that could be rolled back like in most frameworks do (or, better said, provide a connection from a DB connection pool). However, given this is only an small tool, I’m going the dirty way and I will share a single connection to the db that I would reuse for each request :\/.\nWe can store “global vars” (yeah! that’s ugly), into the app directly, like a dictionary:\napp['session'] = create_session() (Here we should put a reference to a function that creates the session instead of directly put the created session).\nThe same can be done for a request. We can hold that for that ‘context’ in using the dictionary syntax. So the clean way to do it would be :\ndef setup_app(args): db_session_factory = create_db_session_factory_from_args(args) app = web.Application() app['db_session_factory'] = db_session_factory def handler_func(request): db_session_factory = request.app['db_session_factory'] db_session = db_session_factory() request['db_session'] = db_session call_whatever_function(request) Actually the handler function should be more like a decorator to wrap all other handler functions, this way we would provide DB access per request. But, for that we have aiohttp web Middlewares\nDeploying the app We can run the server from the command line, in production, under supervisord, but it is better to follor the official documentation about aiohttp deployment\nTesting on a local environment Creating a VMs We can actually use Vagrant to spin up a new VM, however, I have a reference VM that I clone to create new instances, that already has my public key for the root user to login through ssh.\nTo create a new VM I use this script:\nexport SRCNAME=debian9base export SRCIP=192.168.56.180 export DSTIP=192.168.56.70 if [ $# -gt 0 ] then export DSTNAME=$1 fi if [ $# -gt 1 ] then export DSTIP=$2 fi echo "Creating $DSTNAME from $SRCNAME at $DSTIP" VBoxManage clonevm $SRCNAME --mode machine --name $DSTNAME --register VBoxManage startvm --type headless $DSTNAME sleep 10 ssh -T -l root $SRCIP bash -c "' echo $DSTNAME | cat > \/etc\/hostname sed s\/$SRCIP\/$DSTIP\/ \/etc\/network\/interfaces > tst cp tst \/etc\/network\/interfaces rm tst shutdown -h now '" echo "Created VM $1 at $2" What is doing is:\ncloning a virtual machine snapshot: VBoxManage clonevm $SRCNAME --mode machine --name $DSTNAME --groups \/AIOHTTP --register\nthen we start the VM: VBoxManage startvm --type headless $DSTNAME\nsince it is a clone of the previous one, we must log in and change its hostname, and static IP. We could create an Ansible Playbook, but it is too much job for this simple ‘setup’ of our clean install machine\nto delete the VMs: VBoxManage unregistervm AIOHTTPServer --delete\nThen I add a friendly name to the \/etc\/hosts file\n192.168.56.90 server.aiohttp.lh 192.168.56.91 client.aiohttp.lh Deploying a client process Now I only need to create my client process, that fetches the information, performs the computation and then reports back the results.\nI could use a cron process to run every X time the script, to do the job, but I am going to run a process that keeps asking the server for tasks to do, that will run in a raspberry pi.\nReferences AIOHttp Server Usage official docs aiohttp Swagger Getting started with aiothttp tutorial ","permalink":"https:\/\/hontecillas.com\/posts\/simple_server_with_aiohttp\/","summary":"Building a small service with Aiohttp.","title":"Simple server with aiohttp"},{"content":"Motivation Finding image duplicates can be useful for lots of purposes: find fake profiles in social networks (checking if the profile pictures are a copies from other profiles), find content picked from other sources, etc..\nPerhaps we can not only rely on the images to determine that, but they give us a good hint.\nWe can also look to find duplicates of our own images, however, if the images are exactly the same file (not resized \/ edited versions), we can perform better simply looking for file duplicates.\nFinding file duplicates Hashing all files in your disk does not look like a great idea, so, if you know the file extensions of your files, you should filter the list with those extensions (jpeg, jpg, png..).\nSince files can be of different size, and accessing the file size is faster (it only has to access the directory inode info), we can first make a selection and group files by same size.\nOnce done that, best approach is to directly hash the contents of the files, and compare those hashes to check if are the same.\nAlso, if we have very big files, and slow disk access (perhaps in a remote location) hashing the full content could be expensive, so we can repeat the process we did with sizes, but hashing only a block of data from the begining of the file (A few kbs for each file).\nFiles can have the same bytes at the begining (image headers), and is only useful if we are going to discard duplicates in an early pass (files matching should be fully hashed to be sure we have the same file).\nThe last step, is, with the groups we have, split them by full file content hashing, and output the duplicates list.\nHere I have some code that is not optimized, but could serve as a basic implementation for this task:\nPython 2 code to find exact file duplicates\nComparing images Standing on other’s shoulders A lot has been written about image duplicate finding. Actually, one of the first references I used was Skyscanner’s post about image deduplication, where they explain the process used to detect image duplicates and select the best one to show to the user, and the tool they built to check that everything was working as expected.\nIn that article you can find references to the libraries used for image hashing like Python’s ImageHash and some other good articles about image hashing, like how the perceptual hash works. Actually, I must point that there is another good blog post about the topic in the same site about dHash compared with aHash and pHash.\nSome samples In order to show those image differences I have selected an hotel image from different sites (Cardenal Hotel at Monforte de Lemos):\nsource width height aspect ratio file size booking.com 1024 683 1.5 67 232 tripadvisor.com 840 460 1.82 41 078 lol.travel 694 462 1.5 44 994 roomdi 903 500 1.86 31 589 zenhotels.com 1000 666 1.5 82 147 ![booking image](\/images\/imagedups\/booking_cardenal_sm.jpg) ![roomdi image](\/images\/imagedups\/roomdi_cardenal_sm.jpg) If you follow the links provided in the table you will have the full size image downloaded from each site, and you will also see that not only the aspect ratio is different (some are croped), but also que image quality (i.e. the roomdi one is really really bad).\nUsing Python’s ImageHash So I decided to collect some pictures, download them and save three of their hashes: pHash, dHash, and wHash. I left the aHash one out, as it seems to not give as good results for false positives, and I prefer to err on the false negatives side.\nBut as I previous step I croped all the images to a squared size, this way I can get rid of different aspect ratios, not using the “extra pixels” at the borders.\nI used a sqlite3 database and the Peewee ORM, where I stored like 33 different hotels, with an average of 15 images each one.\nToo slooooow So, I started with the naive approach of checking each hotel with each other, to find if they had image duplicates. Yeah, thats an N^2 algorithm, and should be optimized. However, something smelled really bad for an N=33.\nIt took 27.729 seconds to process all of them !!\nSo I decided to profile it with python -m cProfile -s tottime dupfinder.py , to find out that most of time was consumed with image hash comparison:\n9817878 function calls (9788041 primitive calls) in 27.729 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 2458848 11.742 0.000 11.742 0.000 __init__.py:110(<listcomp>) 307356 7.129 0.000 21.626 0.000 __init__.py:97(hex_to_hash) 307394 2.226 0.000 2.226 0.000 {built-in method numpy.core.multiarray.array} 51226 0.693 0.000 23.188 0.000 imghasher.py:82(calc_hash_distances) 153678 0.615 0.000 0.855 0.000 __init__.py:75(__sub__) 21882 0.408 0.000 0.703 0.000 _strptime.py:321(_strptime) 2549715 0.323 0.000 0.323 0.000 {method 'append' of 'list' objects} 7273 0.265 0.000 1.851 0.000 peewee.py:2428(construct_instances) 682 0.252 0.000 0.252 0.000 {method 'read' of '_io.FileIO' objects} This is the offending code inside calc_hash_distance:\nh_a = hex_to_hash(hexhash_a[hk]) h_b = hex_to_hash(hexhash_b[hk]) hash_distances[hk] = (h_a - h_b) \/ 64.0 This was beeing executed for each stored hash (remember that I had 3 for each image, and I am ran an N^2 loop to compare them).\nIf we look at the original source code for hex_to_hash, I am actually creating two ImageHash objects from the hex string only to find the number of different bits from one hash to the other.\nWhat happens if I code it directly ? I changed that snippet for this:\n_LOOKUP_BIT_COUNT = { 0: 0, 1: 1, 2: 1, 3: 2, 4: 1, 5: 2, 6: 2, 7: 3, 8: 1, 9: 2, 10: 2, 11: 3, 12: 2, 13: 3, 14: 3, 15: 4 } def calc_hash_distances(hexhash_a, hexhash_b): for cha, chb in zip(hexhash_a, hexhash_b): dist = dist + _LOOKUP_BIT_COUNT[ord(cha) ^ ord(chb)] hash_distances[hk] = dist \/ 64.0 and run again the profiler:\n2940613 function calls (2911799 primitive calls) in 3.679 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 21896 0.384 0.000 0.685 0.000 _strptime.py:321(_strptime) 7281 0.271 0.000 1.879 0.000 peewee.py:2428(construct_instances) 316495 0.207 0.000 0.453 0.000 {built-in method builtins.setattr} 216598 0.177 0.000 0.225 0.000 peewee.py:866(__set__) 7339 0.114 0.000 0.114 0.000 {method 'fetchone' of 'sqlite3.Cursor' objects} 681 0.112 0.000 0.112 0.000 {built-in method marshal.loads} 179648 0.096 0.000 0.187 0.000 peewee.py:960(python_value) 27686 0.076 0.000 0.076 0.000 {method 'match' of '_sre.SRE_Pattern' objects} 14581 0.074 0.000 0.101 0.000 peewee.py:4734(__init__) Nice! 3.679 seconds. A 7.5x speed up. It seems like a good advice to compare the hashes directly instead of using the library.\nFurther improvements Obviously I can use something better than compare every hotel to each other, and also, there are good articles about it, like this one explaining BK-Trees for finding near-matches to a string, that I found in this also good blog post about Duplicate Image detection with perceptual hashing in Python.\nIn the Hotel’s case I could filter out based on the name of the hotel, so I can reduce N to a small number.\nAlso, it would be great to have a tool to check that everything is ok :)\nReferences Image deduplication from Skyscanner blog PHash algorithm explanation dHash compared with aHash and pHash. Duplicate Image detection with perceptual hashing in Python. BK-Trees for finding near-matches to a string, ","permalink":"https:\/\/hontecillas.com\/posts\/2017_05_25_finding_image_duplicates\/","summary":"Finding image duplicates when they come from different sources have some details that must taken into account, and differs a lot from finding file duplicates.","title":"Finding image duplicates"},{"content":"Your App, your public API If you have an App in the market, you already have a public API.\nAnd getting access to that API is much easier than you might think. I am not a security expert, and I have never done any kind of reverse engineering.\nMotivation I had one of those weekends where I wanted to gather some information from a webpage for a short personal project, and I checked if the web page provided some kind of API. It actually did, but, you had to request access filling a form, that someone would review, and then provide access to the API.\nI did not want to wait, so it was a good excuse to try to extract the API from their app.\nAbout APIs for mobile devices Mobile API are exposed to everyone. With more or less trouble, but your API contract can be extracted.\n** You can not easily deprecate the API ** : Once you have released an App you can not deprecate that API from one day to the next. The point is that, releasing a new version of the App does not mean that your users will update it soon (some of them won’t update it at all).\nYour mobile app is a frontend, that is easily “hackable”. Do not expect your mobile devs to fix API deficiencies.\n** It could be your presentation card **. Are you looking to hire developers? candidates can know a lot about your code by looking at your APIs.\nExample: VIBBO Vibbo is a marketplace, where people can sell and buy used items. It is like LetGo or Wallapop, with the only difference that they are not using HTTPS for their main webpage (ehem! :P).\nI did not choose this app for any particular reason: I selected a random downloaded app from my phone.\nI used the Android App (is easier to extract the app when we need to decompile it). The same process could be done on iOS, but with different tools. Look at the references at the end of the post. (Hint: if the man in the middle is not enough you will need a jailbroken device).\nI did not extract all API calls, I only wanted to show how it can be done.\nSet up the MITM First of all I connected the phone to the same network than my laptop, and looked for a tool to help me do the Man-in-the-middle.\nThere are several tools to inspect traffic, but I decided to use MITMProxy, mainly because:\nIt provides a very easy way to install self-signed certificate. (You run the proxy in interactive mode, configure your device to use that proxy, visit http:\/\/mitm.me, and follow the instructions). It has an “interactive” mode (mitmproxy) where you can modify requests on the fly, and a “dump” mode (mitmdump) to save requests flows to a file so you can later and reproduce the calls as many times as you want. An easy way to Filter the requests. (Take into account that once you set the proxy all the requests from the device will use that proxy configuration). It is scriptable with Python (actually all the tool is written in python) It has a lot of options, I recommend you to read the full MITMProxy documentation.\nKey bindings are pretty similar to those used in the Vi editor.\nCapture all traffic I used mitmdump for the first run, so I saved the request \/ response flows to a file, so after that I load the flows from the file with mitmproxy to inspect everything.\nmitmdump -w vibbo.mitm If you capture before downloading the app from the Android market, you will have the .apk file inside the capture file (that can be useful later, to dump, extract and decompile it).\nOnce we finished some navigation, I closed the app so I could start working with the file using mitmproxy (the interactive mitm tool):\nmitmproxy -nr vibbo.mitm (the n flag is to not run the proxy, otherwise the proxy would read the captured saved file, but also would keep capturing new traffic).\nThen I could have a look at the domains I was interested (because I did not have any filtering yet, I could see all traffic, not only from the App we are inspecting, but also all other apps in the device).\nI chose to inspect domains containing the ‘vibbo’ string:\nmitmproxy -nr vibbo.mitm -f '~d .*vibbo.*' ``` At first saw I could find different kind of apis:\nv1 via HTTP v2 via HTTP the users api call using HTTPS When I looked at some of the requests, I found that the query parameters were encoded. The q param looked like a base64 encoded string, while the s param looked like it was hex encoded.\nI tried to decode the q param, to find that It did not have any meaning to me:\nLooking at the response, it was not much better:\nC3cUjJrZeKy\/hFeBHzn2UHllH2edGisyn6TBew82LKN0XsMaS7DcNYh8Y3PPAzdqBGpzKqhQoNUt8XwMBp+tHCs5Emky74bHje4Dp+JMdIg9NdLsexz6NzSpTguDQBwjSC1lh8PPZcTiMCbkjcUbF3bPqBIlLnDDnvbCElzTQRk6hUlvZhFamppBGYcXLCUk4nn6mrbU50lMpRCDkKq\/3svfR2oRWxB0inNecQ984RNXQ2VT9H73VRZ1PL2MRSK+8C7gFNL9uAhbjErfpfzMnIg4lXXbkzrTGlomxRJ8I8pAE3dTctd58oujGchWb76FAT0l3Gqj0yWHmB460s0KDB0Ip8atZWT+UQ9HEk29uec6s+QxZTMSiPVg4idGQjLdiYTL4nGyVRTp5r3Mczc\/q4jdADcBgwmTjtux9hAgNEFpc4nBHYTd4qXAxLQZ7SOgNK8RHWBparzpG5YKegvDtGUad\/p2SwScKOmItNh5mBwoC63Ra4NvyffG0Ig292nzP\/D8sHC5CvF1jSE4a9s+GUNQVePNLDZyfrHsACkNnKyPPCR2+DbPGgmmHT\/rXrr6J1sfms\/LIFpQGvZ3dKuVfMYNLsJnFXYN1XRn\/dotH50ZCDFGkdVfGOV1jM3anKIX1WHqDGDJCZSJJ9oTH+tI1tt+pb\/olJhsfgUQwmD0z5\/v3E4vvp+deZtbTwKwCe97vBjDiC3FyUxUObQqCfKHVqJatEIdkWfO3B0DC2QJ9huym8NcgoH1UaIcRLa8wLBHKVdTyy+6+LClZpdZ5UEyMiBfA\/MB+aaCPUqGJzXeATe4Yh2CbGnIUitmTZEv6p+1tPRiwHLXZkED0h++WaM32otlk0qOtEUrd9zISStI54tYNmnfscQZ8RJqp7CaSuVoSzzqzl68MH3Lup758IUxZoGV+dj0OQwkwjyivmpSp6W406It5czcED8w7FsJ433dmXMtFCxyKx9Q4RKwBtsm2g== So, I thought I would probably need to access the app code to see how the encoding was done.\nThere were other requests (_ http:\/\/api.vibbo.com\/API\/forms\/search?auth_token= _), that did not had encoding. In this case, the response looked like a set of categories and rules to show suggestions for the app’s search view.\nI was missing something. I could see the \/detail request, but not the actual search that I performed before (for PS4 consoles). So I loaded again the file, this time without filtering by domain name, to find that I actually missed an important request that was not being done against a vibbo domain:\nhttp:\/\/hl-search.scm-pro.schibsted.io\/mixed-ads?q=ps4&search_by_map=true&latitude=41.3850639&longitude=2.1734035&distance=4000&category_parents=15&price=100%3A400&sort=distance%2C-publish_date%2Cprice and that it looked that were the actual search result\nThere were also api calls to get information about a user, following an easier RESTful scheme, with non-encoded params in the url, and also a non encoded response.\nGET https:\/\/ms-vibbo--app-userapi.spain.schibsted.io\/v1\/accounts\/16895146 From my point of view, it looks like the encoding part of the \/detail API endpoints are a measure to compensate for the use of HTTP instead of HTTPS.\nBut I wanted to get access to this endpoint too, so, the next step was to decompile the app to check how to decode the requests.\nDecompiling the APP First of all, I extracted the apk (the body of the request is actually a gzip, once you decompress it you get the .apk file), and decompressed it (the .apk is actually a zip file).\nThen I found the .dex files, and I used the dex2jar tool te convert all .dex files into .jar files, that I fed to the Procyon decompiler.\nProcyon can choke with some methods (where it spits out the Java bytecode opcodes), but it does a very good job producing Java source code.\nprocyon-decompiler classes-dex2jar.java -o . procyon-decompiler classes2-dex2jar.java -o . And it looked that the source code had been obfuscated probably with proguard.\nSo, I did a simple grep to search for the api.vibbo and \/detail strings, so I could find spots to start diving into the code. And I found a couple of definitions:\nConstants.d = "http:\/\/api.vibbo.com" Constants.H = "\/api\/detail" After looking around a little bit for the constants, and not finding anything in the code that could easily hint me how to encode \/ decoded the fields, I decided to grep for Base64. Just before the code that performs the base 64 encoding I should find the encryption part for the query param and the response body.\nAnd finally found two interesting classes:\ncom.anuntis.segundamano.api.ApiHandler : This class gives us information about how the s param is baked: by making an md5 over the q param before the q param is cyphered: public static InputStream a(String s) { final InputStream inputStream = null; try { final Encryption encryption = new Encryption(); final String encryptAsBase64 = encryption.encryptAsBase64(s); s = encryption.md5(s); s = String.format(Constants.h, URLEncoder.encode(encryptAsBase64, "ISO-8859-15"), s); final Response execute = new OkHttpClient().newCall(new Request$Builder().url(s).get().addHeader("charset", "iso-8859-15").addHeader("User-Agent", "android").build()).execute(); InputStream byteStream = inputStream; if (execute != null) { byteStream = execute.body().byteStream(); } return byteStream; } catch (Exception ex) { ex.printStackTrace(); return null; } } com.anuntis.segundamano.utils.Encryption : This is where the main encryption \/ decription algorithm is performed. Encryption The encryption class required some patience, since the decompiler failed to transform byte code into java code.\nprivate static String getPass() { return "4425363b0c9ecd76a4fee7680e41e3285ef507e18521a916ea6bd9f589df89ef2c2fda41547bd32543b38e82bcbc74da"; } public String decrypt(final String p0) { \/\/ \/\/ This method could not be decompiled. \/\/ \/\/ Original Bytecode: \/\/ \/\/ 0: aload_1 \/\/ 1: invokevirtual java\/lang\/String.length:()I \/\/ 4: bipush 16 \/\/ 6: irem \/\/ 7: ifne 54 \/\/ 10: ldc "AES\/CBC\/NoPadding" \/\/ 12: invokestatic javax\/crypto\/Cipher.getInstance:(Ljava\/lang\/String;)Ljavax\/crypto\/Cipher; \/\/ 15: astore_2 \/\/ 16: aload_1 \/\/ 17: ldc "ISO-8859-15" \/\/ 19: invokevirtual java\/lang\/String.getBytes:(Ljava\/lang\/String;)[B \/\/ 22: invokestatic com\/anuntis\/segundamano\/utils\/Base64.decode:([B)[B \/\/ 25: astore_3 \/\/ 26: aload_2 \/\/ 27: iconst_2 \/\/ 28: getstatic com\/anuntis\/segundamano\/utils\/Encryption.secretKey:Ljavax\/crypto\/SecretKey; \/\/ 31: getstatic com\/anuntis\/segundamano\/utils\/Encryption.ivParameterSpec:Ljavax\/crypto\/spec\/IvParameterSpec; \/\/ 34: invokevirtual javax\/crypto\/Cipher.init:(ILjava\/security\/Key;Ljava\/security\/spec\/AlgorithmParameterSpec;)V \/\/ 37: new Ljava\/lang\/String; \/\/ 40: dup \/\/ 41: aload_2 \/\/ 42: aload_3 \/\/ 43: invokevirtual javax\/crypto\/Cipher.doFinal:([B)[B \/\/ 46: ldc "ISO-8859-15" \/\/ 48: invokespecial java\/lang\/String.<init>:([BLjava\/lang\/String;)V \/\/ 51: astore_2 \/\/ 52: aload_2 \/\/ 53: areturn \/\/ 54: ldc "AES\/CBC\/PKCS7Padding" \/\/ 56: invokestatic javax\/crypto\/Cipher.getInstance:(Ljava\/lang\/String;)Ljavax\/crypto\/Cipher; \/\/ 59: astore_2 \/\/ 60: goto 16 \/\/ 63: astore_2 \/\/ 64: aload_2 \/\/ 65: invokevirtual java\/security\/NoSuchAlgorithmException.printStackTrace:()V \/\/ 68: aconst_null \/\/ 69: astore_2 \/\/ 70: goto 16 The byte code was actually an AES encryption algorithm. And I found that in the class there was a getPass() method, but actually that ‘pass’ was used to generate the the secret key and initialization vector for the AES algorithm\npublic Encryption() m{ this.passphrase = getPass(); final byte[] hexStringToByteArray = this.hexStringToByteArray(this.passphrase); final byte[] array = new byte[32]; final byte[] array2 = new byte[16]; while (true) { try { System.arraycopy(hexStringToByteArray, 0, array, 0, 32); System.arraycopy(hexStringToByteArray, 32, array2, 0, 16); Encryption.ivParameterSpec = new IvParameterSpec(array2); Encryption.secretKey = new SecretKeySpec(array, "AES"); } catch (Exception ex) { ex.printStackTrace(); continue; } break; } } Given that I only needed to check if that was what I required to be able to use the \/detail endpoint. So I coded a python script, and fed it the q param, and a response from the captured request.\n# -*- encoding: utf8 -*- # from __future__ import unicode_strings import requests import json from pprint import pprint as _p from datetime import datetime import io import base64 import hashlib import random import struct from Crypto.Cipher import AES class Encryption(object): """ to view the algorithm I used the jvm opcodes: https:\/\/en.wikipedia.org\/wiki\/Java_bytecode_instruction_listings """ def __init__(self, key, initialization_vector=None): self.key = buffer(key) if initialization_vector: self.initialization_vector = buffer(initialization_vector) else: self.initialization_vector = buffer(bytearray(16)) self.cbc_block_size = 16 def _pkcs_7(self, message, blocksize): padd_byte = blocksize - (len(message) % blocksize) padded_message = message + struct.pack("{}B".format(padd_byte), *[padd_byte]*padd_byte) return padded_message def encrypt(self, message): if len(message) % self.cbc_block_size != 0: message = self._pkcs_7(message, self.cbc_block_size) aes_crypt = AES.new(self.key, AES.MODE_CBC, self.initialization_vector) res = aes_crypt.encrypt(message) return res def decrypt(self, message): aes_crypt = AES.new(self.key, AES.MODE_CBC, self.initialization_vector) message = aes_crypt.decrypt(buffer(message)) return message def encrypt_ftcws(self, message): padded_message = self._pkcs_7(message, self.cbc_block_size) aes_crypt = AES.new(self.key, AES.MODE_CBC, self.initialization_vector) res = aes_crypt.encrypt(message) return res def encrypt_to_hex(self, message): enc_message = self.encrypt(message) hex_message = ''.join('{:02x}'.format(x) for x in bytearray(enc_message)) return hex_message def decrypt_from_hex(self, message): message = bytearray.fromhex(message) return self.decrypt(message) def encrypt_to_b64(self, message): enc_message = self.encrypt(message) # b64_enc_message = base64.b64encode(enc_message) b64_enc_message = ''.join('{:02x}'.format(x) for x in bytearray(enc_message)) return b64_enc_message def decrypt_from_b64(self, message): decoded_b64 = base64.b64decode(message) return self.decrypt(str(decoded_b64)) def vibbo_test(encoded_snippet): passphrase = "4425363b0c9ecd76a4fee7680e41e3285ef507e18521a916ea6bd9f589df89ef2c2fda41547bd32543b38e82bcbc74da" bin_passphrase = bytearray.fromhex(passphrase) secret_key = bin_passphrase[:32] initialization_vector = bin_passphrase[32:] enc = Encryption(key=secret_key, initialization_vector=initialization_vector) qparams = str(enc.decrypt_from_b64(encoded_snippet)).encode('ascii') endstr = -1 while ord(qparams[endstr]) < 16: endstr -= 1 qparams = qparams[:endstr+1] try: jparams = json.loads(qparams) _pp(jparams) except Exception as e: print(qparams) test_a="GenYvIyueH+CY1or7Q4AhM5fKB0qF2jr6qkqn6UKllF0BpSeHIAFEAiieRHQ36\/PBc7oLZaOBy+s1iQYgOX\/bxE\/SKjzEOi5CQ5FqncNRrA1extJJdJVfutswgS+WA9vepzO2dRnKvRNaS+OqIkoQLD5AA+AhX4qPlew18yzoPtkP14cqM3bIVcRN3r9M6TVTzGEg9QZL4Ow5qBXvYBLg3VBixYHQku2k6Z5mlwAVBabQTHHg0WKrP\/5sEG5A0+K541HQZvqS4wCIMBzfWojQkDazBt6IOLMez79KW0JaeUkSKf\/DTtao0L6J+Y4LTTWpU6ti8XEtp6lWwwgkhIGF349HhtI3ESDHDOo\/fZJ9fUDtXzTHy10VSVhqvEt1DprLZ\/\/Nzk+NdIoKKJQpDJf7xXjsUD8C2o6CXvf8\/U09vaSB7e\/75SvnfKdwCdKJ\/dyhp3XKeVBkwZE0m5JiDzRauWeEkuTjnm4V8PcwIN0kUcAFfnT1738cu14SvvSRSaJMW2gXqrohCRRS9zk18V4np\/ojFKvrp+W36Z3MDCAXtFhe0KcIBkXXtDZdLomgi5+5MnODVj0oVSL4yeV\/WoVVkXlmq0u1H13CozMQEp5RQphOPt02EmMW2HWhY+zj\/KKSMzh2VA8p0h+anbmjw0goEpl6jwmOVDDv5QxZpwX\/y3uYQoFjy++vMqG8IbjJ33MdLMwJKq8rCdrDS\/aY5Mj\/k0BVbmyMNgyvOJb7+DEmxRbla3oqTqEnAWfFetGM9kAHqe1xK+AblNct5WIEjgxPEMRgtrG+\/D2RArKlwK5+MWBBc7T+vprrLeXjnvdC8E9h4wOhUek53+M7R4Vz3YWDnXiFWGCBDnIZsHEvd6QBC3qn22P5IB2h5hdp6wjuTaNCAK3joQMq6xC9SYiBBBXcw39Anndxu67h1LPPMaoSjNQZre+qqj2M3hnvoy8f4o8nc4dZOyhxjomsio75R2dsEbkSvPUEjKJAAnJDw12OGIPo7T+GTdaWR2zsJKMMuSf80RT+0O2etjyqCew4LCKS3V0KFHawlgXSmvuqzo7G\/sbCUnxUQDwGYa8esE289GbLgZn7WiWG88cchCG+mehLXxJCp4qHg5ppp2P6uhW73d84FRKVgGFJf3F+PT7oAR3h2eTAfXLKvRz8RiTD6yruZWhqafDTm3oiyTT6zvXlJYDXTMkDLCw3EVGfCLEpaXDCTotlEMmqHwv5ZjtkO6hMp5TRqbszayP+vTVr21\/upEvXtSAQSAma\/ZMYHbmlnR0cYe8pXbpqEwS\/rvL\/n3jwbA\/j7R8aGIBemkhH6M8YJVO7tdO7kzi\/heSB2w4O\/KkQkJtXsE5NZN4Ts167b+FRRyyvoY185qC3eV\/7bCZrV7v4+VfVcOYe2zy2guqihWv3yaXns+OSeESIofcTjBPSiDwrLpsS1LMCh9INFdwMJmvjuz5oARh0YC9ZRuDThzuJUKqugdTxDuEVYQmKlZkei6w5u0Pdky6mm9XlOwEISLX9jQn4VNToSxlKxPwGtAMq8ikXCQDriZ4n8Mk43ntZw5qKKbAWgNZOxXsepQNmTKChPD+y9L59LtSXGT7KuKfH0Z6DJSMYzyYcViIruYV9Pmit83rUi9OMspjr9gJM\/i4EKzFbOfIitLizr4wLFDATOvdI4ZZizssJj5e44QxoIidHV\/GC0sWaU\/ZkulHAMD3Xx2BPq1t2+ULctp1BzbK+AIB\/ldhGHWj3lkQUq5txxqDif04w\/XblVRsUCMhy\/XRyNAvjgCBN9D5vs\/VbCpHXuH\/b6hCXOv\/q\/ar\/J\/yZmco4Etp8xFo707FaEaly2BemDgA75sX4vvq+V5Io6tmpFe\/KgAEqDEJ2T8FAO3pyB+KM5b+q3HZgGZqOBMIQD7PBP99\/V2eIh+OWY3qGTmIQ7v19jcShk3QJ1o2wWulCDQAfxrO7aXIfmSMtJinuxwMS6bf43J7ZG1NVGLbb8aH1kfn28awEjdYkPmRjzWh6bGcYSvU1mVNT4G8kFisndO3dxjaE8cCT93nroYTf\/ZINiXhP3kpki4ltIJj8GThIA==" test_b = "FxLN30qJl+zcQfcXh4EMAA==" test_c="C3cUjJrZeKy\/hFeBHzn2UDkH8imDgOo8rl2ma8tldnAt3IOUrX0f7PsNdhU7bHR0swqI4gnGBCpVRoamH6zJQ\/HlP69v6CiVZ6iDECqEZjgetOsZyeDMncXP1WF5YbnaEOCyGApjTYIeX4d\/zwOpJSEsA0xK2NUTKswyIaYdfuIfaKSTbTc96x6hxpUrGs2RUPiJhtMR+Zq9ntbRqe+XWaJyzT1od2MWduqA0Le77xB+ASbe0c46kHK+o+f6rmYp9DQgvT4BeqVTHbwC1V74TQ6kdr\/Xb\/ORhmupYSHy3T00Fck0tqzfSvojVJSlWosHyLgo5+cj8JDOU1PNOBIcrDhtSFj4kMtqBTXPGqqof70amNrR8Fm1Xuuum7CWPavsvh7NwwASi7J0OBHNxXTTwYwBqPf+uXdvcQV9\/QK3A2WZjpK7QXdIhc2JQxDYfy17T0TdQGIehfOussGTfOZhg8Ez0OBNAHe408Mf83mMMepgJPwH5Gdx5njyJZ+0gpa\/IrvMS9R3yXjQKsvBqg2BTOJ+qTujLEAYEJlmW64P7clekHdHrB9Cs7d5\/UEOZqieneh\/3cCTr9A90lYXLjmZ1QKfwkSAkM5nyDdnZOE2xX+lLCpV8uh2nEP3Rwfqa8jj7B2Frf05AV\/8Lq8bcOvqY0VfKRw+QrL\/X\/tR6kJ6ratwnxBZoGM\/TGMkh8l4AqkjWE5WElEK1ugeRivynScVL7hDV1PkkRe67C9ozfJHxKybtvZIOOn0RHPyVEzC8IEIDGH0EMA5NpDHYWhndklfdtfA\/oLRVOq3pnrAivSpOcEys5lJfxwMA4lKO\/XrNPOZjXXtyMGkgTqfs82WFA4pxeQKQatM0cW5BXW7d6apJui2zupV4zNEEJxygkBMHGihWb9nVWdaPi3Hh8yVGU3pApgvMUG8hYS3Eqg9qiZWRBdE7uFhS4DEqeG5TIZ7XCh30vIssafliRUaUoh4G75yl9r2WjBtRS7cwvI0SmAjlNnU3z3N4y6+Gc1F+gMmkOi\/H8PguDwhCOcEitQfJ79oT+bTeUPrUlpS22gL\/bO66RfSQFBaoyzFoPjfG\/vYU7cwkGSNmLonxRNKvp5QRDlG17uQvv32OPu2XnwPfLuBjMPzt0ebCpORc4YEdExQJMKcPoL73tSalASwqeWhzju4yg==" vibbo_test(test_a) vibbo_test(test_b) vibbo_test(test_c) Aaaaand… it worked !\nSo at this point I could do searches, and extract details from the results without problems.\nObviouly there are a lot of API calls that I didn’t use, but that could be ’extracted’ too.\nImprovements to the API ’extraction’ process If you plan to use the explained method, you will probably want to automate some stuff.\nAutomate decompilation Given that mitmproxy support scripts, is easy to create an script that extracts the .apk from the flows file, and automatically decompiles it.\nAutomate schema extraction: There are libraries to automate the extraction of the JSON schema, like skinfer that can be used.\nDifficulting the API ’extraction’ As previously stated, the battle is mostly lost: if someone want to extract your API, will do it. However we can make it a little bit harder:\nUsing certificate pinning: But you should take into account, that If you do certificate pinning, you should provide a way to update the information of the certificate in your app (in case you have your certificate compromised, or it expires). On the other hand, the app could be decompiled, modified and then recompiled in order to tamper with the certificate check. I’m not sure what would be more time consuming, directly inspecting the decompiled code to extract API requests or doing the decompile\/recompile process.\nObfuscate the secret key and initialization vector: We could probably try to compose \/ generate a complex algorithm in order to generate the secret key \/ initialization vector. But since we can get the code, we could run it to generate both of them.\nMove sensitive code to a binary library. Those libraries can also be reversed, but that requires more work and knowledge than has been required for the Vibbo app. The problem is you would need to use jni, and provide binary libraries for ARM, MIPS and Intel architectures.\nReferences Other tools:\nKrakenD : Can help you to hide a more powerful API that runs in the backend, or can combine information from your backend microservices.\nWuzz: Interactive CLI Tool for HTTP inspection\nArticles:\nReversing Runtastic API Removing Edge Magazine DRM The Security Impact of HTTPS Interception ","permalink":"https:\/\/hontecillas.com\/posts\/your_app_your_public_api\/","summary":"Your App is exposing the API to everyone.","title":"Your App, your public API"},{"content":"After reading an article about the benefits of tmux, I have decided to give it a try.\nWhat I find more interesting is the ability to easily setup your development environment, specially for Django when you need to run the server, the celery workers, and probably an HTTPS proxy (like stunnel) to redirect requests from external services (paypal, facebook calls…).\nFirst steps First of all I prefer to change the prefix key binding (the one that tells that the next keys are for tmux and not for the underlying term) to something more comfortable. I’ve found _ C-u _ easy to type, and I think it does not clash with any command that I’m currently using in Vim.\nSet the prefix, and then bind the second time whe press _ C-u _ to send it to the terminal:\nset -g prefix C-u bind C-u send-prefix unbind C-b Also from this article about basic tmux, I like the part of remaping the keys to cycle panes.\nunbind ^A bind ^A select-pane -t :.+ We want to set the default terminal to support colors:\nset -g default-terminal "screen-256color" And for vim inside tmux, if you have any problem having color, check this tips for having 256 colors in vim. (Vim users may also want to set the ttyfast option inside tmux)\nTo enter copy mode C-u [ then we move around the windows. In order to get a vim-like navigation we should set:\nsetw -g mode-keys vi (source: Copy & Paste in tmux ).\nLook at the man page to know the basics about moving, and copy paste commands.\n_ C-u z _ to toggle zoom in your actual pane is very useful if you want to focus on a panel.\nSet the maximum history lines:\nset history-limit 5000 Creating a tmux session for a project You can check how to list the actual screen layout, and then save it to restore it later:\ntmux select-layout 3922,317x74,0,0{200x74,0,0,3,116x74,201,0[116x36,201,0,2,116x37,201,37,1]} Tools to manage tmux sessions tmuxp, tmuxp docs, and some tmuxp examples tmuxinator. References tmux cheatsheet tmux simple start using tmux properly tmux online man page tmux 2 Book The Tao of tmux Book ","permalink":"https:\/\/hontecillas.com\/posts\/settingup_tmux\/","summary":"Improving my environment using tmux","title":"Setting up TMUX"},{"content":"I have “monolith” django application for my website Katagatame.com. It is a very low traffic website at the moment, and given that I am the only developer, it is not something that I’m not going to change right now. (I can live with the drawbacks of everything running in a single server instance).\nA monolith does not mean having a complete mess in the code. Actually, after identifying some reusable code I want to move that code out to a private library, so I can easily reuse it for other projects.\nI do not wat to factor out a django app but simply a python library. Since it is going to be a private library I want to install from a git repo using pip.\nInstall from a git repo The simplest way of all is getting the master branch of your repository:\npip install git+ssh:\/\/git@bitbucket.org\/dhontecillas\/rankedcompos.git In the root directory you will need to have a setup.py file with information about your package:\n:::python from setuptools import setup setup(name='rankedcompos', version='0.1', description='Package to generate competition brackets', url='https:\/\/dhontecillas@bitbucket.com\/dhontecillas\/rankedcompos.git', author='David Hontecillas', author_email='dhontecillas@gmail.com', license='MIT', packages=['rankedcompos'], zip_safe=False) and you can check what you have installed in your virtual environment:\n[your_virtualenvionment_dir]\/lib\/python2.7\/site-packages\/rankedcompos and the egg information about it\n![egg info for checked out git]({filename}\/images\/2016_10_30_moving_out_code_01.png) My PKG-INFO contains:\nMetadata-Version: 1.0 Name: rankedcompos Version: 0.1 Summary: Package to generate competition brackets Home-page: https:\/\/dhontecillas@bitbucket.com\/dhontecillas\/rankedcompos.git Author: David Hontecillas Author-email: dhontecillas@gmail.com License: MIT Description: UNKNOWN Platform: UNKNOWN and if we list the intalled files, we will find that pip already precompiled the .pyc for us.\nWIP repository If we expect to do some work on the external (or separate) package, we can provide the editable flag, that will install the source in another place as source.\nLets give it a try! uninstall rankedcompos\nUninstall your package pip uninstall rankedcompos :::bash (venv)dhontecillas@devdebian:~\/prg\/sandbox\/piptest$ pip uninstall rankedcompos Uninstalling rankedcompos: \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos-0.1-py2.7.egg-info \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/__init__.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/__init__.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/brackets.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/brackets.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/compokeys.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/compokeys.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/elorating.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/elorating.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/genkeys.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/genkeys.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/seedbrackets.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/seedbrackets.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/superteam.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/superteam.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/teambrackets.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/teambrackets.pyc \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/tests.py \/home\/dhontecillas\/prg\/sandbox\/piptest\/venv\/lib\/python2.7\/site-packages\/rankedcompos\/tests.pyc Proceed (y\/n)? y Successfully uninstalled rankedcompos ![uninstallin the repo code]({filename}\/images\/2016_10_30_uninstall_project.png) Install with the editable flag This time to use the -e (editable) flag, you must specify the egg name you want to use at the end of the git url.\npip install -e git+ssh:\/\/git@bitbucket.org\/dhontecillas\/rankedcompos.git#egg=YourEggName And you will find that this time pip does installs it under your virtualenv\/src directory instead of virtualenv\/lib\/python2.7\/site-packages. The egg parameter that you provide in the url is the actual directory name that will be created under the src directory.\nAlso, the egg info dir is in the same base directory that your source code (instead of beeing in sibling directory), and it is an actual git clone of repo (instead of a bare checkout of source files that we previously had).\nAnd a link under the \/lib\/python2.7\/site-packages is created to point to that egg info dir under the src\/YourEggName.\nThis way you can work in the git repo, make updates to the source code, commits, etc .. and you won’t have to update the package every time.\nActually the point of using the -e flag makes more sense if instead of clonning from a remote repository, you give it the path to your local repo:\ngit clone git@bitbucket.org:dhontecillas\/rankedcompos.git my_ranked_compos pip install -e .\/my_ranked_compos This way you can develop in the package, while using it from your virtualenvironment. (Is kind of having a git submodule, but at the “package level”).\nUpdating the package The point is, how do we manage to get an update of the package ? First lets remove the editable package, performing a pip uninstall rankedcompos you wil find that only the link to the \/venv\/src that was created for the egg is removed. Now, you must delete by hand the repo that you have there (Unless it points to your local git repo).\nWe can pass wich branch, tag, or commit we want to checkout:\nIn order to upgrade the package is not enough to use the installed name, since it does not hold any reference to the original git repository, but the good part is that you do not need to change the version inside the setup.py file in order to get the latest version\npip install --upgrade git+ssh:\/\/git@bitbucket.org\/dhontecillas\/rankedcompos.git@v0.1 You only have to remember when doing a pip freeze to change your package name and version to the real url that you used.\nAlso, tag every new update accordingly with a different version number so it can be updated without the need of using the –upgrade flag.\nReferences There is the Python Packaging User Guide that has a complete explanation of how to package your libraries.\nStack overflow pip install from private git repo\n","permalink":"https:\/\/hontecillas.com\/posts\/moving_out_code\/","summary":"Creating modules from reusable units of code","title":"Moving out code"},{"content":"This is a draft of the flow process to subscribe to new BJJ events notifications that I want to have for Katagatame.com.\nIn order to check all the dialogs I prepared a unit test to fake the subscription process (with predefined user responses) so I can easily see how will it look when released in production.\n![subscription flow for the bot]({filename}\/images\/2016_09_15_katagatame_bot_test.gif) (Ehem! yes, those event images are only for the test :)\nYou must take into account, that until your app is not approved for subscription messages, before you run a test like this, you must send a message to your page so your server can respond to you. Check the platform policy overview.\nI’m writing this just some days after facebook released new additions to the bot, like the webview extensions with support for integrated payments that confirms that the approach of using facebook as simpler apps that can receive notifications is not a bad idea at all (specially for MVPs).\nOne cool addition that comes with that update is that quick replies now support sharing your locations. Currently I’m using a geocoder api call to transform addresses into longitude, latitude pairs, but I can benefit from this new button to know where the users are.\nAlso, we have a new share action. I will certainly implement this for the events. Those guys at facebook want to keep me busy!\nNeed for background tasks Once users are subscribed, I need to send them notifications every time a new event in their range is added. I should not do it in the same process that is handling the request, because this is going to be too time consuming (look if the event is inside the radius of each subscribed user, and then post the messages).\nSo, I decided to add Celery to the stack!\nThe most common configurations are with RabbitMQ (the recommended one) or Redis as message brokers. I decided to go with the second one, since I will use Redis for other stuff. Also for the current size of the project is more than enough (and less infrastructure to maintain).\nMy process is:\nManually setup it in the development environment (a Virtual Machine runnning in my laptop), that has the exact same configuration that will be on the production servers.\nOnce everything is running smoothly, I modify my ansible playbook, and deploy it to the demo site. And when the time comes, I will play it into the production site.\nSo, these are the steps without too much detail:\nInstalling Redis Although the official site recommends to build it from sources, I am going with the easy path of installing the one that comes with the official Debian distribution.\nsudo apt-get install redis-server sudo apt-get install redis-tools Once installed, we can check that it is already started\nsudo service redis-server status redis-cli ping pip install Celery In order to use Celery and redis, we can do it with a pip bundle instead of installing each one of the python packages\npip install -U celery[redis] I just followed the official celery with redis documentation for creating a celery worker process, setup a basic task that sends a facebook message to myself, and then bringing up the django shell to try to trigger the background task.\nDevelopment server ready Once I had the develpment server working as expected, I updated the pip_packages files for develop and production to reflect the required changes. (The demo site uses the production settings, and I use demo as staging too).\nUpdating ansible scripts So, at this point we need to install redis (that we were not using) and for that we create a new task file for ansible install_redis_db.yml that will be imported in the main.yml file.\n- name: ensure redis packages are installed apt: name={{item}} sudo: yes with_items: - redis-server - redis-tools - name: check redis server is running service: name=redis-server state=started enabled=yes sudo: yes To have celery workers running in background and ready to receive tasks have several options to deamonize celery.\nIn my case, I am going with the supervisord option, because I am already serving the django app using gunicorn and supervisord.\nFollowing the official documentation, you can reach the sample config file for celeryd, but if you prefer, there is a simpler example in the Thomas Sileo blog post about celery and supervisord\nMy setting looks very much like Thomas Sileo’s one, with the only addition of an environment variable that points to my current deployment dir, so it can find the django packages.\n[program:celeryd] environment=PYTHONPATH="{{app_code_dir}}:$PYTHONPATH" command={{virtualenv_dir}}\/bin\/celery worker --app=bgtasks -l info stdout_logfile=\/var\/log\/celeryd\/{{app_name}}_celeryd.log stderr_logfile=\/var\/log\/celeryd\/{{app_name}}_celeryd.err.log autostart=true autorestart=true startsecs=10 stopwaitsecs=600 killasgroup=true user=dhontecillas Run the playbook into the demo site and test With a simple:\nansible-playbook -i demo_hosts -K django_server_setup.yaml the deploymnet is done.\nI make a live test: I’ve put a simple message notifying myself (with my known facebook receiver_id hardcoded) when any user reports a new event.\nSo I report an event in the demo site, and … it’s there, the message poping in my facebook page ! :)\nOf course this test will be removed and notifications implemented properly.\n","permalink":"https:\/\/hontecillas.com\/posts\/bot_subcription_process_and_celery_tasks\/","summary":"Playing with the FB bot api to notify users about events.","title":"Bot subscription process and Celery tasks"},{"content":"Motivation When facebook announced their messenger bot API, and I had a look at the documentation, I thought that natural language processing is something that really requires effort and time, and that could take some time to have something useful on top of it.\nTo do it properly, it does not only requires to understand each sentence, but also to maintain the context of the conversation.\nI gave a try to wit.ai, but I did not find it so easy to use.\nBut when I read on of the Facebook developer blog a post about massenger platform policies and read that they opened the Subscription Messaging, I thought that could be a very nice substitute for native apps push notifications.\nI mean, if you do not have the time \/ money yet to invest into a native app, and you have a web app that would benefit from having notifications, why not implement them with an already existing app with a huge user base?\nThe facebook messenger platform has the nice feature of showing dialogs, with buttons, that can trigger posts to your endpoint with a payload that you set for each button.\n![nice dialog feature]({filename}\/images\/2016_08_28_fb_messenger_dialogs.png) This way we could use a very dumb interface, based on those dialogs to guide the user to complete a subscription for our notifications.\nFirst steps So, I decided to give it a try. Googled about it, and decided give a click to a couple of results:\nHow to build a Facebook Messenger bot using Django and Ngrok\nFacebook kickstart guide\nI mostly followed the steps on the basic tutorial, with the only difference of not using ngrok.\nInstead of that I used my running web server to forward a port with ssh, to my development virtual machine in my laptop, where I would run an stunnel process (a proxy to add TLS), with the certificate from my web server, that finally would connect to the running http django development server.\nSo, I had to enable port forwarding in the web server, by adding to \/etc\/ssh\/sshd_config\nGatewayPorts yes And then I run the command:\nssh -nNT -R 8443:my_dev_virtual_machine_ip:8334 remote_server And obviously, the webhook parameter in the Facebook developer control panel must include the port:\nhttps:\/\/remote_server:8443\/my_fb_endpoint You can find more about port forwardin in: Trackets Blog\nHowever, one of the main problems when using this approach is that the ssh connection drops very often (and with it the port forwarding, of course).\nI must admit that ngrok would have been an easier path.\nAlready existing python libraries After playing with the very basics of setting the endpoint, and redirecting it to my development server, with raw python code, I though it was worth to invest some time in finding existing libraries to manage the messaging part.\nI’ve found some pet project on github, a Flask targeted repo, and a couple of python libraries that could do the work.\nDavid Chua’s pymessenger (100 stars at the time of writing this in github: contributions graph )\nNam Ngo’s messengerbot (60 stars at the time of writing this in github: contributions graph )\nAlthough the first one has far more stars, looking at the contribution graph I found the second one had more recent contributions. But given that there is not a ‘widely used’ one, the best check is to look at the source.\npymessenger At first look it has less classes.\nIt has a single Bot class, that is used for sending different type of messages. Most of the function calls have inside hardcoded dictionaries to be converted to json. The problems is that there are not following the DRY, and instead of composing the different parts of the message, each call has repeated dictionary structures (look at this snippet from the code):\ndef send_generic_message(self, recipient_id, elements): payload = { 'recipient': { 'id': recipient_id }, 'message': { "attachment": { "type": "template", "payload": { "template_type": "generic", "elements": elements } } } } return self._send_payload(payload) def send_button_message(self, recipient_id, text, buttons): payload = { 'recipient': { 'id': recipient_id }, 'message': { "attachment": { "type": "template", "payload": { "template_type": "button", "text": text, "buttons": buttons } } } } return self._send_payload(payload) Two different functions with a lot of repeated dictionary parts.\nIt also has a FacebookGraphApi object, that is not needed for the messenger bots.\nAlso an empty Receipt class (I did not found where it is used). Also the UserProfileApi that uses the FacebookGraphApi to fetch the user profile. And a utils.py file for some hub signature validation and the app secret proof.\n(BTW: looking at its code I discovered the request_toolbelt package that is a nice feature that I will take into account for the future)\nmessengerbot This project uses a more structured way of providing the facebook messenger functionality.\nIt has its code spread among several files for different clases: Attachments, Elements, Templates, and so one. Each class has its well defined properties that nicely maps to the official facebook bots documentation\nAlso objects have more validation for its properties, like checking for the maximum length of the message or the button title, raising the appropiate exceptions. Generally speaking, the code looks much more pythonic to me. And the fact it does not have empty classes, or code that is not needed for the library functionality makes me feel more confident on the quality and the future of the library.\nSo, finally decided to go with messengerbot and give the repo a star (and also a little contribution :).\n","permalink":"https:\/\/hontecillas.com\/posts\/toying_with_facebook_bot\/","summary":"Playing with the FB bot api to notify users about events.","title":"First steps with the Facebook Messenger Bot API with Python"},{"content":"I imagine that this is always the very first post when you set up a blog that is not hosted in any major platform like Wordpress or Blogspot.\nI was looking for something simple, where I could use markdown to easily write posts. Since I have already toyed with Pelican, it covers my needs and I know Carlos Carrasco is already happy using it, i did not take too much time to evaluate alternatives like Nikola or Hugo Blog.\nFor hosting I use AWS S3: cheap and easy to deploy. (Using google you will find several posts about how to do it, like this one that also explains how to use the AWS CDN for your Pelican blog )\nI took the tuxlite_tbs theme and modified a couple of css colors for a minimal personalization.\niPython notebooks I sometimes test stuff with Jupyter notebooks, so I thought it would be nice to integrate them into the blog. I’ve followed the setup described in dataquest.io blog: How to setup a Data Science Blog but, since i’m not focusing on data science (most of my posts won’t use Jupyter notebooks) I preferred to use the second approach of using liquid_tags plugin to embed the notebooks.\nJupyter notebook test {% notebook setting_up_ipython_blog.ipynb %}\nSome python code test from datetime import datetime import json import logging _l = logging.getLogger('mfcommon.controllers') ERROR_INVALID_DATA = 'ERROR_INVALID_DATA' class CompetitionEventInformation(object): def __init__(self): pass # def _canFinishPlanification(self, valid_state_transitions, result): # if 'disabled_valid_state_transitions' not in self.result: # result['disabled_valid_state_transitions'] = {} # return result def transitionsInformation(self): valid_state_transitions = self.comp_ops.validStateTransitions() self.cese = CompetitionEventStatesExplanations() res = {} for x in valid_state_transitions: r = self.cese.stateInformation(x) res[x] = r return res def championshipInformation(self, championship): c = ChampionshipClashScheduler(championship=championship) is_scheduled = c.isScheduled() info = {'scheduled' : is_scheduled} if is_scheduled: info.update(c.scheduleFilesDict()) return info def print_a_and_b(a, b): print("{} -> {}".format(a, b)) And a little bit of C++ #include "bsdconnection.h" #include <string> #include <cstring> #include <cerrno> #include <sys\/types.h> #include <sys\/socket.h> #include <netdb.h> #include <netinet\/in.h> #include <unistd.h> using namespace std; bool BSDConnection::sendBuffer( unsigned long size, char *buffer ) { if( size == 0 ){ return false; } unsigned long offset = 0; bool err = false; while( !err && offset != size ){ long res = send( socket_, buffer + offset, static_cast<ssize_t>( size - offset ), 0); if( res >= 0 ){ offset += res; }else if( errno != EAGAIN ){ err = true; } } return !err; } And a little test image to finnish the first post:\n![I feel like I could take on the world]({filename}\/images\/2016_08_15_dott_tentacle.png) ","permalink":"https:\/\/hontecillas.com\/posts\/settingup_pelican\/","summary":"Setting up a pelican site generator blog for my personal web page, with ipython integration.","title":"Setting up Pelican Blog"},{"content":"","permalink":"https:\/\/hontecillas.com\/about\/","summary":"","title":"About"},{"content":"Live SendRules A backend system to send notifications to users.\nIt takes care of selecting the appropriate language for the recipient, and handling the user preferences for what channel to use to receive each kind of message (email, sms, telegram message).\nIt can also be seen as an internal proxy to different notification services providers, like twilio, sendgrid, mailgun, etc.. useful to avoid vendor lock-in.\nThis project was born after having seen different companies building this very same system in different ways and just with slightly different needs.\nSending notifications is a need that almost every company has, but it is rarely a core business feature, so, having a software that covers those needs would result in saving many hours of software engineering.\nTechnologies used: Go, Postgresql, Redis.\nOpensource code Handy Framework A small set of Go code that can be used to bootstrap a Web or an API.\n(Currently is work in progress and should not be used in production environments).\nTechnologies used: Go\nHandy Framework Ansible Ansible playbooks to publish dockerized apps in an small Virtual Private Server.\nOne for server provisioning One for app deployment The intention of these scripts are not to have production level services but to be able to deploy conteinarized apps into a VPS (for example to show case personal projects). So, the idea is to deploy a shared DB (postgresql) and a shared redis instance, and made those available to the running docker containers in the VPS.\nA container to provide a docker respository (backed by AWS S3) is used, as well as an additional container that checks the health status of others and respawn them in case those fail.\nTechnologies: Docker, Python, Ansible.\nDynLimits A proxy server for rate limiting request based on a per endpoint and api key (user).\nIt allows to have different rate limits for POST or GET endpoints, as well as dynamically changing those limits.\nLive API Checker The main idea behind this api proxy was to check the requests received agains an OpenAPI spec.\nThe main idea was to actually have a “coverage” of the requests made on integration tests (and know what endpoints have not been hit).\nBracket tool Tool to generate competition brackets.\nTechnologies: Python.\nAutobackup script A small python script tool, to check new and updated files in a directory and create a backup for them.\nTechnologies: Python.\nTwchess An old toy project to tweet chess boards and play by\nOffline old projects Alquilino.com A web page to keep track of prices of places to rent, scrapping the two main spanish sites: idealista.com and fotocasa.com (the api was put public in this pyappapi github repo).\nThe API used was extracted from reversing the android mobile apps, and several different servers (workers) were used at the same time to not hit the rate limit, extract the information and send it to a main server.\nOn the main server, the flat images and other data where used to identify duplicates of the same flate.\nTechnologies used: Python, Django, Postgresql\nKatagatame Katagatame was a website to help organize brazillian jiu jitsu competitions.\nIt helped with:\nHandling inscriptions: receive inscriptions payments through PayPal, and automatically assigns competitors to its corresponding categories. Generate brackets: automatically put to a maximum distance the competitors of the same team in the same category, and create clashes as diverse as possible, trying to avoid repeated team confrontations in the first round. Create a schedule: Given the brackets, it creates a schedule for each mat. Create a printable PDF with brackets and Excel file to keep track of results. It scrapped new upcoming competitions, and had a facebook bot to notify those new events. Technologies used: Python, Django, Postgresql\n","permalink":"https:\/\/hontecillas.com\/projects\/","summary":"

Live<\/h1>\n

SendRules<\/a><\/h2>\n

A backend system to send notifications to users.<\/p>\n

It takes care of selecting the appropriate language for the\nrecipient, and handling the user preferences<\/strong> for what channel to\nuse to receive each kind of message (email, sms, telegram message).<\/p>\n

It can also be seen as an internal proxy to different notification\nservices providers<\/strong>, like twilio, sendgrid, mailgun, etc.. useful to\navoid vendor lock-in.<\/p>\n

This project was born after having seen different companies\nbuilding this very same system in different ways and just with slightly\ndifferent needs.<\/p>","title":"Projects"}]