stream of engineering
Note, this is the final part of a two part series about this project; article #1 is here. Continuing on from where we last left off, now that we had a functioning collection engine producing full graphs of crawled data all the way down to interrogable dataset_items, it was now time to get down to … Continue reading Architecture for non-deterministic mass data collection: part 2: dynamic data lake schemas
Note, this is part one of a two part series about this project; article #2 is here. One of my more recent projects was spawned from a pretty interesting idea. The team wanted to build a system that would permit them to scour the Internet for information regarding a particular set of targets; a "target" … Continue reading Architecture for non-deterministic mass data collection: part 1: collection engine
Earlier this year I re-entered the rabbit hole which is the dizzying world of CI/CD platforms and solutions. Today's marketplace presents so many choices that I can only imagine how daunting it is for a newcomer to the space to decide on what solution to go with. Thankfully the the industry is starting to invest … Continue reading State management in Tekton pipelines with Slack interactive messages
This is a long overdue followup to my prior article titled "Reactive re-engineering with Akka" in which I described an approach to processing large amounts of data more efficiently by using reactive actors and how they were used to massively scale up an existing data synchronization platform that was starting to bottleneck. In this article … Continue reading Retrying failed operations
I've recently been working on a project where a set of application runtime configuration was stored in Consul by using Consul's key-value store functionality. The application could dynamically materialize its configuration at runtime by using consul-template which is another handy tool by Hashicorp. Consul-template permits you "watch" a Consul KV space and automatically render Golang … Continue reading git2consul alternatives: key-value filesystem syncing for Consul
CI/CD systems, for all their fascinating inner workings, complexity and inordinate amounts of configurability; at the end of the day, are really no different than any other piece of custom software out there. Just like any other software system, logically CI/CD engines are comprised of contracts for inputs, outputs, control flows and error handling etc. … Continue reading CI/CD control flow with Git commit message arguments
Slack "Slash Commands" can be a very useful tool to write a CLI to interact with other applications that you manage. In short the contract works something like this; from within a slack channel that the commands are available you would type: /my-command [text arguments] Seems simple enough, however you need to implement a REST … Continue reading Slack Slash Command security
Lets migrate everything to containers. Everything. The app infrastructure, the apps themselves, deployment workflows, DevOps procedures... everything. Recently started winding down one of those "on and off again" projects with long term goals but was constantly susceptible to the day to day distractions and diversions typical for any team managing a ton of different apps. … Continue reading Migrate everything to containers, you will
Everyone once in a while during the life cycle of any given piece of software comes that time where you have the opportunity to improve it in a major way....if that is, its lucky enough to still be in production. One particular system I've been involved with is responsible for processing a lot of data … Continue reading Reactive re-engineering with Akka
This is a book review for "97 Things Every Software Architect Should Know" by O'Reilly with dozens of contributors. This book is a quick read at roughly 200 pages and is targeted towards those folks who find themselves in the role of the "software architect". Many of the contributors will be names you recognize such … Continue reading Review: 97 Things Every Software Architect Should Know
bits.of.info 