Recent use-case with microservices using spring cloud and docker
Reacting to filesystem io events with Node.js
Reacting to filesystem io events with Node.js with the io-reactor framework
Docker container IP and port discovery with Consul and Registrator
Do you use Docker? Does your containerized app have the need to discover both its own IP and one or more mapped ports? How can another container access my exposed ports and how can I do the same of my peers? As it stands today, simple self discovery of your container's accessible IP and one … Continue reading Docker container IP and port discovery with Consul and Registrator
Reactive re-engineering with Akka
Everyone once in a while during the life cycle of any given piece of software comes that time where you have the opportunity to improve it in a major way....if that is, its lucky enough to still be in production. One particular system I've been involved with is responsible for processing a lot of data … Continue reading Reactive re-engineering with Akka
Hazelcast discovery with Etcd
I've used Hazelcast for years and have generally relied upon the availability of multicast for Hazelcast cluster discovery and formation (within a single data-center). Recently was faced with two things, expand the footprint into a non-multicast enabled data-center and secondly pre-prep the service for containerization where nodes will come and go as scaling policies dictate … Continue reading Hazelcast discovery with Etcd
Hazelcast discovery with Consul
I've used Hazelcast for years and have generally relied upon the availability of multicast for Hazelcast cluster discovery and formation (within a single data-center). Recently was faced with two things, expand the footprint into a non-multicast enabled data-center and secondly pre-prep the service for containerization where nodes will come and go as scaling policies dictate … Continue reading Hazelcast discovery with Consul
Aggregate, backup elasticsearch fs snapshots across a widely distributed cluster
One of the Elasticsearch clusters I've worked on is spanned across multiple data-centers around the world and stores some very large indexes. Sometimes, but not often we have the need to get a backup of one of these indexes off of the cluster for restoration onto another cluster, but due to the sheer size of these … Continue reading Aggregate, backup elasticsearch fs snapshots across a widely distributed cluster
Book review: Building Microservices
Recently I read Sam Newman's "Building Microservices" , at ~280 pages its a fairly quick read. The reviews on this book overall are mixed and I can see where readers are coming from. By the title of this book one might expect some coverage of some of the microservices frameworks out there, concrete examples, maybe … Continue reading Book review: Building Microservices
Fix: HDP, YARN, Spark “check your cluster UI to ensure that workers are registered and have sufficient resources”
Are you trying to submit a Spark job over YARN on an HDP Hadoop cluster and encounter these kinds of errors? (below) If so just add the following 2 lines to your [spark-home]/conf/spark-defaults.conf file: ERRORS You will see the errors below, stem from the root issue that occurs on an Spark Executor node where … Continue reading Fix: HDP, YARN, Spark “check your cluster UI to ensure that workers are registered and have sufficient resources”
Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1”
Recently have been playing with HortonWorks HDP 2.2. Was starting to configure some oozie workflows and when submitting the job the first step's Hive script failed with this error and stack. To fix this, SSH into your HDP instance VM and edit: /etc/hadoop/conf/core-site.xml and change the following config to add "localhost". Save and restart the relevant services … Continue reading Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1”
bits.of.info 