Showing posts with label distributed-computing. Show all posts
Showing posts with label distributed-computing. Show all posts

Saturday, June 8, 2013

Riemann and Bernhard, a distributed systems monitor and Python client


(Updated the post to fix two errors; see below.)

Riemann is a monitoring tool for distributed systems.

https://github.com/aphyr/riemann

It seems to be named after the famous mathematician Riemann:

http://www.google.com/m?q=riemann

Riemann and its configuration files are written in Clojure.

I saw Riemann recently and took a brief look at it, also looked at some of the related links below, and found it pretty interesting.

A brief overview of Riemann:

You write a Riemann configuration file in Clojure.

This file describes what events from what systems (i.e. hosts on your network) you are interested in, and how you want Riemann to handle them.

Though Clojure is a Lisp, the Riemann config file syntax is easy to understand, even without looking at the documentation (for simple uses, anyway, such as in the easier examples).

Processing can include things like summarization (within or across hosts, event types, threshholds, etc.), grouping, filtering, emailing alerts to concerned entities based on events or the (conditional) results of processing events, and even some support for taking action on events, such as restarting a process that has failed.

Riemann also has integrations with Graphite and Librato Metrics.

(It was interesting to see that Librato is headed by Fred van den Bosch, who was earlier a senior technology executive (CTO, SVP) at Veritas Software (now part of Symantec), where I worked earlier.)

Riemann clients, which can of course be servers of various kinds, send events to Riemann using (Google's) Protocol Buffers, over TCP or UDP.

http://en.m.wikipedia.org/wiki/Protocol_Buffers

http://code.google.com/p/protobuf/

Client libraries for Riemann are available for several popular languages, and there is a guide to writing your own client.

http://riemann.io/clients.html

Bernhard is a Python client library for Riemann.

https://github.com/banjiewen/bernhard

It seems straightforward to use for simple cases:
you import the Client class from Bernhard, create an instance of it, and call methods on it, to send events that are of interest to Riemann, to be processed and acted upon.

Riemann also comes with a web dashboard written with Ruby and Sinatra.

(Updated the paragraph below for errors in the spelling of the Riemann author's name, and the names of the big users.)

I wrote to Kyle Kingsbury (@aphyr on Twitter), the creator of Riemann; he said that there are at least a few big Riemann users, like The Guardian and the New York Times.

Related links for Riemann:

https://news.ycombinator.org/item?id=4962849

A Python wrapper for Riemann that used Bernhard:

https://github.com/exoscale/python-riemann-wrapper

http://www.spootnik.org/entries/2013/05/21_using-riemann-to-monitor-
python-apps.html

A Node.js tool inspired partly by Riemann:

http://blog.nodejitsu.com/waiting-for-godot

- Vasudev Ram
dancingbison.com

Sunday, October 28, 2012

Performance: ZeroMQ: throughput is not the inverse of latency


By Vasudev Ram

Interesting study of performance, throughput and latency (among other things) in the chapter about the ZeroMQ asynchronous messaging library, in the book Architecture of Open Source Applications (Vol. 2), which I blogged about recently.

See Section 24.3. Performance, in that chapter, for the stuff about throughput and latency.

- Vasudev Ram - Dancing Bison Enterprises


Tuesday, September 4, 2012

The Spread toolkit for distributed computing

By Vasudev Ram


The Spread toolkit seems interesting. I just saw it. It is a toolkit for distributed computing. It supports C/C++, Java and Python.

Excerpts from the site:

[ Some of the services and benefits provided by Spread:

Reliable and scalable messaging and group communication.

A very powerful but simple API simplifies the construction of distributed architectures.

Easy to use, deploy and maintain.

Highly scalable from one local area network to complex wide area networks.

Supports thousands of groups with different sets of members.

Enables message reliability in the presence of machine failures, process crashes and recoveries, and network partitions and merges.

Provides a range of reliability, ordering and stability guarantees for messages.

Emphasis on robustness and high performance.

Completely distributed algorithms with no central point of failure. ]

Spread Credits (people who worked on Spread.)

The Spread license.

Spread Concepts, the company behind the Spread toolkit.

About Spread Concepts.

The Spread Concepts Team. They seem to have good backgrounds.

- Vasudev Ram - Dancing Bison Enterprises

Friday, August 31, 2012

ICE from ZeroC, a cooler and lighter CORBA

By Vasudev Ram



I had come across ICE from ZeroC some time ago.

[ UPDATE: Customers of ICE / ZeroC. They seem to be doing well since I last checked them out. The customers include (mentioning only some well-known ones): GE Healthcare, Digium (creator of Asterisk, open source PBX/telephony software, Lockheed Martin, SGI, Northrop Grumman, Hewlett-Packard, and Boeing/SAIC. ]

ICE - Internet Communications Engine, from ZeroC.com, is (excerpts):

[ a distributed computing platform with support for C++, .NET, Java, Python, Objective-C, Ruby, PHP, and ActionScript. Ice is used in mission-critical projects by companies all over the world.

Ice is easy to learn, yet provides a powerful network infrastructure and vast array of features for demanding technical applications.

Ice is free software, available with full source, and released under the terms of GNU General Public License (GPL). Commercial licenses are available for customers who wish to use Ice for closed-source software. ]

It also has support for Android, the .NET Compact Framework, Red Hat Enterprise Linux 6 and Amazon Linux.

- Vasudev Ram - Dancing Bison Enterprises

Monday, August 13, 2012

Inferno on Disco, Python MapReduce library / daemon for structured text

By Vasudev Ram


Inferno is an open-source Python MapReduce library. It has (from the site):

[ A query language for large amounts of structured text (CSV, JSON, etc).

A continuous and scheduled MapReduce daemon with an HTTP interface that automatically launches MapReduce jobs to handle a constant stream of incoming data. ]

Overview of Inferno.

This overview page has a nice serial example: starting with a small set of test data, it shows how to query for a certain result, in SQL and then in AWK (both are easy one-liners), but then goes on to show how the achieve the same result using Inferno.

The interesting point is that the Inferno code is also small (a "rule" of ~10 lines, presumably stored in a config file) and a one-line command, but the difference from the SQL and AWK examples is that this runs a Disco MapReduce job to distribute the work across the nodes on a cluster. There is almost nothing in the Inferno code to indicate that this is a distributed computing MapReduce job.

Inferno uses Disco.

Disco is "a distributed computing framework based on the MapReduce paradigm. Disco is open-source; developed by Nokia Research Center to solve real problems in handling massive amounts of data."

Some users of Disco: (Chango, Nokia, Zemanta). Chango staff seem to be the developers of Disco.

- Vasudev Ram - Dancing Bison Enterprises