An introduction to
node.js & Riak
Francisco Treacy
May 31st, 2010 @ ams.rb
Amsterdam
`whoami`
Building a web-based
collaborative e-reading
platform
J2EE veteran
Trumpetless wannabe trumpet player
@frank06 on the interwebs
Let’s look back, shall we?
How did we learn to program?
Does this sound familiar?
printf("What is your name?");
gets(string);
printf("Your name is: %s", string);
...it is how we learn how to program
So then we go build our first webapp...
<?php
$query = "SELECT astronaut FROM spacecraft";
$result = mysql_query($query);
while($result) { ... }
?>
(with the first language we come across)
Let’s imagine a bunch of people
trying to access your website
(run that code) at the same time
It’s like hosting a party
and provide only one toilet
© hopechurchokc / flickr
The cost of I/O
L1 cache: 3 cycles
L2 cache: 14 cycles
RAM: 250 cycles
Disk: 41,000,000 cycles
Network: 240,000,000 cycles
The cost of I/O
L1 L2 RAM Disk Network
The cost of I/O
In other words, reaching RAM is like going from
here to the Red Light District.
Accessing the network is like going to the moon.
So... what is your program still
doing while MySQL goes fetch
Neil Armstrong?
$result = mysql_query($query);
But we have threads!
(to do other stuff while we wait)
Programming with threads
Race conditions
Coarse-grained locks – more blocking
Fine-grained locks – more complexity
Risk for deadlocks
Hard to reason about and therefore to get
right
Context switching overhead
Programming with threads
A look at Apache and Nginx
Memory vs concurrent connections
http://blog.webfaction.com/a-little-holiday-present
A look at Apache and Nginx
Apache uses 1 thread per connection
Nginx doesn’t use threads
it runs an event loop
small memory allocation per connection
1 thread per connection is
limiting for massive concurrency
Event loops
Especially relevant when dealing with I/O
All code runs in a single thread
No need for multithreading – no locks!
Event loops
window.onload = function() {
alert("Apollo 11 landed!")
}
Rings a bell?
Javascript!
Fortunately, Google has been
developing a brand new
Javascript VM for Chrome...
It’s robust, blazingly fast, and open-source:
V8
The node way
A set of bindings to Google’s V8 Javascript VM
A purely evented, non-blocking infrastructure
that makes it super simple to build highly
concurrent programs
Ability to handle thousands of concurrent
connections with minimal overhead on a
single process
The node way
Why should I care?
I love Ruby
I can use EventMachine for async
And, after all, my Rails app doesn’t have
youporn’s traffic
The node way
“Libraries like eventmachine will never be
truly intuitive to use, because event-
driven I/O is enough of a fundamental
shift that it requires deep language
integration. Javascript, it turns out, is a
fundamentally event-driven language
because of its origins in the browser”
Adam Wiggins – Heroku
The node way
"Threads should be used by experts only"
Javascript is perfect for event loops with first
class function objects and closures
It is arguably the most popular programming
language
Full Javascript stack?
Full Javascript stack
Client-side
$.ajax({
url: '/api/feedme',
success: function(data) {
$('.result').html(data);
}
});
Full Javascript stack
Client-side – even better
var socket = new io.Socket('localhost');
socket.connect();
socket.send('some data');
socket.addEvent('message', function(data){
$('.result').html(data);
});
WebSockets through Socket.IO-node
Full Javascript stack
Server-side – node.js
http.createServer(function (request, response) {
response.writeHead(200, {'Content-Type': 'text/
html'});
response.end('I am back!');
}).listen(8000);
Full Javascript stack
Database
?
Introducing Riak
Content-agnostic key/value store
REST API embraces HTTP
Javascript Map/Reduce
Distributed, “assume that failures will happen”
Linearly scalable
Introducing Riak
What is scalable anyway?
(other than a buzzword)
€ throughput – cost predictability
Both up and down; less headaches for
operations and development
Brief Riak overview
db.save('astronauts', 'neil', {
name: "Neil Armstrong",
retired: true,
daysInSpace: 8,
missions: ['Apollo 11', 'Gemini 8']
})()
riak-js – http://github.com/frank06/riak-js
Available for node.js and jQuery
Brief Riak overview
var map = function(v, keydata, args) {
var result = [],
doc = Riak.mapValuesJson(v)[0]
doc.missions.forEach(function(mission) {
if (mission.match(new RegExp
(args.mission))) result.push(doc)
})
return result;
}
Map/Reduce jobs can be written in Javascript
and submitted via the HTTP interface
Brief Riak overview
db.mapReduce({
inputs: "astronauts",
query: [{ map:
{ source: map,
arg: { mission: "Apollo" } }
}]
})(function(results) { // ... });
Bring the computation to the data – map
phases are executed in parallel
Aggregation happens in one node
Brief Riak overview
db.get('astronauts', 'neil', {r: 2})(
function(neil) {
$('.result').html(neil)
})
Tunable N/R/W values to tweak CAP behaviour
Eventual consistency: Brief sacrifices of consistency in
failure conditions
“Choose your own fault tolerance/performance tradeoff”
Brief Riak overview
CAP theorem
Consistency: Reads and writes reflect a
globally consistent system state
Availability: System is available for reads
and writes
Partition tolerance: System can handle
the failure of individual parts
Brief Riak overview
CAP theorem
No real-world data store can serve
completely consistent data while
being 100% available and handling
disconnected networks
Internet
NGINX
Rails Rails Rails Rails Node
Collaborative platform for studying
Tens of thousands of books alongside with user
generated content
Highlighting, note-taking, sharing
Web-based, use it anywhere: laptop, phone, iPad
HTML5 + Javascript + node.js + Riak (and Rails!)
Expect a beta release in October
Riak is developer- and ops-friendly: it scales
down to your laptop as easily as up to a cluster –
especially during exams period!
Allows us to store multimedia assets just as JSON
Lucene-like search coming soon
Node is used for Ajax calls and WebSocket
Rails for the rest (it’s convenient and mature)
There are caveats though
Cutting-edge technologies are not bug-free
Riak still has some rough edges (some in terms of
performance)
node is approaching its first stable version
Async JS code can get “boomerang-shaped”
There are caveats though
Boomerang-shaped code
db.save(bucket, doc, content)(function(response, meta) {
db.get(bucket, doc)(function(response2) {
assert.equal(response2, content);
db.remove(bucket, doc)(function() {
db.get(bucket, doc)(null, function(r, meta3) {
assert.equal(404, meta3.statusCode);
db.get(bucket, other)(function() {
// ...
})
});
});
});
});
Step or flow-js
Address boomerang-shaped code
Step(
function readSelf() {
fs.readFile(__filename, this);
},
function capitalize(err, text) {
if (err) {
throw err;
}
return text.toUpperCase();
},
function showIt(err, newText) {
sys.puts(newText);
}
);
CoffeeScript
is a new language inspired by Javascript and Ruby
grade: (student) ->
if student.excellent_work
"A+"
else if student.okay_stuff
if student.tried_hard then "B" else "B-"
else
"C"
eldest: if 24 > 21 then "Liz" else "Ike"
CoffeeScript
compiles down to Javascript
var eldest, grade;
grade = function(student) {
if (student.excellent_work) {
return "A+";
} else if (student.okay_stuff) {
if (student.tried_hard) {
return "B";
} else {
return "B-";
}
} else {
return "C";
}
};
eldest = 24 > 21 ? "Liz" : "Ike";
And can be run on node!
There’s no doubt about it.
Javascript as a platform
is serious stuff.
Full JS stacks will become
more and more popular.
Maybe it’s about time
we started teaching this?
puts("Enter your spacecraft:")
gets(function(s) {
puts("You’re flying your " + s)
})
Go fetch a beer –
see you in a bit for a
demo and questions!
http://nodejs.org
http://howtonode.org
http://nodejs.org/cinco_de_node.pdf
http://www.slideshare.net/marcusf/nonblocking-io-
event-loops-and-nodejs
http://dailyjs.com/2010/05/26/realie
http://wiki.basho.com
http://jashkenas.github.com/coffee-script/
http://widescript.com
http://franciscotreacy.com
