{"id":7803,"date":"2015-10-15T12:15:02","date_gmt":"2015-10-15T09:15:02","guid":{"rendered":"http:\/\/www.webcodegeeks.com\/?p=7803"},"modified":"2015-12-16T11:04:47","modified_gmt":"2015-12-16T09:04:47","slug":"logfmt-log-format-thats-easy-read-write","status":"publish","type":"post","link":"https:\/\/www.webcodegeeks.com\/web-development\/logfmt-log-format-thats-easy-read-write\/","title":{"rendered":"Logfmt: A Log Format That’s Easy To Read and Write"},"content":{"rendered":"

You don\u2019t need to be a logging expert to know that when it comes to logging in applications, there\u2019s a wide variety of options to choose from.<\/p>\n

There\u2019s the Common Log Format<\/a>, the Combined Log Format<\/a>, and Nginx\u2019s log format<\/a>; and on and on the list goes.<\/p>\n

But are any of these really the right choice? Honestly, what\u2019s right and wrong is debatable. It\u2019s something that varies from application to application. What\u2019s more, each of the aforementioned formats, as well as the countless others, has its strengths and weaknesses. But are any of them the right one \u2014 at least for you?<\/p>\n

Some formats allow for logs which contain copious amounts of information, but then they can turn into an unintelligible mess when it comes time to read them.<\/p>\n

Others are simpler, yet don\u2019t allow for as much information to be stored. You can be left without enough information when it comes time to parse them. Dumping a JSON object object can be a good thing, but it too has its drawbacks.<\/p>\n

Wouldn\u2019t it be great if there was one that was pretty easy to both read and write, both by humans and by computers? That sounds a little like a Holy Grail of sorts.<\/p>\n

However, as part of an application I was developing lately, I came across a log format which, whilst not completely fitting the bill, still does a pretty good job of working for both humans and computers. It\u2019s called logfmt.<\/p>\n

What Is logfmt?<\/h2>\n

Logfmt, in essence, simply logs data as a series of one or more key\/value pairs. No more, no less! This might seem simplistic, trivial even. But I find there\u2019s a certain beauty both in the simplicity and flexibility of the format.<\/p>\n

There aren\u2019t a set of predefined options, such as in the Common Log Format, which are the only ones you\u2019re able to use. Instead, you\u2019re able to specify exactly what your application needs on a case-by-case basis.<\/p>\n

This is great \u2014 you\u2019re able to create log entries to suit specifically what you\u2019re doing. Then, regardless of what\u2019s logged, the format\u2019s always the same. This makes it simpler to create parsers for the information stored.<\/p>\n

Let\u2019s look at a simple, concrete example so you can see what it looks like and how to get started. Specifically, I\u2019m using the Express framework for Node.js<\/a>. This example isn\u2019t the most sophisticated, but it shows a good example of using the library.<\/p>\n

var express = require('express'); \r\nvar logfmt = require('logfmt'); \r\nvar app = express();\r\n\r\nfunction getSomeEnvironmentData(req, res) \r\n{ \r\n  return { \r\n    baseUrl: req.baseUrl, \r\n    hostname: req.hostname, \r\n    protocol: req.protocol, \r\n    name: req.params.name \r\n  } ; \r\n}\r\n\r\napp.get('\/', function (req, res) { \r\n  console.log(logfmt.stringify(getSomeEnvironmentData(req, res))) \r\n  res.send('Hello World!'); \r\n});\r\n\r\napp.get('\/:name', function (req, res) { \r\n  console.log(logfmt.stringify(getSomeEnvironmentData(req, res))) \r\n  res.send('Hello World, ' + req.params.name); \r\n});\r\n\r\nvar server = app.listen(3000, function () { \r\n  var host = server.address().address; \r\n  var port = server.address().port;\r\n\r\n  console.log('Example app listening at http:\/\/%s:%s', host, port); \r\n});<\/pre>\n
Here I\u2019ve written a simple function, getSomeEnvironmentData()<\/code>, which takes the request and response objects and composes a user-defined object that stores the baseUrl<\/em>, hostname<\/em>, request protocol<\/em>, and the value of the :name<\/code> parameter, if passed.<\/p>\n
In both of the defined routes, the function is called and the return value passed to logfmt.stringify()<\/code>, which serializes the object to the logfmt format. This is in turn passed to console.log(<\/code>) for an easy way of logging the information.<\/p>\n
In more sophisticated setups, you could write to log files or remote logging services, such as Papertrail<\/a> or Splunk<\/a>. If you were to run the above code and make a request to either \/<\/code> or to \/matthew<\/code>, the following would be output to the console from which the process was initiated:<\/p>\n
baseUrl=\"\/\" hostname=localhost protocol=http\r\nbaseUrl=\"\/\" hostname=localhost protocol=http name=matthew<\/pre>\n
Now, in more production applications, there\u2019d likely be much more sophisticated instrumentation. This could include details about the logged in user, the dispatched route, as well as a variety of other information, about the current application\u2019s executing state. But for the purposes of a simple example, the above code works well.<\/p>\n
How to Parse Logs<\/h2>\n
Now that we\u2019ve looked at generating logs, how do you parse them? Parsing comes down to what you\u2019ve logged, so it\u2019s hard to provide a definitive, codified answer as to how to parse them.<\/p>\n
However, there are a couple of tools available, called hutils<\/a>. They\u2019re developed at Heroku<\/a>, and they help make the process simpler. There are four tools available in the set:<\/p>\n