About Me
> Eduardo Silva
● Github & Twitter: @edsiper
● Personal Blog : http://edsiper.linuxchile.cl
> Treasure Data, Inc.
● Open Source Engineer
Fluentd / FluentBit
> Projects
● Monkey Server / monkey-project.com
● Duda I/O / duda.io
Logging
Logging Matters
Pros
● Applications status
● Debugging
● General information about anomalies: errors
● Troubleshooting / Support
● Local or Remote (network)
Logging Matters
From a business point of view
● Input Data → Analytics
● User interactions / behaviors
● Improvements
Assumptions
Logging Matters
Assumptions
● “I have enough disk space”
● “I/O operations will not block”
● “It's fast to write a simple message”
● “Log formats are human readable”
● “My logging mechanism scale”
Logging Matters
Assumptions
Basically, yeah.. it should work.
Concerns
Logging Matters
Concerns
● Logs increase = data increase
● Message format get more complex
● Did the Kernel flush the buffers ? (sync(2))
● Multi-thread application ?, Locking ?
● Multiple Applications = Multiple Logs
Logging Matters
Concerns
If... Multiple Applications = Multiple logs
Multiple Host x Multiple Applications = ???
OK...so:
1. Logging Matters
2. It's really beneficial
3. ???
but...
It needs to
be done right.
Logging
Common Sources / Inputs
● Applications Logs
• Apache
• Nginx
• Syslog (-ng)
● Custom applications / Languages
C, Ruby, Python, PHP, Perl, NodeJS, Java...
How to parse/store
multiple data inputs ?
note: performance matters!
Fluentd is an open
source data collector
It let's you unify the data collection for a better
use and understanding of data.
Fluentd
Highlights
● High Performance
● Built-in Reliability
● Structured Information/Logs
● Pluggable Architecture
● More than 300 plugins (input/filtering/output)
Fluentd
Before
Fluentd
After
Configuration
Who uses Fluentd
in Production ?
Logging and other
use cases
We collect
Logging and other
500.000 events
use cases per second!
Logging and other
use cases
Logging
More use cases
We talked mostly about Server applications, let's talk now about
other contexts:
● Internet of Things (IoT)
● Embedded Applications
● OS Containers (Docker)
The fun continues ;)
Internet of Things
(IoT)
Internet of Things
Facts
● IoT will grow to many billions of devices over the next decade.
● Now it's about device to device connectivity.
● Different frameworks and protocols are emerging.
● It needs Logging.
Internet of Things
Frameworks
In order to solve the device to device connectivity and proper
messaging, multiple framework/stacks are emerging, all of them
backed by different companies and vendors. The most
remarkable at the moment are:
Brillo, by Google
Just announced, no logo..sorry!
Internet of Things
Logging
● How do we approach data collection for IoT ?
● How can we improve our IoT based product if we have data
from thousand of devices with multiple sensors, meaning
many inputs ?
Embedded
Embedded
Importance of the Embedded Market
● Embedded devices are everywhere:
● Medical Devices
● Automotive
● Home Automation
● Many...
Fluentd
for Embedded ?
Fluent Bit
Objective
Open Source data collection tool for Embedded Linux:
● Services
● Sensors / Signals / Radios
● Operating System Info
● Automotive / Telematics
Fluent Bit
Requirements
An Embedded environment requires special handling, specifically on
performance and resources utilization, for hence it needs to be:
● Lightweight
● Written in C Language.
● Customizable, pluggable architecture.
● Integrate with Fluentd
Fluent Bit
Requirements
● Support custom Inputs
● Support custom Outputs
● Use Binary Serialization (MessagePack)
● Configurable (though file system files)
Features
Fluent Bit
Features
● Collection & Distribution.
● Built-in system metrics.
● C API for Developers (WIP).
● Integration with third party services.
● Open Source / Apache License v2.0
Fluentd Integration
Fluent Bit
Integration
Direct Output
Fluent Bit
Direct Output
Examples
Fluent Bit
Built-in Metrics: CPU usage
Fluent Bit
Kernel Log Messages
Fluent Bit Configuration Example
Roadmap
Fluent Bit
Roadmap
● Library mode.
● Support a stock of sensors (inputs).
● HTTP input.
● Release first stable version, work in process!
● Documentation.
Thanks!
http://fluentd.org
http://fluentbit.io
github.com/fluent/fluentd
github.com/fluent/fluent-bit We are Hiring!
Eduardo Silva
@edsiper / [email protected]
