Improved logging support

Topics:

• Logging drivers
• Initial logging drivers
• Default driver improvements

Logging drivers

The driver interface should be able to support the smallest subset available for logging drivers to
implement their functionality. Stdout and stderr will still be the source of logging for containers
in this proposal. Docker will, however, take the raw streams from the containers and create discrete
messages delimited by writes. This parsed struct will then be sent to the logging drivers.

type Message struct {
    // ContainerID is the container id where the message originated from
    ContainerID string 

    // RawMessage is the raw bytes from the write
    RawMessage []byte 

    // Source specifies where this message originated, stderr, stdout, syslog
    Source string

    // Time is the time the message was received
    Time time.Time

    // Fields are user defined fields attach to the message
    Fields map[string]string
}

type Driver interface {
    // Log begins the logging of the stdout and stderr streams for a specific id
    Log(message *Message) error

    // ReadLog fetches the messages for a specific id
    ReadLog(containerID string) (messages []*Message, err error)

    // CloseLog tells the driver that no more log messages will be written for the specific id
    // drivers can implement this to their requirements, it may mean compressing the logs or deleting
    // them off of the disk
    CloseLog(containerID string) error

    // Close ensures that any writes for the logger are properly flushed and can be
    // stopped without data loss
    Close() error
}

When creating or initializing the drivers they will be provided with a key/value map with the user defined configuration specific to the driver. Each driver will also be provided a root directory where it is able to store and manage any type of state on disk.

Initial logging drivers

none - This driver will ignore the streams and log nothing for the containers. This is a totally valid
driver as the docker daemon has to manage the logs for all container it's a memory and performance bottleneck
on the daemon.

default - This driver will be the current implementation of logs that docker currently has. It is a single
file on disk with json objects with the message, timestamp, and stream of the log message separated by a
new line char.

syslog - This driver will write to a syslog socket and use the tag field to insert the container id.

Default driver improvements

One of the biggest issues with the default driver is that there is no log truncation or rotation. Both of
these issues need to be addressed. We can either truncate based on filesize or date. I believe filesize
is better.

Truncation size can default to 10mb with an option when you select the driver to specify additional options.
Rotation can also be set a specific size limit defaulting to 500mb. To change the defaults I propose a
--logging-opt flag on the daemon, similar to --storage-opt for the storage drivers.

Usage

The usage for this feature will be managed via the daemon:

docker -d --logging none
docker -d --logging default --logging-opt truncation=20mb --logging-opt rotation=1gb