Background on logging

A long long time ago, docker only supported one method of logging for containers. This is now affectionately known as jsonfilelog. There was no support for logging to external services (be it remote endpoints or a local out of process service like journald).

The json logger itself also had many performance related issues: excessive memory allocations, slow (reflection-based) json marshalling, and leaking resources. It also did not support log rotation, which means that every container's log file would just grow unbounded and no (good) way to rotate it without data loss.

Because of this we introduced the ability to disable logging as well as the concept of logging drivers and slowly started adding new loggers for external services (syslog, journald, cloudwatch, etc).

The performance issues have been partially dealt with (see the hack that is pkg/jsonlog) and of course resource leaking dealt with. Log rotation support added (but not enabled by default).

At this point most everything is great with logging... unless you need to run docker logs, since only jsonfilelog and journald support reading logs. For this reason many people just use the built-in json logger and consume the logs API to ingest into another service.

Proposal

Dual-logging

Always have containers log to a local logger that supports reading if/when the user selected log driver does not support reads in addition to the selected logger. This will allow users to have a consistent experience regardless of the logging driver selected.

This is not allowing users to specify two(or multiple) log drivers, but rather to log locally in addition to the user specified log driver

An issue with this...the json logger still has some performance issues... json marshalling is slow and still makes several allocations (I think around 3) per log entry, and can potentially use up loads of disk-space unexpectedly.

Replace jsonfilelog

For this reason, I also propose replacing the jsonfilelog driver with a new driver: local.
jsonfilelog is tied to json. We've tweaked the hell out of logging to json for container logs but it's still not ideal (for performance reasons) and really awful/fragile code.

The local logger would, like jsonfilelog, log to a local file, but would not be tied to any particular encoding format. This is possible by encoding the version/format into the top of each log file. This can be a binary encoded version, with the version corresponding to the version of the driver's encoding format (this could encapsulate both changes to the underlying encoding itself as well as major changes to log data). Doing this on each file allows Docker to support mixed versions for a single container.

Version 1 of this format would be encoded as protobuf following the same format as used for logging plugins: https://github.com/moby/moby/tree/master/api/types/plugins/logdriver.

This encoding format does not require any allocations and requires far less CPU than json.
Another plus is the protobuf format uses something like 1/3-1/2 less space (in my tests) on disk compared to the same output in json.

The new driver should also have sensible defaults for rotation (i.e. rotate by default) and rotated files should also be compressed (by default) to further save space on disk.
When a container is only logged locally the sensible thing to do is to not ever remove log data (unless specified), but when logging to external services it should be ok to clean out older files.

Make local the default

Ideally this new local log driver would become the default logger since it should be better in absolutely every way than jsonfilelog. jsonfilelog should then be deprecated, and eventually removed.

Some problems here may be with users who are directly accessing container log files, expecting a particular format, place on disk, etc... this sucks... but ultimately on-disk state is not part of our API and people should not be messing around in there.... if they are, they should expect to be easily broken between versions.

• Implement new local logger - Add "local" log driver #37092
• Implement dual logging