However I think it's good, I'm not sure that we should use format with @ for this. But it's just my opinion. I think we can start work on code. WDYT @cpuguy83 ?

Also thinking msgpack or protobuf.

I'm not too fond of yet another (JSON) log format. We already have many log drivers that outputs processable logs, why do we want to have another one ?

@dqminh it's basically to make this "pluggable" so we don't have to keep adding more and more drivers to the codebase.

@cpuguy83 i would argue that any of the current structured-based log driver ( gelf / fluentd / even journald ) is already pluggable, although each having pros/cons of their own.

If we decide to have another pluggable one, i would like to see its benefits outweighs the rests.

@dqminh I think there is value in being able to blast out the logs in the "native" structure we have, (logging) service providers can implement their own adapters without being bound by shortcomings in the currently supported formats.

Most logging services implements one (or likely multiple) formats (like syslog and journald) so that they can consume logs from just about any system that exists, but these are lowest common denominator, which is really icky.

However I think it's good, I'm not sure that we should use format with @ for this. But it's just my opinion. I think we can start work on code. WDYT @cpuguy83 ?

Actually it would probably be better to not emulate how logstash does it, especially if we're also going to support other output formats, such as msgpack/protobufs. That way we could try to make the structure of the data similar.

Also thinking msgpack or protobuf.

Would be pretty trivial to support different formats. Personally I think it's worth supporting JSON because the barrier to entry is really low.

@dqminh I think there is value in being able to blast out the logs in the "native" structure we have, (logging) service providers can implement their own adapters without being bound by shortcomings in the currently supported formats.

Exactly. I feel something like this should just dump as much data as possible, with the least amount of processing, and basically not be very opinionated about how the data should be used.

Thanks for the input so far.

Looks like we're okay implementing this, @cpuguy83 @LK4D4?

@adamtulinius are you already working on the code for this?

@adamtulinius are you already working on the code for this?

Yes, I have quite a lot of it implemented, I just chose not to push it before reaching some sort of consensus about what to do.

I should be able to get it rebased on current master and add it to the PR during this weekend.

Thanks @adamtulinius, we can continue discussing the details when the code is there

Also note this proposal btw; #18604

ping @adamtulinius, have you had time to look at this?

Sorry about the delay. Had managed to break my dev environment.
Anyways, i've pushed an implementation that is basically working, but I'm going to clean up the code the following days, and make golint happy.
Also, I'm in the process of implementing tests for delivering messages to tcp, udp and unix domain sockets.

I thought about something, though:
When log messages can't be delievered, what should the driver do?

1. drop messages and print an error
2. attempt delivery later on, either storing messages for X minutes, or storing up to Y messages for later delivery

Thoughts?

Thanks @adamtulinius !

ping @LK4D4 @cpuguy83 can you have a look at this, perhaps some thoughts on #18001 (comment) ?

ping @cpuguy83 @LK4D4 can you have a look please?

Looks good.

I would also want to make sure we can customize the fields that get sent much like we do with a couple of the other drivers. If not right away, at least in a way that we don't have to make a breaking change to the outgoing message format.

I still think we need to look at either using msgpack or protobuf to encode the messages... and of course json can be benched as well.
We need to make sure we can get these messages out of docker as quickly and w/ as low impact to the host as possible.

overall +1 on the design, but am wondering what are the drawbacks? how would performance be impacted? /cc @adamtulinius

Ping @adamtulinius @LK4D4: shall we move to code review?

I've created some (naive) benchmarks for encoding log messages for reference:

BenchmarkJSON-8      1000000          2254 ns/op         568 B/op          8 allocs/op
BenchmarkProtobuf-8  2000000           602 ns/op         336 B/op          5 allocs/op
BenchmarkMsgpack-8   1000000          2037 ns/op         384 B/op          6 allocs/op
BenchmarkFlatbuf-8   5000000           303 ns/op           0 B/op          0 allocs/op

https://gist.github.com/cpuguy83/ac5d8f87bce9c37c0009

@cpuguy83 hell, flatbuffers is pretty cool :)

<3 flatbuffers

Let's move to code-review, it seems pretty easy to extend later.

And was able to get protbuf a bit better than flatbuff with some tweaks to use an encoder instead of calling Marshal directly:

BenchmarkProtobuf-8  5000000           244 ns/op           0 B/op          0 allocs/op
BenchmarkFlatbuf-8   5000000           312 ns/op           0 B/op          0 allocs/op

@cpuguy83 Check out https://github.com/alecthomas/go_serialization_benchmarks/blob/master/README.md

ping @adamtulinius looks like this needs a rebase; also my can you address the comments so far?

ping @adamtulinius can you rebase and address the comments?

ping @cpuguy83 @LK4D4 should we carry this?

Maintainers meeting today;

Although we like this feature, we should look at this together with the general "logging plugin" proposal #18604. There's really an overlap in the two, and we should decide what kind of mechanism we want to use to have external plugins. We should be careful developing our own protocols, unless there are no other options.

It's possible that we arrive at the solution in this PR, but moving this forward without having a decision on the general logging plugin system would be bad, as we may end up with two things doing exactly the same.

In addition, we have the GELF driver, which is a standard solution and already can be used for many purposes that this PR tries to resolve https://docs.docker.com/engine/admin/logging/overview/#gelf-options

For now, I'm going to close this PR until we have agreement on the plugin design. Thanks so much for starting the work on this.

I've proposed a solution that lets docker stream the logs to a container you specify with the image option for a "custom" logging driver here. The container can implement whatever logic it wants to forward the logs to another destination.