Any comment from upstream? This would be a really nice feature. I already have logging people asking me about it.

Think they've been busy bug-hunting for the release.

I get Unrecognized input header with this driver. Also I have double output for json-file log-driver on first run of docker logs -f.

I can't reproduce the first symptom; can you give me more details about how you're getting it to happen? The duplicate log data doesn't appear to be driver-specific (now that I know what to look for, I'm seeing it with the journald driver, too), but it goes away if I revert 153f98b.

@nalind I run

docker run -d --log-driver=journald busybox ping google.com

and then just do docker logs -f on that container. Nothing in daemon logs also.

Ah, thanks! It's still not giving me the Unrecognized input header error, though it does show me the first few lines of the output twice if I run docker logs -f quickly enough.

It looks as though, if the client and daemon disagreed about whether or not the container had a tty allocated to it, that the data stream coming back from the server wouldn't be muxed using an stdcopy.StdWriter, so it wouldn't be in the format that stdcopy.StdCopy expects to demux, and that should cause the Unrecognized input header message to appear, but I'm at a loss as to how to actually trigger it.

@nalind I'm using systemd-220. Maybe it returns crap instead of logs sometimes.

I'm also running with version 220 (Fedora 23's systemd-220-3.fc23), but the like messages we retrieve from json-logs, data we read back from the journal should be wrapped up and chunked before it gets to the client, so if the driver was getting garbage back, I'd expect to get garbage in the output of the docker logs command rather than an error parsing the HTTP stream that's carrying it. Is there anything else that might be making a difference?

@nalind I can't imagine. I'll try to put some logging to your code later.

Ah, found it. When built without the journald tag, which the build scripts try to enable automatically when the libraries are detected, this is triggered when the daemon sends back an error message indicating that it couldn't read the logs, because it sends that message back without the framing that the client is expecting.

The Dockerfile installs libsystemd-journal-dev, and hack/make.sh tries to check for it, but that check fails because pkg-config isn't also available, so the journald build tag wasn't being enabled. Adding it as a requirement to the top-level Dockerfile should handle most of the cases that the changes to contrib/builder/deb/generate.sh and hack/make/.build-rpm/docker-engine.spec didn't already cover.

As for the formatting of the error when it's sent back to the client, if ContainerLogs replaces the OutStream member of the ContainerLogsConfig that it gets with its wrapper, and getContainersLogs sends the error message through that, the client displays the error that the server sent.

I'm tweaking the patch set to fix both of these problems and rebasing against the current master.

Rebasing to resolve reported merge conflicts.

Catching up with the new fluentd driver.

ping @LK4D4 have you been able to successfully test this?

Hi, the log reader interface was refactored and should (I hope!) be much simpler to implement now, as all you need to do is implement a ReadLogs function on journald.

Yes, I waited for this refactoring. Now implementation of journald logs should be simpler and more robust.

The current refactor keeps knowledge of the number of lines to read when it's in "last" mode, and the cutoff time in "since" mode, in ContainerLogs. The journald driver could definitely use that information to skip sending data that is only going to be ignored by ContainerLogs, and I suspect other log readers would, too. (The current ContainerLogs also spends some time dealing with file numbers, which really seems to me like a jsonfilelog-specific notion, but a reader can ignore that.)

@nalind Not anymore. ContainerLogs does nothing except call the ReadLogs on the configured logger and Listens for logger.Messages on a channel.

@cpuguy83 Oh, you're right, that's new today, and it's much better.

@dmp1ce Are you getting error or not? I'm not sure how your docker is built, but with --log-driver=journalctl all should work fine.

@LK4D4 Sorry, I AM getting the error. I updated my comment with the removed "not".

I'm installing using the instructions here: https://docs.docker.com/engine/installation/ubuntulinux/

I'm using the journald driver using Docker Compose and docker run --log-driver=journald. I'm not using --log-driver=journalctl.

@dmp1ce it should be fine if binary built as dynamic with journald build-tag. I think @jfrazelle or @tianon know more than me about which binary we ship for ubuntu.

I am using official docker-engine package for jessie and getting this message too:

Error running logs job: configured logging reader does not support reading

ivan@36com10:~$ ldd `which docker`
    linux-vdso.so.1 (0x00007ffdec1b1000)
    libapparmor.so.1 => /usr/lib/x86_64-linux-gnu/libapparmor.so.1 (0x00007fe08b90a000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe08b6ed000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe08b4e8000)
    libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007fe08b29f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe08aef6000)
    /lib64/ld-linux-x86-64.so.2 (0x0000562f5b55f000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fe08acd0000)
    libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fe08aac1000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fe08a853000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe08a64a000)

Getting logs through journalctl works except for #15722.

@bobrik How do you start your container and what version do you use? Seems like works fine for me on master with

docker run -it --rm --log-driver=journald busybox ping google.com

I think it's packaging problem. The binary must be linked with libsystemd.
Am I right, @tianon ?

Interesting, I can reproduce on both 1.9.1 and 1.10.0-rc1 using the official docker-engine Debian packages (for debian-jessie specifically); both give something like:

Error running logs job: configured logging reader does not support reading

@tianon check ldd. Also, maybe we don't set journald tag.

Hmm, I think the build containers (per contrib/builder/deb/generate.sh) might not have pkg-config in them. I had erroneously assumed that they were pulled in by build-essential on Debian, and the various development tools groups.

If nm -CDu /usr/bin/docker | grep sd_journal_add_match turns up empty, then the logic's not getting compiled in.

Yeah, I don't see systemd or journal anywhere in ldd, so I'll be digging into the build containers next.

$ nm -CDu /usr/bin/docker | grep sd_journal_add_match
$ 

Into the build system I go! 😄

Man, I could've sworn pkg-config was part of build-essential also, but it's not there!

I have Ubuntu 15.10 running with Docker version 1.10.0. I'm still getting Error running logs job: configured logging reader does not support reading. Should this have been fixed?

Yes, it should have. When I poke at the packages in the apt repository, I'm finding the expected references to the right unresolved symbols in the 1.10.1 packages for jessie, precise, trusty, and wheezy, but not in the packages for stretch or wily.

On stretch and wily, the expected headers are included in libsystemd-dev, which the build scripts install, but there is no libsystemd-journal.pc for pkg-config to find when hack/make.sh calls it, so the logic doesn't get built at build-time. Given that those are newer versions, we might be better off changing pkg-config references to just look for libsystemd, and have make.sh look for a version of it that's at least 209, when systemd's news file says those were all merged together.

Whoops, packages for precise and wheezy don't link with the right libraries - I was taking their packages apart wrong. So only jessie and trusty are being built with the feature.

@nalind OK. Thanks. Is this already merged then for Wily? What about the future Xenial?

@dmp1ce I was mistaken in thinking that we could fix it by changing the pkg-config dependency (we'd stop pulling things in on older releases, and there's no single option that's present across all of the desired build targets), so that was closed without merging. So for the moment at least, it's not working in stretch or wily.

@nalind Should I create an issue for Wily?

@dmp1ce Yeah, I think that's how we make sure it stays on the radar.