OOM when following "local" logs of high-log-output container #42125
Description
Description
When logging to the local driver, Docker will quickly run out of memory when following the logs of a container with a lot of log output.
This also occurs when following local logs while using a driver such as syslog that supports dual logging.
This issue appears to be similar or identical to #39963.
Steps to reproduce the issue:
-
Run a container that logs heavily to the
localdriver:ID=$(docker run --rm -d --log-driver local alpine cat /dev/urandom) -
Follow the container's logs:
docker logs -f $ID > /dev/null
Alternatively:
-
Run a container that logs heavily to a driver that supports dual logging:
ID=$(docker run --rm -d --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:0 alpine cat /dev/urandom) -
Follow the container's logs:
docker logs -f $ID > /dev/null
Describe the results you received:
dockerd exits with fatal error: runtime: out of memory.
Describe the results you expected:
Docker remains stable and keeps a reasonable memory footprint.
Additional information you deem important (e.g. issue happens only occasionally):
I have been unable to reproduce the issue when logging to the json-file driver.
Output of docker version:
Client: Docker Engine - Community
Version: 20.10.5
API version: 1.41
Go version: go1.13.15
Git commit: 55c4c88
Built: Tue Mar 2 20:18:20 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 363e9a8
Built: Tue Mar 2 20:16:15 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: local
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-54-generic
Operating System: Ubuntu 20.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 981.1MiB
Name: ubuntu-focal
ID: 5X6F:5OLY:MQHX:I5T3:AVJQ:AMIX:5OBO:XI4L:CI62:LRA7:INKW:26IT
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 30
Goroutines: 40
System Time: 2021-03-08T20:11:15.100104538Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
This was initially observed in a Nomad environment. Nomad follows logs by default for Docker tasks. The host was a lightly loaded AWS m5.xlarge instance.
I have reproduced it on Ubuntu 16.04 and 20.04 using Virtualbox VMs, and on Docker for Mac 3.1.0 (20.10.2).