Skip to content

Docker exec commands leak memory in docker daemon #12899

@briangough

Description

@briangough

Please read the following comment: #12899 (comment)

This looks like an issue with data structures which aren't being freed.


original issue report

We have long running containers which we access with docker exec commands periodically (e.g. every minute) to run scripts. When the scripts produce output we see the docker daemon growing in size until it consumes all available memory.

The problem is reproducible with a simple script that starts a container and calls docker exec repeatedly with a command that generates output . With 10000 exec calls each producing 100KB of output on stdout we see the docker daemon memory usage grow by ~2.5GB.

memusage

Here's the script that produced the data above

sudo restart docker
pid=$(pgrep -f '/usr/bin/docker')  # pid of docker daemon
container=$(docker run -d ubuntu sleep 1d)

rm -f size.log

function log_memory {
    ps --no-headers -o size -p $pid >> size.log
}    

log_memory
for i in $(seq 1 10000) ; do
    docker exec $container dd if=/dev/zero bs=100k count=1
    log_memory
done

docker kill $container
log_memory

The memory growth increases with the amount of stdout from the command, so may be related to #7757 (comment)

If the command has no output then the leak is negligible in comparison, only 20MB for 10000 exec calls.

I've tried with and without log-driver=none and the results were the same. Removing the container after the test above has finished does not reduce the memory usage of the daemon either.

We use the docker-1.6 Ubuntu packages from docker.com

Test system details:

$ uname -a
Linux Lenovo-H530s 3.13.0-51-generic #84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

$ docker -D info
Containers: 467
Images: 405
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1339
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-51-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 7.715 GiB
Name: Lenovo-H530s
ID: 7TYM:DIRX:YWLF:SPT5:S2AK:F7WS:3MU2:G2TT:W2PE:CBMA:JRPA:JK4L
Debug mode (server): false
Debug mode (client): true
Fds: 11
Goroutines: 15
System Time: Thu Apr 30 12:58:22 BST 2015
EventsListeners: 0
Init SHA1: 9145575052383dbf64cede3bac278606472e027c
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/1.6

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions