Skip to content

Docker cgroup memory limits bypassed for mmap'ed tmpfs files #39004

Description

@maekrbot

Description
Docker cgroup memory limits bypassed for mmap'ed tmpfs files.

Steps to reproduce the issue:

  1. docker run --rm -ti -m=20m --ipc=host ubuntu python3
  2. use some tmpfs memory. E.g.,
import os
import mmap
f = open('/dev/shm/foo', 'a+')
f.truncate(25000000)         
mm = mmap.mmap(f.fileno(), 0, mmap.MAP_SHARED)
for i in range(mm.size()):
  mm.write_byte(1)
  1. repeat (usually 2x or 3x times)

Describe the results you received:
First time, the Python (and by extension, the container) is Killed as it should be for writing too much to memory. Maybe ~18mb are written to the file. After 2 or 3 runs of the above steps (depending on container overhead), the file is fully written to and the PID is not killed. Moreover, with each successive run, another ~18mb more is written to the file than in the previous run.

Describe the results you expected:
The PID should be killed every time (1) & (2) are performed and ~the same amount of data should be written to memory (aka the tmpfs mmap'ed file) each time.

Additional information you deem important (e.g. issue happens only occasionally):
Looking at /sys/fs/cgroup/memory/, it looks like the mmap'd memory for the container instance is 'inherited' by the docker daemon's parent cgroup. In my case above, that means each time the container is run, because that memory is being accounted for in the docker daemon itself, the new container instance effectively has N*20mb of data (N being the number of times steps (1) and (2) have been performed) it is 'allowed' to write to the mmap'ing (the first (N-1)*20mb doesn't count towards the container's memory limit). Also if it matters, there is no swap space on the host.

Output of docker version:
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64
Experimental: false

Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 180
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.10.0-37-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 31.38GiB
Name: Redacted
ID: Redacted
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions