Containers with memory limit < 6M fail with obscure error message

## TL;DR

- Likely related to the security fix in runc, containers now require a bigger memory-footprint to start (see https://github.com/opencontainers/runc/issues/1980).
- Trying to start a container with less than 6m of memory will fail, and produce a confusing error message.
- Containers that fail to start because of this limit, will not be marked as `OOMKilled`, which makes it harder for users to find the cause of the failure
    - can we detect this situation? is it because the shim was killed?
- Docker defines a minimum memory-limit of 4m, which no longer is adequate

We should probably look into the memory-consumption needed, and try to reduce this overhead. Possibly alternative approaches could help (see "possible enhancements for consideration" at the end).


## Test environment

Steps were executed on a DigitalOcean machine (1 CPU, 2GB memory). Details below

> **Note**: this was running on a machine without swap, so each of the below was printing
>
> ```
> WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
> ```
>
> I removed those warnings for readability. I later re-configured the machine to enable swap (described further down)

<details>
    <summary>docker version and docker info</summary>

```
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:07 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
```


```
Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 10
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-66-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.947GiB
 Name: ubuntu-s-1vcpu-2gb-ams3-01
 ID: 7VHE:PVY6:PXSQ:APMB:RC3E:ITUA:RWAJ:6EZK:QLLC:PHT6:DIIJ:5A6K
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
```

</details>

## Problem description

For some time, we defined a minimum limit for `--memory` limits (mostly based on some "trial and error" to see what would be a reasonable minimum). Currently that limit is 4MB

```bash
docker run --rm --memory=3m alpine echo success
docker: Error response from daemon: Minimum memory limit allowed is 4MB.
See 'docker run --help'.
```

While looking at https://github.com/docker/for-linux/issues/1054 on a Ubuntu 18.04 machine (kernel `4.15.0-66-generic`), I noticed that this limit is too low, and that low values now fail with a confusing error message (possibly related to security fixes in runc now causing more memory to be consumed during creation/starting the container).

Trying to find what the minimum was, here's trying to set 4m, 5m, 6m limits:

```bash
run --rm --memory=4m alpine echo success
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled

docker run --rm --memory=5m alpine
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"5242880\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/723b06ffb440d3b3629ba80cc160ee1aacb8aaa6e583f4b2289d4f649404c0e9/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/723b06ffb440d3b3629ba80cc160ee1aacb8aaa6e583f4b2289d4f649404c0e9/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled

docker run --rm --memory=6m alpine echo success
success
```

Cleaned up error message:

```
setting cgroup config for procHooks process caused
failed to write "4194304" to "/sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes":
write /sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes: device or resource busy "": unknown.
```

Next, finding a more "exact" limit (is it 6m exactly?), some interesting bits:

`5.1m` did not print the cgroup-failure, but looks like the process did not come up; notice that no container-ID is printed (which could be a bug in our code?)

```bash
docker run --rm --memory=5.1m alpine echo success
docker: Error response from daemon: cannot start a stopped process: unknown.
```

Trying again, without `--rm`, so that I can inspect the container's state:

```bash
docker run --memory=5.1m alpine echo success

docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
96cf0c635fd2        alpine              "echo success"      3 seconds ago       Created                                 objective_bohr

docker container inspect --format='{{json .State}}' objective_bohr | jq .

{
  "Status": "created",
  "Running": false,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": false,
  "Dead": false,
  "Pid": 0,
  "ExitCode": 128,
  "Error": "cannot start a stopped process: unknown",
  "StartedAt": "0001-01-01T00:00:00Z",
  "FinishedAt": "0001-01-01T00:00:00Z"
}
```

So, the container was not marked as `OOMKilled`


In the 5.1m ... 5.8m range, I also often get a `context canceled`

```bash
docker run --rm --memory=5.1m alpine echo success
docker: Error response from daemon: cannot start a stopped process: unknown.
ERRO[0000] error waiting for container: context canceled
```

And the corresponding logs (`/var/log/syslog`). Relevant information;

```
Task in /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e killed as a result of limit of /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e
memory: usage 5220kB, limit 5220kB, failcnt 26
memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
kmem: usage 440kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e: cache:0KB rss:4780KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:4768KB inactive_file:0KB active_file:0KB unevictable:0KB
[ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[15121]     0 15121   103754     2473   143360        0             0 runc:[2:INIT]
Memory cgroup out of memory: Kill process 15121 (runc:[2:INIT]) score 1864 or sacrifice child
Killed process 15121 (runc:[2:INIT]) total-vm:415016kB, anon-rss:4720kB, file-rss:5172kB, shmem-rss:0kB
oom_reaper: reaped process 15121 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
```



<details>

```
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:09.867865476Z" level=warning msg="Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.323768] docker0: port 1(veth6de25f3) entered blocking state
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.323771] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.324087] device veth6de25f3 entered promiscuous mode
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 networkd-dispatcher[854]: WARNING:Unknown index 115 seen, reloading interface list
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15081]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15081]: Could not generate persistent MAC address for veth551ed8f: No such file or directory
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15084]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.333033] IPv6: ADDRCONF(NETDEV_UP): veth6de25f3: link is not ready
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15084]: Could not generate persistent MAC address for veth6de25f3: No such file or directory
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 containerd[7309]: time="2020-07-01T08:12:09.981585296Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e/shim.sock" debug=false pid=15095
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: veth6de25f3: Gained carrier
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: docker0: Gained carrier
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.650910] eth0: renamed from veth551ed8f
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.651164] IPv6: ADDRCONF(NETDEV_CHANGE): veth6de25f3: link becomes ready
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.651188] docker0: port 1(veth6de25f3) entered blocking state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.651190] docker0: port 1(veth6de25f3) entered forwarding state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663278] runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663279] runc:[2:INIT] cpuset=4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e mems_allowed=0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663285] CPU: 0 PID: 15121 Comm: runc:[2:INIT] Not tainted 4.15.0-66-generic #75-Ubuntu
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663286] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663286] Call Trace:
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663296]  dump_stack+0x63/0x8e
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663299]  dump_header+0x71/0x285
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663301]  oom_kill_process+0x21f/0x420
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663302]  out_of_memory+0x2b6/0x4d0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663306]  mem_cgroup_out_of_memory+0xbb/0xd0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663307]  mem_cgroup_oom_synchronize+0x2e8/0x320
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663309]  ? mem_cgroup_css_online+0x40/0x40
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663310]  pagefault_out_of_memory+0x36/0x7b
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663314]  mm_fault_error+0x90/0x180
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663315]  __do_page_fault+0x46b/0x4b0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663317]  ? __schedule+0x256/0x880
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663319]  do_page_fault+0x2e/0xe0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663321]  ? async_page_fault+0x2f/0x50
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663324]  do_async_page_fault+0x51/0x80
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663325]  async_page_fault+0x45/0x50
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663327] RIP: 0033:0x7f77a63f30dd
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663328] RSP: 002b:00007fff84b6ba40 EFLAGS: 00010202
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663329] RAX: 000000000000ef61 RBX: 00000000000000f1 RCX: 000055e3eeff8fb0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663330] RDX: 00000000000000f1 RSI: 0000000000000000 RDI: 000055e3eeff90a0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663330] RBP: ffffffffffffffa8 R08: 000055e3eeff59e0 R09: 0000000000000000
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663331] R10: 000055e3eefe7010 R11: 0000000000000000 R12: 000000000000000d
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663331] R13: 00007f77a6749ca0 R14: 00007f77a6749c40 R15: 0000000000000000
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663332] Task in /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e killed as a result of limit of /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663337] memory: usage 5220kB, limit 5220kB, failcnt 26
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663337] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663338] kmem: usage 440kB, limit 9007199254740988kB, failcnt 0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663338] Memory cgroup stats for /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e: cache:0KB rss:4780KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:4768KB inactive_file:0KB active_file:0KB unevictable:0KB
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663362] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663408] [15121]     0 15121   103754     2473   143360        0             0 runc:[2:INIT]
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663409] Memory cgroup out of memory: Kill process 15121 (runc:[2:INIT]) score 1864 or sacrifice child
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.666109] Killed process 15121 (runc:[2:INIT]) total-vm:415016kB, anon-rss:4720kB, file-rss:5172kB, shmem-rss:0kB
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.671888] oom_reaper: reaped process 15121 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 containerd[7309]: time="2020-07-01T08:12:10.316214199Z" level=info msg="shim reaped" id=4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.328018040Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: veth6de25f3: Lost carrier
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.742548] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.745452] veth551ed8f: renamed from eth0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.757493] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 networkd-dispatcher[854]: WARNING:Unknown index 114 seen, reloading interface list
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.762798] device veth6de25f3 left promiscuous mode
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.762808] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 networkd-dispatcher[854]: ERROR:Unknown interface index 114 seen even after reload
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.402854115Z" level=error msg="4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e cleanup: failed to delete container from containerd: no such container"
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.412135381Z" level=error msg="Handler for POST /v1.40/containers/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e/start returned error: cannot start a stopped process: unknown"
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.426702914Z" level=error msg="failed to process event" container=4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e error="could not find container 4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e: No such container: 4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e" event=exit event-info="{4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e 4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e 15121 137 2020-07-01 08:12:10.285461759 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15167]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15167]: link_config: could not get ethtool features for veth551ed8f
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15167]: Could not set offload features of veth551ed8f: No such device
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: docker0: Lost carrier
```

</details>


Trying with `--network=host` (skipping creation of a network namespace), didn't help; memory limit still looks to be too low:

```bash
docker run --rm --network=host --memory=5.1m alpine echo success
docker: Error response from daemon: cannot start a stopped process: unknown.
ERRO[0000] error waiting for container: context canceled
```

For 5.9m, I got an interesting one. Unfortunately I wasn't able to reproduce it after, and didn't capture logs (weren't able to find them back);

```bash
docker run --rm --memory=5.9m alpine echo success
standard_init_linux.go:211: exec user process caused "argument list too long"
```

(whaaa?)

With 5.9m or up, things went _mostly_ ok;

```bash
docker run --rm --memory=5.9m alpine echo success
success
```

### updating memory limit after starting

So, the problem looks to be the memory consumption when _starting_ the container.

```bash
docker run -dit --name mycontainer --memory=8m alpine
acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86
```

Once the container is started, it's consuming little memory:

```bash
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
acdd326419f0        mycontainer         0.00%               460KiB / 8MiB       89.84%              1.12kB / 0B         0B / 0B             1
```

And we can reduce the memory-limit to go as low as 512k:

```bash
echo 4194304 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
echo 2097152 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
echo 1048576 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
echo 524288  > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
```

At this point, the container is still running fine (within limits);

```
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
acdd326419f0        mycontainer         0.00%               460KiB / 512KiB     89.84%              1.12kB / 0B         0B / 0B             1
```

Going to 256k, an error occurred, and the memory limit cannot be updated;

```bash
echo 262144 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
-bash: echo: write error: Device or resource busy

cat /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
524288
```

So, looks like the kernel refuses to set a limit < amount of actual memory used in the cgroup

Trying if stopping and re-starting a container with the low limit works (or if memory use is needed to _start_ the container);

```bash
docker kill mycontainer
mycontainer

docker start mycontainer
mycontainer
```

That didn't work, because the cgroup is reset to the container's config when starting again;

```
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
acdd326419f0        mycontainer         0.00%               468KiB / 8MiB       5.71%               906B / 0B           0B / 0B             1
```

Trying to update the container's limit to 4M before killing;

```bash
docker container update --memory=4m mycontainer
mycontainer

docker kill mycontainer
mycontainer

docker start mycontainer
Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes: device or resource busy\\\"\"": unknown
Error: failed to start containers: mycontainer
```

So, looks like _starting_ the container requires more than 4M, so this is not a solution.


### Does swap help?

No, enabling swap does not help. I tried enabling swap limit support:

```bash
cat /etc/default/grub |grep GRUB_CMDLINE_LINUX=
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
```

After enabling swap and rebooting the machine, the problem is the same;

```bash
docker run --rm --memory=4m alpine echo success
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/10ef86ea7cd32cd775f0e6b1a9fd882a0b0d4191f1817b1375d478ce2cbac90d/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/10ef86ea7cd32cd775f0e6b1a9fd882a0b0d4191f1817b1375d478ce2cbac90d/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.

docker run --rm --memory=4m --memory-swap=16m alpine echo success
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/6772c052387d1a4a6321c97cc2a4d1f2e5a3908e89510e433eae3aecc36b7438/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/6772c052387d1a4a6321c97cc2a4d1f2e5a3908e89510e433eae3aecc36b7438/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.
```


### possible enhancements for consideration

Wondering if it would be useful to allow lower memory limits, and to achieve that, we could;

1. create and start container (with a minimum limit of (e.g.) `memory-limit as defined by user + 6m` (accounting for the overhead of starting the container))
2. after container is started update the limit to the specified value (e.g. `512k`)

Not sure if worth looking into (and if it would be racy)


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Containers with memory limit < 6M fail with obscure error message #41167

TL;DR

Test environment

Problem description

updating memory limit after starting

Does swap help?

possible enhancements for consideration

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Containers with memory limit < 6M fail with obscure error message #41167

Description

TL;DR

Test environment

Problem description

updating memory limit after starting

Does swap help?

possible enhancements for consideration

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions