Docker stats memory doesn't include tmpfs usage

**Description**

The `MEMORY USAGE` and `MEM %` columns in `docker stats` don't include RAM used by `tmpfs` filesystems.

**Steps to reproduce the issue:**

1) Run
    ```sh
    docker run --shm-size=256MB --memory=256MB -it alpine sh -c "
            dd if=/dev/urandom of=/dev/shm/file bs=32M
            ls -lh /dev/shm
            read
    "
    ```
    and/or
    ```sh
    docker run --shm-size=256MB --memory=256MB -it alpine sh -c "
            dd if=/dev/urandom of=/tmp/file bs=32M
            ls -lh /tmp
            read
    "
    ```
2) Observe that the `dd` process is killed after exhausting the `memory`/`shm-size` limits and the RAM resident filesystem now contains a `200M`ish file:
    ```
    Killed
    total 217M   
    -rw-r--r--    1 root     root      216.6M Mar 20 05:23 file
    ```
3) Run `docker stats`

**Describe the results you received:**
```
CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
8c601a0d8404        keen_elbakyan          0.00%               7.316MiB / 256MiB   2.86%               8.49kB / 0B         0B / 0B             1
7066646c1a76        infallible_engelbart   0.00%               1.613MiB / 256MiB   0.63%               5.06kB / 0B         0B / 0B             1
```

**Describe the results you expected:**

The `MEM USAGE` and `MEM %` columns in the above table to be around `200M`ish and `78%`ish.

**Output of `docker version`:**

```
Docker version 19.03.7-ce, build 7141c199a2
```

**Output of `docker info`:**

<details>

```
Client:
 Debug Mode: false

Server:
 Containers: 21
  Running: 2
  Paused: 0
  Stopped: 19
 Images: 5
 Server Version: 19.03.7-ce
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.24-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.56GiB
 Name: ruro-laptop
 ID: XHD5:R5HL:34CY:3UJ2:4QD6:IP3P:FIOD:NUZX:ZBEE:XND6:OJ5P:J6RM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```
</details>

**Additional details:**

Output of
```sh
curl --unix-socket /var/run/docker.sock "http://localhost/containers/<container-id>/stats?stream=0" | python -m json.tool
```

<details>
<summary>For the <code>/tmp</code> container</summary>

```json
{
    "read": "2020-03-20T05:24:03.754425567Z",
    "preread": "2020-03-20T05:24:02.752857441Z",
    "pids_stats": {
        "current": 1
    },
    "blkio_stats": {
        "io_service_bytes_recursive": [],
        "io_serviced_recursive": [],
        "io_queue_recursive": [],
        "io_service_time_recursive": [],
        "io_wait_time_recursive": [],
        "io_merged_recursive": [],
        "io_time_recursive": [],
        "sectors_recursive": []
    },
    "num_procs": 0,
    "storage_stats": {},
    "cpu_stats": {
        "cpu_usage": {
            "total_usage": 1841965540,
            "percpu_usage": [
                992402,
                1665649,
                1619827291,
                2729087,
                4554574,
                5189767,
                201081062,
                5925708
            ],
            "usage_in_kernelmode": 1620000000,
            "usage_in_usermode": 200000000
        },
        "system_cpu_usage": 2611727730000000,
        "online_cpus": 8,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "precpu_stats": {
        "cpu_usage": {
            "total_usage": 1841965540,
            "percpu_usage": [
                992402,
                1665649,
                1619827291,
                2729087,
                4554574,
                5189767,
                201081062,
                5925708
            ],
            "usage_in_kernelmode": 1620000000,
            "usage_in_usermode": 200000000
        },
        "system_cpu_usage": 2611719640000000,
        "online_cpus": 8,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "memory_stats": {
        "usage": 234754048,
        "max_usage": 268439552,
        "stats": {
            "active_anon": 0,
            "active_file": 113635328,
            "cache": 227082240,
            "dirty": 197885952,
            "hierarchical_memory_limit": 268435456,
            "hierarchical_memsw_limit": 536870912,
            "inactive_anon": 0,
            "inactive_file": 113577984,
            "mapped_file": 0,
            "pgfault": 8976,
            "pgmajfault": 0,
            "pgpgin": 64152,
            "pgpgout": 8749,
            "rss": 0,
            "rss_huge": 0,
            "total_active_anon": 0,
            "total_active_file": 113635328,
            "total_cache": 227082240,
            "total_dirty": 197885952,
            "total_inactive_anon": 0,
            "total_inactive_file": 113577984,
            "total_mapped_file": 0,
            "total_pgfault": 8976,
            "total_pgmajfault": 0,
            "total_pgpgin": 64152,
            "total_pgpgout": 8749,
            "total_rss": 0,
            "total_rss_huge": 0,
            "total_unevictable": 0,
            "total_writeback": 0,
            "unevictable": 0,
            "writeback": 0
        },
        "failcnt": 19,
        "limit": 268435456
    },
    "name": "/keen_elbakyan",
    "id": "8c601a0d84047b079b4e409c0fd3197efc1ce80bd964c472f49cbf769517ee5d",
    "networks": {
        "eth0": {
            "rx_bytes": 5610,
            "rx_packets": 39,
            "rx_errors": 0,
            "rx_dropped": 0,
            "tx_bytes": 0,
            "tx_packets": 0,
            "tx_errors": 0,
            "tx_dropped": 0
        }
    }
}
```
</details>

<details>
<summary>For the <code>/dev/shm</code> container</summary>

```json
{
    "read": "2020-03-20T05:32:09.684746601Z",
    "preread": "2020-03-20T05:32:08.683481358Z",
    "pids_stats": {
        "current": 1
    },
    "blkio_stats": {
        "io_service_bytes_recursive": [],
        "io_serviced_recursive": [],
        "io_queue_recursive": [],
        "io_service_time_recursive": [],
        "io_wait_time_recursive": [],
        "io_merged_recursive": [],
        "io_time_recursive": [],
        "sectors_recursive": []
    },
    "num_procs": 0,
    "storage_stats": {},
    "cpu_stats": {
        "cpu_usage": {
            "total_usage": 1494756157,
            "percpu_usage": [
                822498,
                856679,
                558496,
                8284470,
                200931734,
                1192872,
                1278601167,
                3508241
            ],
            "usage_in_kernelmode": 1270000000,
            "usage_in_usermode": 200000000
        },
        "system_cpu_usage": 2615624390000000,
        "online_cpus": 8,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "precpu_stats": {
        "cpu_usage": {
            "total_usage": 1494756157,
            "percpu_usage": [
                822498,
                856679,
                558496,
                8284470,
                200931734,
                1192872,
                1278601167,
                3508241
            ],
            "usage_in_kernelmode": 1270000000,
            "usage_in_usermode": 200000000
        },
        "system_cpu_usage": 2615616430000000,
        "online_cpus": 8,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "memory_stats": {
        "usage": 234721280,
        "max_usage": 268439552,
        "stats": {
            "active_anon": 0,
            "active_file": 0,
            "cache": 233029632,
            "dirty": 0,
            "hierarchical_memory_limit": 268435456,
            "hierarchical_memsw_limit": 536870912,
            "inactive_anon": 232894464,
            "inactive_file": 0,
            "mapped_file": 0,
            "pgfault": 8976,
            "pgmajfault": 0,
            "pgpgin": 65604,
            "pgpgout": 8741,
            "rss": 0,
            "rss_huge": 0,
            "total_active_anon": 0,
            "total_active_file": 0,
            "total_cache": 233029632,
            "total_dirty": 0,
            "total_inactive_anon": 232894464,
            "total_inactive_file": 0,
            "total_mapped_file": 0,
            "total_pgfault": 8976,
            "total_pgmajfault": 0,
            "total_pgpgin": 65604,
            "total_pgpgout": 8741,
            "total_rss": 0,
            "total_rss_huge": 0,
            "total_unevictable": 0,
            "total_writeback": 0,
            "unevictable": 0,
            "writeback": 0
        },
        "failcnt": 30,
        "limit": 268435456
    },
    "name": "/infallible_engelbart",
    "id": "7066646c1a76b421db4ccee09d9bf6e2802203b42509b4138913bcf16a5132b1",
    "networks": {
        "eth0": {
            "rx_bytes": 5737,
            "rx_packets": 40,
            "rx_errors": 0,
            "rx_dropped": 0,
            "tx_bytes": 0,
            "tx_packets": 0,
            "tx_errors": 0,
            "tx_dropped": 0
        }
    }
}
```
</details>

Addressing similar issues:

- This issue might be an unintended side effect of issue #10824 / https://github.com/docker/cli/pull/80. Not including caches in memory usage is correct, since they will be automatically freed, if memory is needed. Not including `tmpfs` is not correct since the kernel can't just delete these files and so exhausting this resource will result in dead processes.
- Possibly related to #39004, but not likely imo.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Docker stats memory doesn't include tmpfs usage #40727

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Docker stats memory doesn't include tmpfs usage #40727

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions