gnu screen run extremely slow when using 'root' user in container since docker 23.0.1 #45380
Description
Description
When run command screen by root user in container , this command would take a long time to respond .
If creating a new user , and su to this user and run screen , the speed back to normal
Reproduce
- Pull a image, feora:latest or ubuntu:18.04
docker pull fedora
- Create a container, No matter if '--privileged' is given
docker run -itd fedora bash
- Enter into container
docker exec -it fedora bash
- run gnu screen under root
dnf install screen
screen -lsrespond is extremely slow
- create a new user windy and run command under this user
useradd -m windy
su windy
screen -ls
respond is fast
Expected behavior
'screen ' command should finish immediately instead of block more than 4 seconds
docker version
# Local Machine
Client:
Version: 23.0.4
API version: 1.42
Go version: go1.20.3
Git commit: f480fb1e37
Built: Fri Apr 21 22:05:37 2023
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 23.0.4
API version: 1.42 (minimum version 1.12)
Go version: go1.20.3
Git commit: cbce331930
Built: Fri Apr 21 22:05:37 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.0
GitCommit: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
runc:
Version: 1.1.6
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0
# Server Machine
Client:
Version: 23.0.1
API version: 1.42
Go version: go1.20
Git commit: a5ee5b1dfc
Built: Sat Feb 11 13:58:04 2023
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 23.0.1
API version: 1.42 (minimum version 1.12)
Go version: go1.20
Git commit: bc3805a0a0
Built: Sat Feb 11 13:58:04 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.0
GitCommit: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
runc:
Version: 1.1.4
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
# Local Machine
Client:
Context: default
Debug Mode: false
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 23.0.4
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
runc version:
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.2.12-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 15.44GiB
Name: suzakuwcx
ID: 14fa8633-36e9-4e6c-a16d-ebf0c9df933f
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
# Server Machine
Client:
Context: default
Debug Mode: false
Plugins:
compose: Docker Compose (Docker Inc.)
Version: 2.17.1
Path: /usr/lib/docker/cli-plugins/docker-compose
Server:
Containers: 4
Running: 2
Paused: 0
Stopped: 2
Images: 14
Server Version: 23.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
runc version:
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.21-1-lts
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 62.71GiB
Name: suzakuwcx-server
ID: TNDU:4543:ZPKW:SNJ4:RVXH:4M5F:3NSU:4IAC:OE4X:D4SM:JQ27:DCS5
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
-
No matter which image I choose , this problem is still exist in my two linux machine
-
The entirely system is idle. It is unlikely performance problem.
-
Some debug infomation, debugging on fedora:latest image
I try to download sources code of the gnu screen, and debugging by gdb, I found that most of the time is caused by snprintf function below
Here is the FlameGraph I got:
perf record -F max -a -g -- screen -ls
It seems that the bottleneck is syscall, So I also try to downgrade linux version from 6.2.12-arch1-1 to 6.1.12-arch1-1 , but problem not solved. Only when I downgrade docker to 20.10.23 can solved this problem