What is the problem you're trying to solve
The data flow of stdout/stderr of container is as follows:
[container] --pipe-> [containerd-shim] --fifo-> [containerd-cri] --file-> disk

We find the containerd uses a lot of cpu to format the log, so I try to optimize the function redirectLogs of cri to reduce the usage with PR #5286.

Describe the solution you'd like

I start a nginx pod on a 1c2g node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-tlinux
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

And then use wrk on another node:

 wrk -t10 -d10s -c100  --latency http://<pod-ip>:80

We got the following results:

We got the flame graph of containerd following:

According to the flame graph, I have done some base optimization:

• Use the same buffer in the closures writeLineBuffer.
• Reduce object allocation. It will reduce time of GC also.
• Reduce byte array copy.

After the optimization, the rps of wrk increased from 14096.50 to 16072.36.
The cpu of nginx is up to 91% :

The flame graph is as following:

There are some ideas need to be discussed:

• Use syscall or something else to replace the time.now(). It works ! But it may reduce the time accuracy.
• Use writev to write multiple lines at once. I havn't try this, because the solution needs to modify more code.

Additional context
PR is here #5286 . I'll add the unit test soon.

More information

uname -a

# uname -a
Linux VM-10-16-centos 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

containerd version

containerd --version
containerd github.com/containerd/containerd v1.5.0-beta.4-92-g1be7e844f.m 1be7e844f2d26a2df61862980f18c3af41dc2ade.m