Skip to content

Performance Degradation Observed on Containerd After Migrating from 1.7.17 to 2.x #11726

@sajuptpm

Description

@sajuptpm

Description

Description:
We analyzed performance metrics for both single and parallel deployments involving 5 pods. For all tests, the imagePullPolicy was set to "Always."

Performance Data for Containerd Versions

Containerd Version Single Deployment Time (ms) Parallel Deployment Time (ms)
Containerd 1.7.17 2740.2 4110.0
Containerd 2.0.2 2904.0 5137.8
Containerd 2.1 Dev 2720.4 4667.0
Containerd 2.1 Dev + ImageVerifier Patch 2933.4 4891.0

Image

Key Observations:

  1. Migration from Containerd 1.7.17 to 2.0.2 resulted in performance degradation:

    • Single deployment: Additional delay of 164 milliseconds.
    • Parallel deployment (5 pods): Additional delay of 1027 milliseconds.
  2. Migration from Containerd 2.0.2 to 2.1 (without the ImageVerifier plugin) improved performance:

    • Single deployment: Deployment time reduced by 183 milliseconds.
    • Parallel deployment (5 pods): Deployment time reduced by 470.8 milliseconds.
  3. Enabling the ImageVerifier plugin in Containerd 2.1 (PR Update CRI to use transfer service for image pull by default #8515) resulted in slight performance delays compared to Containerd 2.1 without the ImageVerifier plugin:

    • Single deployment: Additional delay of approximately 213 milliseconds.
    • Parallel deployment (5 pods): Additional delay of approximately 224 milliseconds.
    • These delays are considered acceptable due to the added image verification functionality provided by the Cosign Image Verifier tool.

Deployment YAML used for testing (different images were used in all deployments)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
      - name: app1-container
        image: localhost:5000/test-image-x:1
        imagePullPolicy: Always
        command: ["sleep", "3600"]


### Steps to reproduce the issue

1.
2.
3.


### Describe the results you received and expected

Should not degrade performance

### What version of containerd are you using?

1.7.17 to 2.x

### Any other relevant information

_No response_

### Show configuration if it is related to CRI plugin.

_No response_

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions