fix: sanitize error before gRPC return to prevent credential leak in pod events by aadhar-agarwal · Pull Request #12801 · containerd/containerd

aadhar-agarwal · 2026-01-20T22:41:47Z

PR #12491 fixed credential leaks in containerd logs but the gRPC error returned to kubelet still contains sensitive information (e.g., SAS tokens). This is visible in Kubernetes pod events via kubectl describe pod.

Issue:
The SanitizeError call was placed inside the defer block:

defer func() {
    if err != nil {
        err = ctrdutil.SanitizeError(err)  // Runs AFTER return is evaluated
        log.G(ctx).WithError(err).Errorf(...)
    }
}()
res, err = in.c.PullImage(...)
return res, errgrpc.ToGRPC(err)  // Uses ORIGINAL unsanitized error

Go evaluates errgrpc.ToGRPC(err) before the defer runs, so the gRPC message contains the original unsanitized error with credentials.

Fix:
Move SantiizeError before the return statement:

res, err = in.c.PullImage(...)
if err != nil {
    err = ctrdutil.SanitizeError(err)
}
return res, errgrpc.ToGRPC(err)  // Now uses sanitized error

This ensures both the logged error and the gRPC error use the sanitized error.

Testing:

Tested on an AKS cluster
Blocked blob storage IPs via iptables to force image pull failure
Deployed pod with ACR image requiring SAS token authentication

When describing pod:

containerd log:

Jan 20 22:35:45 aks-nodepool1-35109453-vmss000000 containerd[38541]: time="2026-01-20T22:35:45.730403025Z" level=error msg="PullImage \"akscon
tainerhost.azurecr.io/newimage:v1\" failed" error="rpc error: code = Unknown desc = failed to pull and unpack image \"akscontainerhost.azurecr
.io/newimage:v1\": failed to copy: httpReadSeeker: failed open: failed to do request: Get \"https://eusmanaged213.blob.core.windows.net/d46180
728b794a35a39c3b22ed20f8a3-6k5n90bm44//docker/registry/v2/blobs/sha256/bc/bc717f9cc8f343d7e59d10bc7fc0c811530afa98d437c9f6bcd8fb3dfc2308da/dat
a?anon=%5BREDACTED%5D&regid=%5BREDACTED%5D&se=%5BREDACTED%5D&sig=%5BREDACTED%5D&ske=%5BREDACTED%5D&skoid=%5BREDACTED%5D&sks=%5BREDACTED%5D&skt
=%5BREDACTED%5D&sktid=%5BREDACTED%5D&skv=%5BREDACTED%5D&sp=%5BREDACTED%5D&spr=%5BREDACTED%5D&sr=%5BREDACTED%5D&sv=%5BREDACTED%5D\": dial tcp 2
0.60.63.193:443: connect: connection refused"
Jan 20 22:35:45 aks-nodepool1-35109453-vmss000000 containerd[38541]: time="2026-01-20T22:35:45.730424836Z" level=info msg="stop pulling image
akscontainerhost.azurecr.io/newimage:v1: active requests=0, bytes read=0"
root@aks-nodepool1-35109453-vmss000000 [ / ]#

…pod events PR containerd#12491 fixed credential leaks in containerd logs but the gRPC error returned to kubelet still contained sensitive information. This was visible in Kubernetes pod events via `kubectl describe pod`. The issue was that SanitizeError was called inside the defer block, but errgrpc.ToGRPC(err) was evaluated before the defer ran, so the gRPC message contained the original unsanitized error. Move SanitizeError before the return statement so both the logged error and the gRPC error are sanitized. Ref: containerd#5453 Signed-off-by: Aadhar Agarwal <[email protected]>

AkihiroSuda

Thanks, but next time please follow https://github.com/containerd/containerd/security

aadhar-agarwal · 2026-01-21T00:28:47Z

Thanks, but next time please follow https://github.com/containerd/containerd/security

Ah, thank you for pointing this out - Sounds good!

AkihiroSuda · 2026-01-21T07:46:08Z

/cherry-pick release/2.2
/cherry-pick release/2.1
/cherry-pick release/1.7

k8s-infra-cherrypick-robot · 2026-01-21T07:46:55Z

@AkihiroSuda: #12801 failed to apply on top of branch "release/1.7":

Applying: fix: sanitize error before gRPC return to prevent credential leak in pod events
Using index info to reconstruct a base tree...
A	internal/cri/instrument/instrumented_service.go
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): internal/cri/instrument/instrumented_service.go deleted in HEAD and modified in fix: sanitize error before gRPC return to prevent credential leak in pod events. Version fix: sanitize error before gRPC return to prevent credential leak in pod events of internal/cri/instrument/instrumented_service.go left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 fix: sanitize error before gRPC return to prevent credential leak in pod events

Details

In response to this:

/cherry-pick release/2.2
/cherry-pick release/2.1
/cherry-pick release/1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2026-01-21T07:47:30Z

@AkihiroSuda: new pull request created: #12803

Details

In response to this:

/cherry-pick release/2.2
/cherry-pick release/2.1
/cherry-pick release/1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2026-01-21T07:48:08Z

@AkihiroSuda: new pull request created: #12804

Details

In response to this:

/cherry-pick release/2.2
/cherry-pick release/2.1
/cherry-pick release/1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrishenzie · 2026-04-07T22:19:41Z

/cherry-pick release/2.0

k8s-infra-cherrypick-robot · 2026-04-07T22:20:20Z

@chrishenzie: new pull request created: #13180

Details

In response to this:

/cherry-pick release/2.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

github-project-automation Bot added this to Pull Request Review Jan 20, 2026

k8s-ci-robot added the do-not-merge/work-in-progress label Jan 20, 2026

github-project-automation Bot moved this to Needs Triage in Pull Request Review Jan 20, 2026

k8s-ci-robot added the size/S label Jan 20, 2026

aadhar-agarwal mentioned this pull request Jan 20, 2026

Error messages returned from registries should hide sensitive information #5453

Closed

aadhar-agarwal marked this pull request as ready for review January 20, 2026 22:43

k8s-ci-robot removed the do-not-merge/work-in-progress label Jan 20, 2026

dosubot Bot added area/cri Container Runtime Interface (CRI) kind/bug labels Jan 20, 2026

AkihiroSuda approved these changes Jan 21, 2026

View reviewed changes

AkihiroSuda added cherry-pick/1.7.x Change to be cherry picked to release/1.7 branch cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch cherry-pick/2.2.x Change to be cherry picked to release/2.2 branch labels Jan 21, 2026

mxpv approved these changes Jan 21, 2026

View reviewed changes

github-project-automation Bot moved this from Needs Triage to Review In Progress in Pull Request Review Jan 21, 2026

mxpv added this pull request to the merge queue Jan 21, 2026

Merged via the queue into containerd:main with commit 207bf2b Jan 21, 2026
52 checks passed

github-project-automation Bot moved this from Review In Progress to Done in Pull Request Review Jan 21, 2026

This was referenced Jan 21, 2026

[release/2.1] fix: sanitize error before gRPC return to prevent credential leak in pod events #12803

Merged

[release/2.2] Harden error handling to strip potentially-sensitive registry parameters #12804

Merged

AkihiroSuda mentioned this pull request Jan 21, 2026

[release/1.7] fix: sanitize error before gRPC return to prevent credential leak in pod events #12805

Merged

AkihiroSuda removed cherry-pick/1.7.x Change to be cherry picked to release/1.7 branch cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch labels Jan 21, 2026

k8s-infra-cherrypick-robot mentioned this pull request Apr 7, 2026

[release/2.0] fix: sanitize error before gRPC return to prevent credential leak in pod events #13180

Closed

chrishenzie added the cherry-pick/2.0.x Change to be cherry picked to release/2.0 branch label Apr 7, 2026

chrishenzie mentioned this pull request Apr 7, 2026

[release/2.0] fix: sanitize error before gRPC return to prevent credential leak in pod events #13181

Merged

aadhar-agarwal mentioned this pull request Apr 14, 2026

Upgrade containerd2 to 2.1.7 microsoft/azurelinux#16665

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: sanitize error before gRPC return to prevent credential leak in pod events#12801

fix: sanitize error before gRPC return to prevent credential leak in pod events#12801
mxpv merged 1 commit intocontainerd:mainfrom
aadhar-agarwal:aadagarwal/sanitize-grpc-error-5453

aadhar-agarwal commented Jan 20, 2026 •

edited

Loading

Uh oh!

AkihiroSuda left a comment

Uh oh!

aadhar-agarwal commented Jan 21, 2026

Uh oh!

Uh oh!

AkihiroSuda commented Jan 21, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 21, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 21, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 21, 2026

Uh oh!

chrishenzie commented Apr 7, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

aadhar-agarwal commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

aadhar-agarwal commented Jan 21, 2026

Uh oh!

Uh oh!

AkihiroSuda commented Jan 21, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 21, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 21, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 21, 2026

Uh oh!

chrishenzie commented Apr 7, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aadhar-agarwal commented Jan 20, 2026 •

edited

Loading