Description
|
ctx, cancel := context.WithTimeout(ctx, 5*time.Second) |
|
err := publisher.Publish(ctx, runc.GetTopic(e), e) |
|
cancel() |
The shim tries to send an event.
If the send fails, the event is added to a queue and it is retried a few times.
|
i := &item{ |
|
ev: &v1.Envelope{ |
|
Timestamp: time.Now(), |
|
Namespace: ns, |
|
Topic: topic, |
|
Event: any, |
|
}, |
|
ctx: ctx, |
|
} |
|
|
|
if err := l.forwardRequest(i.ctx, &v1.ForwardRequest{Envelope: i.ev}); err != nil { |
|
l.queue(i) |
|
return err |
|
} |
The problem is the context is re-used for all send requests, but it has already been cancelled by the time the send is attempted again since the original function will have returned.
Steps to reproduce the issue:
- Start a container
- Kill containerd
- Kill the container process
- Start containerd again
You will see logs from the shim saying that it can't send the event due to context cancelled.
Describe the results you received:
time="2020-07-17T23:55:43.878255535Z" level=error msg="forward event" error="context canceled"
time="2020-07-17T23:55:49.879116394Z" level=error msg="evicting /tasks/exit from queue because of retry count"
Describe the results you expected:
Event should be able to send.
Output of containerd --version:
containerd github.com/containerd/containerd v1.4.0-beta.2-17-g4feb8c46.m 4feb8c462393ce6834dda9e3464c4fee8ee73232.m
Description
containerd/runtime/v2/runc/v2/service.go
Lines 772 to 774 in 4feb8c4
The shim tries to send an event.
If the send fails, the event is added to a queue and it is retried a few times.
containerd/runtime/v2/shim/publisher.go
Lines 112 to 125 in 4feb8c4
The problem is the context is re-used for all send requests, but it has already been cancelled by the time the send is attempted again since the original function will have returned.
Steps to reproduce the issue:
You will see logs from the shim saying that it can't send the event due to context cancelled.
Describe the results you received:
Describe the results you expected:
Event should be able to send.
Output of
containerd --version: